
The GeForce RTX 4090 and the A100 both sit at the high end of NVIDIA's GPU lineup, but they're built for very different worlds.

The **RTX 4090** is a powerful consumer-grade GPU, designed for ultra high-performance gaming, professional creative work, and even some entry-level AI workloads. It's surprisingly capable across a variety of tasks, especially for its price point.

The **A100**, on the other hand, isn't aimed at consumers at all. It's an enterprise-grade GPU built for data centers, research labs, and AI teams running large-scale model training, inference, and simulation workloads – designed to move serious data at serious speed.

In this post, we'll look at how these two machines compare with each other, and why you might choose one over the other based on your specific needs.

### NVIDIA GeForce RTX 4090: A Workhorse with Range

When it launched in September 2022 as the top-tier consumer GPU of its generation, the RTX 4090 quickly became the go-to choice for 4K gaming, 3D rendering, and AI-enhanced creative workflows.

Built on NVIDIA's **Ada Lovelace** architecture and powered by the **AD102 chip**, the RTX 4090 introduced significant upgrades in ray tracing and Tensor Core acceleration, and delivered notable performance gains with **DLSS 3** in a variety of games.

While it's no longer the flagship GPU of NVIDIA's GeForce line (that title now goes to the RTX 5090, which you can read about [here](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know)), the RTX 4090 definitely still holds its own. It remains one of the most capable and well-rounded GPUs available at its price point for consumers and technical teams alike.

#### Some of the RTX 4090's key specs and features include:

-   **24 GB of GDDR6X memory** with **1,008 GB/s bandwidth** on a **384-bit** interface.  

-   **16,384 CUDA cores** for high-parallel throughput.  

-   **128 third-gen RT cores** and **512 fourth-gen Tensor cores** for advanced ray tracing and AI workloads.  

-   **DLSS 3** support for real-time upscaling and performance boosts in many games.  

-   **450 watts** of thermal design power (TDP), requiring an **850-watt power supply** due to its high processing capabilities.

If you're looking for serious performance in a desktop form factor, the RTX 4090 continues to deliver. It may not be the newest card on the market, but it remains a workhorse that gets the job done across a wide range of applications.

As capable as the RTX 4090 is, it's not built for everything. If your workloads go beyond what a high-end consumer GPU can handle, the A100 just might bring the compute power you need.

### NVIDIA A100: A Powerhouse with Purpose

The A100 is built for scale. Based on NVIDIA's Ampere architecture and powered by the **GA100 chip**, it's designed to tackle massive workloads – like foundational AI model training – thanks to its raw throughput, high-bandwidth memory, and ability to scale across multi-GPU environments.

When the A100 launched in June 2021, it introduced **third-gen Tensor Cores**, support for **TF32** precision, and **Multi-Instance GPU** (MIG) capabilities – enabling flexible resource partitioning and efficient parallel processing across shared environments.

While some of its core specs might look modest compared to the RTX 4090, the A100 delivers superior performance in AI and data-intensive workloads because it's optimized where it counts: memory bandwidth, interconnects, and specialized compute.

For instance, the A100's memory clock is much lower than the RTX 4090's on paper (roughly 3 Gbps vs. 21 Gbps). However, the A100 uses **HBM2e memory** with a much wider **5,120-bit interface**. This design allows it to deliver around **2 TB/s** of bandwidth – double the RTX 4090 – despite the lower frequency. It's an approach that prioritizes efficiency and scale.

#### To recap, here are a few notable features of the A100:

-   **80 GB of HBM2e memory** with about **2 TB/s of bandwidth** (twice that of the RTX 4090).  

-   **5,120-bit memory bus**, an exceptionally wide interface enabling fast access to memory, reducing bottlenecks in AI and high-performance computing (HPC) workloads.  

-   **432 third-gen Tensor Cores**, with high-throughput **TF32** performance for AI and robust **FP64** support for HPC workloads.  

-   **Multi-Instance GPU (MIG)** support, so it can be partitioned into up to 7 isolated GPU instances – ideal for multi-tenant workloads and fine-grained resource allocation.  

-   **NVLink and NVSwitch** allow up to 16 A100 GPUs to be interconnected at up to **600 GB/s** of bi-directional bandwidth per GPU.

In short, the A100 delivers serious performance for large-scale AI and HPC workloads. But if you're comparing it head-to-head with the RTX 4090 on specs alone, here's how the two stack up.

### RTX 4090 vs. A100: Key Specs Compared

For reference, this table highlights some of the features and specs of the RTX 4090 and A100:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Specification
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 4090
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                A100
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Ada Lovelace
            </td>
            <td class="border border-gray-300 px-4 py-2">Ampere</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">VRAM</td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                24 GB GDDR6X
            </td>
            <td class="border border-gray-300 px-4 py-2">80 GB HBM2e</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1313 MHz 21 Gbps
            </td>
            <td class="border border-gray-300 px-4 py-2">1512 MHz 3 Gbps</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Bus Width
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">384</td>
            <td class="border border-gray-300 px-4 py-2">5120</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Bandwidth
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1.01 TB/s
            </td>
            <td class="border border-gray-300 px-4 py-2">1.94 TB/s</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Streaming Multiprocessors (SMs)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">128</td>
            <td class="border border-gray-300 px-4 py-2">108</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">512</td>
            <td class="border border-gray-300 px-4 py-2">432</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                CUDA Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">16,384</td>
            <td class="border border-gray-300 px-4 py-2">6,912</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Ray Tracing Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">128</td>
            <td class="border border-gray-300 px-4 py-2">N/A</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Base Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2235 MHz
            </td>
            <td class="border border-gray-300 px-4 py-2">1065 MHz</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Boost Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2520 MHz
            </td>
            <td class="border border-gray-300 px-4 py-2">1410 MHz</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Multi-Instance GPUs
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                No MIG Support
            </td>
            <td class="border border-gray-300 px-4 py-2">
                Up to 7 MIGs @ 10GB each
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Thermal Design Power (TDP)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">450W</td>
            <td class="border border-gray-300 px-4 py-2">300W</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Recommended Power Supply
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">850W</td>
            <td class="border border-gray-300 px-4 py-2">700W</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Launch Date
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Sept. 20, 2022
            </td>
            <td class="border border-gray-300 px-4 py-2">June 28, 2021</td>
        </tr>
    </tbody>
</table>

As mentioned, specs are just one part of the equation. Ultimately, choosing the right GPU depends on your workload, environment, and priorities.

### Use Cases: Which GPU Should You Choose?

Here’s how the strengths of each GPU align with different types of use cases.

#### You should consider the RTX 4090 if you prefer:

-   **A powerful, desktop-friendly GPU** for high-end gaming, creative workflows, and light AI experimentation.
-   **Excellent performance-per-dollar in single-GPU setups**, particularly for developers and small teams.
-   **A widely available, less costly option** for rendering, video editing, or running small to medium AI models locally.

#### Whereas the A100 may be a better fit if you need:

-   **Scalable performance** for training large models, running high-throughput inference, or powering complex simulations.
-   **A data center-optimized GPU** with advanced memory architecture and precision modes tailored for AI and HPC workloads.
-   **Flexible deployment options**, including support for multi-instance workloads and high-speed interconnects in clustered environments.

\------------------------------

At **Vast.ai**, we understand there are other considerations, as well. Access, flexibility, and affordability are just as important, especially when you're experimenting, scaling, or moving fast.

That's why our market-based cloud GPU rental platform gives you affordable, on-demand access to RTX 4090s, A100s, and much more. With Vast.ai, you can instantly match the right hardware to your workload – without the upfront investment and overhead of managing your own infrastructure.

Try Vast today and discover what you can achieve with compute power that fits your needs and your budget\!


Compare NVIDIA’s RTX 4090 vs A100 GPUs to find out which is better for gaming, AI, HPC, or creative work—and how to access both on Vast.ai.

NVIDIA RTX 4090 vs. A100: Two Powerhouses, Two Purposes


NVIDIA's GeForce RTX 5090D was supposed to be a clever workaround – a GPU specially created to comply with U.S. export regulations while still delivering advanced gaming performance to Chinese consumers.

But recent reports suggest that even this modified version may not make it to market.

### NVIDIA RTX 5090D: A China-Specific Solution

The RTX 5090D was designed specifically for the Chinese market in response to the U.S. government's increasingly strict export rules. These rules target high-performance chips like the NVIDIA H100 and H20, and AMD's MI300 series, banning them from being sold in China if they exceed certain memory and I/O bandwidth thresholds.

To get around these limits, NVIDIA modified the RTX 5090D to lower its AI throughput while keeping specs that matter to gamers intact – such as core count, memory size, and clock speeds. That may not have been sufficient, however.

### The Bandwidth Problem

Intended to limit China's ability to develop advanced AI systems, U.S. export regulations on chips impose strict caps on memory and I/O bandwidth: **1,400 GB/s and 1,100 GB/s, respectively – _or_ a combined total over 1,700 GB/s**.

If the RTX 5090D exceeds any of these thresholds, it could still be restricted, despite being designed with reduced AI capabilities and not typically suited for large-scale enterprise workloads. The problem is, it very well might fall into the restricted category based on total bandwidth.

NVIDIA appears to be [taking preemptive action](https://wccftech.com/nvidia-reportedly-prepares-for-a-ban-on-the-geforce-rtx-5090d-in-china/) – asking add-in board (AIB) partners to halt shipments and prep work, and even putting a pause on related orders by cooling module manufacturers.

So far, there's been no official statement from NVIDIA. But after getting burned when the RTX 4090 was unexpectedly swept up in a prior wave of export controls, NVIDIA is reportedly acting early this time with this "precautionary" move amid ongoing regulatory uncertainty between the U.S. and China.

### What This Means for the Market

Whether this sales halt is temporary or permanent remains to be seen, but the wider implications are potentially far-reaching. The RTX 5090D was the only flagship-class Blackwell GPU intended for Chinese gamers. If it's taken off the table, that leaves a gap in the market – and an opportunity for rivals like AMD, which is reportedly preparing a China-exclusive gaming GPU.

Furthermore, these ongoing developments reflect a growing reality: that geopolitical decisions are impacting GPU availability worldwide.

While Chinese consumers lose access to another high-end option, global suppliers are keeping a close eye on what happens next. Constraints in one region can ripple across the supply chain, affecting availability, pricing, and competitive positioning elsewhere, as well.

### High-End Compute On Demand

While geopolitical tensions disrupt supply in some regions, developers, researchers, and creators still need reliable access to powerful hardware. Vast.ai offers a flexible and cost-effective alternative.

Our cloud GPU rental platform delivers high-performance compute – H100s, H200s, RTX 5090s, and more – at a fraction of the typical cost. With on-demand pricing and spot instance options, **Vast users save 5–6X on GPU compute** compared to traditional cloud providers.

Explore your options and get started today\! With Vast, you don't have to wait to access the power you want.


NVIDIA's RTX 5090D, designed specifically for the Chinese market to bypass US export restrictions, now faces uncertainty as it may still exceed bandwidth thresholds. NVIDIA has reportedly asked partners to halt shipments amid regulatory concerns, potentially leaving a gap for competitors in China's high-end GPU market.

RTX 5090D Faces Uncertainty in China Amid Export Crackdown


NVIDIA has [confirmed](https://nvidianews.nvidia.com/news/nvidia-blackwell-rtx-pro-workstations-servers-agentic-ai) the launch of the RTX Pro Blackwell series – a revolutionary generation of workstation and server GPUs. Designed to meet the needs of professional designers, data scientists, developers, and content creators, the lineup features a cutting-edge **RTX Pro 6000 Blackwell** GPU for workstations, plus other desktop and laptop variants and a data center version.

After early signs of the card [appeared in shipping logs](https://vast.ai/article/nvidia-rtx-pro-6000-x-rumors-blackwell-gpu-with-96gb-vram-spotted-in-shipping-log) a couple of months ago, we now have full details straight from the source – and this GPU is no minor upgrade. Built on NVIDIA's Blackwell architecture and featuring an incredible **96GB of GDDR7 memory** on a **512-bit bus**, the RTX Pro 6000 is set to redefine what's possible on a single professional workstation.

Here's what we now know about the RTX Pro 6000 Blackwell GPU.

### NVIDIA RTX Pro 6000: Specs That Mean Business

At the heart of the RTX Pro 6000 is the **GB202 die**, aligning with previous high-end Blackwell GPUs, like the RTX 5090\. But while the RTX 5090 is geared toward gamers and other enthusiasts, the RTX Pro 6000 goes all in for professional use.

We've certainly seen professional-grade GPUs with massive VRAM configurations before – there's the RTX 6000 Ada with 48GB of memory, for instance. However, the RTX Pro 6000 offers **double that amount**, with the faster speeds and efficiency of GDDR7 ECC memory. This makes it one of the most powerful workstation GPUs ever released.

And that's not all. On top of **24,064 CUDA cores** and **1,792 GB/s of memory bandwidth**, here's what else you get with this beast of a machine:

- **Streaming multiprocessors** with new neural shaders, delivering up to **1.5X** faster throughput for AI-augmented graphics workflows.
- **Fourth-gen RT cores** offering up to **2X** faster ray tracing performance for photorealism and complex 3D environments, with optimizations for **RTX Mega Geometry**.
- **Fifth-gen Tensor cores** capable of **4,000 AI TOPS**, supporting **FP4** precision and **DLSS 4 Multi Frame Generation** for cutting-edge AI graphics and accelerated model prototyping.
- **Ninth-gen NVENC** **& sixth-gen NVDEC**, improving encoding and decoding speed and quality for pro video work.
- **PCIe Gen 5 interface**, doubling bandwidth for faster CPU-GPU data transfer and smoother performance on heavy data workloads.
- **DisplayPort 2.1** support for ultra-high refresh rates – up to **4K at 480Hz** and **8K at 165Hz** – enabling multi-monitor setups with high color accuracy.
- **Multi-Instance GPU (MIG)**, allowing secure partitioning into up to **four GPU instances** for better workload isolation and resource allocation.

Not surprisingly, the RTX Pro 6000 does require a substantial **600 watts** of power, albeit only slightly more than the 575 watts of the RTX 5090\. But with that power comes the performance needed for agentic AI, immersive XR, large-scale simulation, and autonomous systems development – workloads where speed, memory, and throughput are non-negotiable.

(For more details and specs, check out the table at the bottom of this page\!)

So the only question that remains is: when can you get your hands on one?

### Availability and Rollout Timeline

The RTX Pro 6000 Workstation Edition is [expected](https://nvidianews.nvidia.com/news/nvidia-blackwell-rtx-pro-workstations-servers-agentic-ai) to **start shipping in May 2025** from manufacturers like BOXX, Dell, HP, Lambda, and Lenovo.

The server version of the RTX Pro 6000 is also on the way, with systems from Cisco, Dell, HP, Lenovo, and Supermicro expected to launch "soon," according to NVIDIA. Laptop variants, featuring **24GB of VRAM**, are expected to ship later in the year, from Dell, HP, Lenovo, and Razer.

NVIDIA hasn't announced official pricing yet, but estimates place the RTX Pro 6000 around $8500 – quite the investment. Fortunately, for those who need top-tier professional-grade GPU performance without the upfront cost, there are more accessible options.

### Access GPU Power Without the Overhead

Here at Vast.ai, we're excited to expand our GPU rental offerings as the **RTX Pro 6000 series** rolls out this year. In the meantime, you can tap into powerful GPUs like the **H100**, **H200**, and other high-performance options, all available on-demand through our platform, today.

With Vast, you can enjoy flexible, cost-effective access to serious compute power – with no long-term commitments and no hardware overhead – that's ready when you are.

### NVIDIA RTX Pro 6000: Key Specs

| Specification                    | RTX Pro 6000                                         |
| :------------------------------- | :--------------------------------------------------- |
| **Architecture**                 | Blackwell                                            |
| **GPU**                          | GB202                                                |
| **GPU Memory**                   | 96GB GDDR7 ECC                                       |
| **Memory Bandwidth**             | 1792 GB/s                                            |
| **Memory Interface**             | 512-bit                                              |
| **CUDA Cores**                   | 24,064                                               |
| **5th-Gen Tensor Cores**         | 752                                                  |
| **4th-Gen RT Cores**             | 188                                                  |
| **L2 Cache**                     | 128 MB                                               |
| **AI TOPS**                      | 4000                                                 |
| **Single-Precision Performance** | 125 TFLOPS                                           |
| **RT Core Performance**          | 380 TFLOPS                                           |
| **System Interface**             | PCIe 5.0 x16                                         |
| **Display Connectors**           | 4x DisplayPort 2.1b                                  |
| **Video Engines**                | 4x NVENC (9th Gen); 4x NVDEC (6th Gen)               |
| **MIG Support**                  | Up to 4x 24GB; or 2x 48GB; or 1x 96GB                |
| **Graphics APIs**                | DirectX 12, Shader Model 6.6, OpenGL 4.6, Vulkan 1.3 |
| **Compute APIs**                 | CUDA 12.8, OpenCL 3.0, DirectCompute                 |
| **Thermal Solution**             | Double-flow-through                                  |
| **Form Factor**                  | 5.4” H x 12” L, dual slot, extended height           |
| **Total Board Power**            | 600 watts                                            |


NVIDIA has confirmed the launch of the RTX Pro Blackwell series – a revolutionary generation of workstation and server GPUs. Designed to meet the needs of professional designers, data scientists, developers, and content creators, the lineup features a cutting-edge RTX Pro 6000 Blackwell GPU for workstations, plus other desktop and laptop variants and a data center version.

NVIDIA RTX Pro 6000 Blackwell: Redefining the Workstation GPU


## Introduction

Meta's Llama 4 is a breakthrough AI model combining state-of-the-art multimodal capabilities with the computational efficiency of mixture-of-experts (MoE) architecture. These models can process both text and images through an early fusion design that seamlessly integrates different modalities. Perhaps most impressively, they support context windows of up to 10 million tokens - dramatically expanding what's possible with large language models.

The Llama 4 family includes:

- **Llama 4 Scout**: 17B active parameters with 16 experts - efficient enough to run on modest GPU setups.

- **Llama 4 Maverick**: 17B active parameters but with 128 experts - offering enhanced capabilities while maintaining efficiency.

- **Llama 4 Behemoth**: 288B active parameters with 16 experts - Meta's most powerful model (not yet publicly released).

What makes these models special:

- **Mixture-of-Experts (MoE)**: Unlike traditional models that activate all parameters for every token, MoE models selectively activate only a fraction - dramatically improving computational efficiency.
- **Early Fusion**: Vision and text processing are integrated directly into the model backbone, enabling true multimodal reasoning.
- **10M Token Context Window**: Process entire books, codebases, or document collections in a single prompt - enabling comprehensive analysis impossible with earlier models.

In this guide, we'll deploy Llama 4 models on Vast.ai using three practical hardware configurations:

1. Llama 4 Scout on 8× H100 GPUs
2. Llama 4 Scout on 4× H100 GPUs
3. Llama 4 Maverick on 8× H200 GPUs

You'll learn exactly how to set up each configuration, deploy the models with the vLLM server, and interact with them through an OpenAI-compatible API. By the end, you'll have hands-on experience running some of the most advanced AI models available today.

## Installing Vast

Let's start by installing the Vast.ai command-line tools and setting up our API key:

```bash
pip install --upgrade vastai
```

```bash
# Here we will set our api key
export VAST_API_KEY="" #Your key here
vastai set api-key $VAST_API_KEY
```

## Running Llama 4 Scout on 8× H100s

### Choosing the Right Hardware

First, we need to search for GPUs on Vast.ai to run the Llama 4 Scout model on 8× H100 GPUs with a 200k token context window. This model requires specific hardware capabilities to run efficiently with vLLM's optimizations. Here are our requirements:

1. 8× H100 GPUs (80GB of VRAM each) to accommodate:

   - Llama 4 Scout model weights (17B active parameters with 16 experts)
   - KV Cache for handling the 200k token context window

2. A static IP address for:

   - Stable API endpoint hosting
   - Consistent client connections

3. At least one direct port that we can forward for:

   - vLLM's OpenAI-compatible API server
   - External access to the model endpoint
   - Secure request routing

4. At least 600GB of disk space to hold the model and other dependencies

Let's search for machines that meet these requirements:

```bash
vastai search offers " \
gpu_name = H100_NVL \
num_gpus = 8 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 600 \
rentable = true"
```

### Deploying the Server via Vast

We'll choose a machine from our search results and copy and paste the id below to set `INSTANCE_ID`.

We use `vastai create instance` to create an instance that:

1. Uses `vllm/vllm-openai:latest` docker image. This gives us an OpenAI-compatible server.
2. Forwards port `8000` to the outside of the container, which is the default OpenAI server port.
3. Uses `--model meta-llama/Llama-4-Scout-17B-16E-Instruct` to serve the Llama 4 Scout model.
4. Uses Llama 4-specific parameters:
   - `--tensor-parallel-size 8` to distribute inference across multiple GPUs.
   - `--max-model-len 200000` to set a 200k token context window.
   - `--override-generation-config='{"attn_temperature_tuning": true}'` for optimal text generation quality.
5. Uses `--disk 600` to ensure that we have 600GB of disk space for model weights and dependencies.

Note: Ensure that you fill in your huggingface token `HUGGING_FACE_HUB_TOKEN` to access the model. You'll need to accept the Llama 4 license terms on the model's huggingface page: https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct

```bash
export INSTANCE_ID=19319856 #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGING_FACE_HUB_TOKEN' --disk 600 --args --model meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 8 --max-model-len 200000 --override-generation-config='{"attn_temperature_tuning": true}'
```

### Get Instance IP Address and Port

Now, we need to get our IP address and port to call our model. We first need to wait for our machine to download the image and the model and start serving. This will take a few minutes. The logs will show you when it's done.

Next, go to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console and find the instance you just created.

At the top of the instance, there is a button with an IP address in it. Click this and a panel will show up of the ip address and the forwarded ports.
You should see something like:

```
Open Ports
XX.XX.XXX.XX:YYYY -> 8000/tcp
```

You will need the IP address (XX.XX.XXX.XX) and the port (YYYY) for the next step.

### Call our Model

#### Install OpenAI

To call our model we will install the OpenAI SDK.

```bash
pip install --upgrade openai
```

#### Download our Data

To show the power of our large context window, we will download The Great Gatsby from Project Gutenberg and ask Llama 4 to summarize it.

Here we download the text for the book:

```python
import requests

# Download the text file to a variable
url = "https://www.gutenberg.org/ebooks/64317.txt.utf-8"
text = requests.get(url).text

# Print the first 100 characters to confirm it worked
print(text[:200])
```

### Call Our Model

We will then use the OpenAI SDK. To do this, we need to set our `VAST_IP_ADDRESS` and `VAST_PORT` that we found above.

```python
from openai import OpenAI

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client with your server URL
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an actual API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

prompt = "please summarize this book in two sentences:" + text

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    messages=[
        {"role": "user", "content": prompt}
    ],

)
print(response.choices[0].message.content)
```

### Model Response

We see that our model was able to ingest a large amount of text and summarize our book.

```
Here is a two-sentence summary of The Great Gatsby:

Set in the roaring twenties, The Great Gatsby is a classic novel by F. Scott Fitzgerald that revolves around the mysterious millionaire Jay Gatsby and his obsession with winning back his lost love, Daisy Buchanan. Through the eyes of narrator Nick Carraway, the novel explores themes of wealth, class, love, and the corrupting influence of materialism in the excesses of the Jazz Age
```

## Running Llama 4 Scout on 4× H100s

After successfully deploying Scout on 8× H100s, we'll now try a more cost-effective setup using 4× H100s. This will reduce our context window capacity but allow us to save on GPU costs.

We'll start by searching for an instance that meets our specifications:

```bash
vastai search offers " \
num_gpus = 4 \
gpu_name = H100_SXM \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 600 \
rentable = true"
```

We'll choose a machine from our search and create our instance with 4 GPUs and a 100k token context window using `--tensor-parallel-size 4` and `--max-model-len 100000`:

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGING_FACE_HUB_TOKEN' --disk 600 --args --model meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 4 --max-model-len 100000 --override-generation-config='{"attn_temperature_tuning": true}'
```

Once our instance is running, we'll set the new `VAST_IP_ADDRESS` and `VAST_PORT` and call our instance like we did earlier:

```python
from openai import OpenAI

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client with your server URL
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an actual API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

prompt = "please summarize this book in two sentences:" + text

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    messages=[
        {"role": "user", "content": prompt}
    ],

)
print(response.choices[0].message.content)
```

### Model Response

```
Here is a two-sentence summary of The Great Gatsby:

Set in the roaring twenties, The Great Gatsby is a classic novel by F. Scott Fitzgerald that revolves around the mysterious millionaire Jay Gatsby and his obsession with winning back his lost love, Daisy Buchanan. Through the eyes of narrator Nick Carraway, the novel explores themes of love, greed, class, and the corrupting influence of wealth, ultimately leading to a tragic confrontation that exposes the dark underbelly of the American Dream.
```

## Running Llama 4 Maverick on 8× H200s

Finally, we'll upgrade to Llama 4's larger Maverick model with a 100k token context window on 8× H200 GPUs - a more powerful configuration that delivers enhanced capabilities.

To start, we'll search for an H200 machine with 8 GPUs:

```bash
vastai search offers " \
num_gpus = 8 \
gpu_name = H200 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 1000 \
rentable = true"
```

We'll find a machine from our search and create an instance with Llama 4 Maverick using 8 GPUs and a 100k token context window by setting `--disk 1000`, `--model meta-llama/Llama-4-Maverick-17B-128E-Instruct`, `--tensor-parallel-size 8` and `--max-model-len 100000`:

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGING_FACE_HUB_TOKEN' --disk 1000 --args --model meta-llama/Llama-4-Maverick-17B-128E-Instruct --tensor-parallel-size 8 --max-model-len 100000 --override-generation-config='{"attn_temperature_tuning": true}'
```

Finally, we'll set the new `VAST_IP_ADDRESS` and `VAST_PORT`, change our model to `"meta-llama/Llama-4-Maverick-17B-128E-Instruct"`, and call our instance:

```python
from openai import OpenAI

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client with your server URL
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an actual API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

prompt = "please summarize this book in two sentences:" + text

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct",
    messages=[
        {"role": "user", "content": prompt}
    ],

)
print(response.choices[0].message.content)
```

### Model Response

```
Here is a two-sentence summary of The Great Gatsby:

The novel is narrated by Nick Carraway, who becomes fascinated with his wealthy neighbor Jay Gatsby and becomes entangled in Gatsby's quest to win back his lost love, Daisy Buchanan, through a complex web of relationships and tragic events. Ultimately, Gatsby's dream is destroyed, and he is murdered by George Wilson, the husband of Myrtle Wilson, who was having an affair with Tom Buchanan, Daisy's husband, highlighting the corrupting influence of wealth and the elusiveness of the American Dream.
```

## Conclusion

In this post, we've demonstrated how to deploy Meta's Llama 4 models on Vast.ai using three specific hardware configurations: Llama 4 Scout on 8× H100 GPUs with a 200k token context window, Scout on 4× H100 GPUs with a 100k context window, and Llama 4 Maverick on 8× H200 GPUs with a 100k context window. We've shown that with the right GPU setup, you can easily deploy and interact with these models through an OpenAI-compatible API.

Now that you have a basic deployment working, here are some next steps you might consider:

- **Expand the context window**: While we demonstrated context windows of 100k-200k tokens, Llama 4 supports up to 10M tokens. Try increasing `--max-model-len` to experiment with longer documents, books, or codebases. Note: You will also need to increase the number of GPUs.

- **Explore multimodal capabilities**: The tutorial focused on text processing, but Llama 4 is natively multimodal. Try adapting the deployment to accept image inputs using vLLM's multimodal capabilities and the early fusion architecture.

By leveraging Vast.ai's flexible GPU infrastructure, you can cost-effectively experiment with these cutting-edge models and build applications that take advantage of their sophisticated reasoning capabilities.


Learn how to deploy and interact with Meta's Llama 4 models on Vast.ai using three specific hardware configurations: Llama 4 Scout on 8× H100 GPUs with a 200k token context window, Scout on 4× H100 GPUs with a 100k context window, and Llama 4 Maverick on 8× H200 GPUs with a 100k context window.

Running Llama 4 Models on Vast.ai


The H100 and H200 GPUs represent the top end of NVIDIA's current lineup for AI and high-performance computing (HPC). Both are designed for serious, large-scale workloads – from training foundation models to running complex scientific simulations.

At a glance, the two GPUs may appear similar, but for those working across enterprise environments or pushing the boundaries of exascale computing, the differences between them can have a major impact on performance and scalability.

Today, we're taking a look at how the H100 and H200 stack up against each other to help you decide which of these two heavyweights may be the better fit for your needs.

### NVIDIA H100: The Proven Performer

The H100 GPU made waves when it launched in March 2023\. Quickly adopted by big names like OpenAI, Meta, and Stability AI, it became the engine behind some of the most advanced generative AI systems in production today.

Built on NVIDIA's innovative Hopper architecture, the H100 was engineered for scale, handling everything from trillion-parameter language models to high-throughput inference. It introduced **fourth-gen Tensor Cores** and a dedicated **Transformer Engine**, designed to accelerate deep learning workloads with speed and FP8 precision.

The H100 boasts up to a **30X boost in performance** for large language models (LLMs) over its predecessor, the A100 GPU, according to NVIDIA. This massive improvement helped cement the H100 as the new standard for foundational model training, and to this day, it's still one of the very best GPUs on the market.

#### Some of the H100's more notable features include:

-   **80 GB of HBM3 memory**, with some models offering up to 94 GB.

-   **3.35 TB/s of memory bandwidth** to handle data-intensive workloads.

-   **Fourth-gen Tensor Cores** with superior throughput performance, along with an advanced Transformer Engine and FP64 and FP8 capabilities to reduce memory usage.

-   **PCIe Gen5 and fourth-gen NVLink** (with 900 GB/s of GPU-to-GPU interconnect), as well as NDR Quantum-2 InfiniBand networking for efficient scalability.

-   **Multi-Instance GPU (MIG) support** for secure partitioning into up to 7 isolated right-size instances that maximize quality of service for multi-tenant environments and parallel workloads.

If your projects involve massive datasets, large-scale simulations, or advanced AI training for foundational models, the H100 has proven itself as a reliable, high-performance solution. But with the arrival of the H200 in late 2024, NVIDIA has taken things a step further, introducing upgrades that push the envelope even more for memory-intensive and high-throughput workloads.

We ran a side-by-side comparison of inference performance on the **DeepSeek R1 Distill Llama 8B** model using `vLLM` with both NVIDIA H100 and H200 GPUs on Vast.ai.

#### **Benchmark Configuration**

-   **Model:** `deepseek-ai/DeepSeek-R1-Distill-Llama-8B`
-   **Requests:** 5,000
-   **Batch:** 512
-   **Input Tokens:** 16
-   **Output Tokens:** 128
-   **Backend:** `vLLM`

---

### **Results**

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Metric
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H100
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H200
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Benchmark Duration
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                45.28 s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                40.02 s
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Request Throughput
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                110.42 req/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                124.93 req/s
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Output Throughput
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                13,650 tok/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                15,435 tok/s
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Total Throughput
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                15,416 tok/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                17,434 tok/s
            </td>
        </tr>
    </tbody>
</table>

### NVIDIA H200: The Supercharged Successor

The H200 builds directly on the foundation laid by the H100, bringing meaningful improvements that target one of the biggest bottlenecks in AI and HPC workloads: memory.

While it shares the same Hopper architecture as the H100 and has similar core compute specs, the H200 significantly increases both memory capacity and bandwidth, two factors that become critical when operating at enterprise scale or pushing into exascale territory.

Equipped with **141 GB of HBM3e memory** and **4.8 TB/s of bandwidth**, the H200 nearly doubles the memory of the H100 and delivers about **1.4X faster** data access. Early benchmarks reflect that improvement. In MLPerf testing using Llama 2 70B, the H200 achieved over 31,000 tokens per second – about **45% faster than the H100**.

<img
    src="/uploads/h100vsh200_2.png"
    alt="Image credit: NVIDIA"
    style={{ width: "100%" }}
/>

The H200 retains all of the architectural strengths of the H100, including **fourth-gen NVLink**, **PCIe Gen5**, **MIG support**, and full **Tensor Core and Transformer Engine capabilities**. But for developers and organizations looking to squeeze out more performance without overhauling their infrastructure, the H200 offers a straightforward upgrade path with [measurable gains](https://resources.nvidia.com/en-us-data-center-overview/hpc-datasheet-sc23-h200).

#### To recap, here are a few advantages of the H200:

-   **141 GB of HBM3e memory at 4.8 TB/s** – nearly double the capacity of the H100, with 1.4X more memory bandwidth.

-   **Up to 1.9X the inference performance** of the H100 when handling LLMs, with more efficient processing per token and per inference cycle.

-   **50% reduced energy use and total cost of ownership** (TCO) due to improved thermal management and energy efficiency.

-   **Drop-in compatibility with existing Hopper-based systems**, enabling accelerated performance gains in H100-optimized infrastructure with minimal or no additional changes.

In short, the H200 represents a meaningful leap beyond the H100 in memory capacity, bandwidth, and throughput, bringing a new level of performance to AI and HPC workloads. That comes at a premium price, of course. But if you're weighing the tradeoffs between the two GPUs in terms of specs alone, let's take a side-by-side look at how they compare.

### H100 vs. H200: Key Specs Compared

For reference, here's a quick outline of the various features and specs of the NVIDIA H100 and H200:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Specification
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H100
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H200
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                GPU Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">Hopper</td>
            <td class="border border-gray-300 px-4 py-2 text-center">Hopper</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                GPU Memory
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                80 GB HBM3
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                141 GB HBM3e
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                GPU Memory Bandwidth
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3.35 TB/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                4.8 TB/s
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP64 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                33.5 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                33.5 TFLOPS
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">FP32</td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                67 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                67 TFLOPS
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                TF32 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                989 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                989 TFLOPS
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP16 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1,979 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1,979 TFLOPS
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP8 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TFLOPS
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                INT8 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TOPS
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Decoders
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                7 NVDEC, 7 NVJPEG
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                7 NVDEC, 7 NVJPEG
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                L2 Cache
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">50 MB</td>
            <td class="border border-gray-300 px-4 py-2 text-center">50 MB</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Max. Thermal Design Power
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 700 W
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 1000 W
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Multi-Instance GPUs
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 7 MIGs @ 10GB each
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 7 MIGs @ 18GB each
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Form Factor
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">SXM</td>
            <td class="border border-gray-300 px-4 py-2 text-center">SXM</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Interconnect
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                PCIe 5.0 x16
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                PCIe 5.0 x16
            </td>
        </tr>
    </tbody>
</table>

Ultimately, choosing the right GPU largely depends on your workload demands and performance priorities.

### Use Cases: Which GPU Should You Choose?

While both the H100 and H200 excel in AI and HPC environments, they serve slightly different needs.

#### You should consider the H100 if you prefer:

-   **A proven, high-performance GPU** for training medium to large models and handling intensive AI or HPC workloads.
-   **Use cases involving compute-bound workloads**, where total memory capacity is not as critical.
-   **A time-tested solution** already in widespread production use, balancing performance with lower power and cost.

#### Whereas the H200 may be a better fit if you need:

-   **Greater memory capacity and bandwidth** for running _even larger_ models, handling memory-bound workloads, or scaling LLM inference on the enterprise level.
-   **Higher throughput**, especially for tasks involving extended context windows, multi-modal AI, or very large batch sizes.
-   **A forward-looking upgrade path** with improved energy efficiency and lower TCO, thanks to higher performance per watt.

\------------------------------

At **Vast.ai**, we understand that choosing a GPU isn't only about specs. Often, the decision comes down to access and affordability, as well – especially when we're talking about some of the most powerful and expensive GPUs in existence. (Speaking of, check out our previous posts comparing the [H100 vs. A100](https://vast.ai/article/H100-vs-A100-Comparing-two-Powerhouse-GPUs), and the [H100 vs. L40S](https://vast.ai/article/nvidia-h100-vs-l40s-power-meets-versatility).)

That's why our cloud GPU rental platform is built to offer high-performance compute at a fraction of the typical cost. Vast gives you flexible, cost-effective access to H100s, H200s, and more.

With options for on-demand pricing and budget-friendly spot instances, you can scale up as needed without overspending. On average, **Vast users save 5- 6X on GPU compute** compared to traditional cloud providers.

Explore our platform today and get the power you need, when you need it – with no upfront investment required.


The H100 and H200 GPUs represent the top end of NVIDIA's current lineup for AI and high-performance computing (HPC). Both are designed for serious, large-scale workloads – from training foundation models to running complex scientific simulations.

NVIDIA H100 vs. H200: Two Hopper-based Heavyweights


A new NVIDIA Blackwell GPU with 96GB of VRAM? If rumors are to be believed, that could be exactly what's coming. 

A recent shipping manifest [revealed](https://wccftech.com/nvidia-rtx-pro-6000-x-blackwell-leak-96-gb-gddr7-600w/) an entry for the "**RTX PRO 6000 X Blackwell**," evidently being sent to India "for testing purposes." At least two different shippers have sent the GPUs for testing.

This isn't your typical consumer-grade gaming card, however. The as-yet-unannounced Blackwell-based GPU appears to be aimed at AI and data-intensive workloads. Here's what we know so far about this new entry in NVIDIA's professional lineup.

### NVIDIA RTX Pro 6000 X: What We Know So Far

With the impressive [RTX 5090](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know) expected to remain at the top of the GeForce 50-series lineup, the professional segment is where we'll likely see ultra-high-end specs that push the limits on VRAM – and this latest shipping log discovery suggests that will be the case. 

According to reports, as mentioned, the RTX Pro 6000 X features a staggering **96GB of GDDR7 memory** on a 512-bit memory bus. That's significantly higher than even the most powerful gaming cards. For example, the RTX 5090 has 32GB of GDDR7 memory – which is more than enough for most purposes – but higher VRAM is obviously beneficial for workloads that require handling massive datasets, such as foundation AI model training and large-scale simulations.

The RTX Pro 6000 X also features a **GB202 die** and **600 watts** of total board power (TBP). This aligns with previous high-end Blackwell architecture designs, like the RTX 5090, which also has the GB202 die and comes in at 575 watts of TBP.

Interestingly, the discovery of the RTX Pro 6000 X on the shipping manifest follows an [earlier sighting](https://wccftech.com/nvidia-rtx-4090-with-96gb-vram-reportedly-exists/) of a 96GB RTX 4090 GPU – perhaps an engineering sample. While NVIDIA hasn't made any announcements, these repeated sightings nonetheless suggest a broader push for high-VRAM GPUs.

### Comparison to Existing RTX Workstation GPUs

We've seen professional-grade GPUs with massive VRAM configurations before, such as the RTX 6000 Ada with 48GB of memory, as well as the A100/A800 series. However, the RTX Pro 6000 X's 96GB of GDDR7 memory would make it one of the highest-capacity workstation GPUs to date. 

Assuming we see the RTX Pro 6000 X emerge, it would set a new benchmark for professional GPUs, doubling the memory of the RTX 6000 Ada while offering the faster speeds and efficiency of GDDR7 memory.

Plus, unlike the A100 and A800 series, which are designed primarily for data centers and enterprise AI workloads, the RTX Pro 6000 X could serve as a more flexible solution. Its ultra-high VRAM would allow professionals to work with larger datasets without necessarily relying on multi-GPU setups. 

### Looking Ahead

NVIDIA hasn't officially announced the RTX Pro 6000 X, so all information should be taken with a grain of salt, but the repeated mentions of this new GPU in shipping logs suggest that an announcement may be imminent. 

For Vast.ai users, this could mean more powerful GPU rental options in the near future. Stay tuned for more developments\! In the meantime, check out our current selection of high-performance GPUs available on Vast.ai for flexible and cost-effective access to cutting-edge hardware today.  

NVIDIA’s rumored RTX Pro 6000 X packs 96GB of GDDR7 VRAM, a GB202 die, and 600W TBP—ideal for AI, data science, and high-end workstation workloads.

Posts about: Nvidia

NVIDIA RTX 4090 vs. A100: Two Powerhouses, Two Purposes

RTX 5090D Faces Uncertainty in China Amid Export Crackdown

NVIDIA RTX Pro 6000 Blackwell: Redefining the Workstation GPU

Running Llama 4 Models on Vast.ai

NVIDIA H100 vs. H200: Two Hopper-based Heavyweights

NVIDIA RTX Pro 6000 X Rumors: Blackwell GPU with 96GB VRAM Spotted in Shipping Log