
The NVIDIA GeForce RTX 5090 launched in January 2025 – and over a year later, it's still surprisingly difficult to get your hands on one.

Despite retail prices officially starting at $1,999 MSRP, limited availability and strong demand from both gamers and AI developers means stock sells out quickly and real-world pricing has pushed higher.

As NVIDIA's most powerful consumer GPU, the RTX 5090 is undoubtedly an impressive machine. Is it worth the steep price tag, though? That depends on more than just raw power – let's take a look.

### RTX 5090: The Consumer Powerhouse

For starters, here are some of the official specs for the RTX 5090:

-   **Architecture:** Powered by NVIDIA's cutting-edge Blackwell architecture.
-   **Gaming Chip:** Featuring the next-gen GB202 gaming chip.
-   **Memory:** 32GB of GDDR7 memory with a blazing-fast 1,792GB/sec bandwidth, supported by a super-wide 512-bit memory interface for even faster data transfer.
-   **CUDA Cores:** 21,760 cores for high-performance graphics workloads, complementing the 680 Tensor cores that accelerate AI inference and training.
-   **L2 Cache:** Expanded to 98MB, up from 73MB on the RTX 4090, improving memory efficiency and reducing latency.
-   **Display Support:** PCIe Gen 5 with DisplayPort 2.1b connectors supporting up to 8K at 165Hz.
-   **Power Requirements:** 575W total graphics power (TGP), with a 1000W power supply recommended for optimal performance.

Another bonus: the RTX 5090 has a redesigned **Founders Edition** with a three-slot design and a dual flow-through fan configuration – so it's more compatible with tighter builds, a remarkable shift from its predecessors.

For reference, here's a quick overview of the RTX 5090:

<table>
    <thead>
        <tr>
            <th>Spec</th>
            <th>RTX 5090</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><strong>CUDA Cores</strong></td>
            <td>21,760</td>
        </tr>
        <tr>
            <td><strong>Ray Tracing Cores</strong></td>
            <td>170 (4th-gen)</td>
        </tr>
        <tr>
            <td><strong>Tensor Cores</strong></td>
            <td>680 (5th-gen)</td>
        </tr>
        <tr>
            <td><strong>Base Clock</strong></td>
            <td>2,017 MHz</td>
        </tr>
        <tr>
            <td><strong>Boost Clock</strong></td>
            <td>2,407 MHz</td>
        </tr>
        <tr>
            <td><strong>L2 Cache</strong></td>
            <td>98,304KB</td>
        </tr>
        <tr>
            <td><strong>ROPs</strong></td>
            <td>176</td>
        </tr>
        <tr>
            <td><strong>VRAM</strong></td>
            <td>32GB 28Gbps GDDR7</td>
        </tr>
        <tr>
            <td><strong>VRAM Interface</strong></td>
            <td>512-bit</td>
        </tr>
        <tr>
            <td><strong>Memory Bandwidth</strong></td>
            <td>1,792 GB/s</td>
        </tr>
        <tr>
            <td><strong>Interface</strong></td>
            <td>16x PCIe 5.0</td>
        </tr>
        <tr>
            <td><strong>Power Connectors</strong></td>
            <td>1 x 16-pin (PCIe 5.0 / ATX 3.0) native connector; includes 4 x 8-pin adapter</td>
        </tr>
        <tr>
            <td><strong>Total Graphics Power (TGP)</strong></td>
            <td>575W</td>
        </tr>
    </tbody>
</table>

(For more specifications – and to see how the RTX 5090 stacks up against its younger siblings in the RTX 50-series lineup as well as the RTX 4090 – check out the [chart at the bottom of this page](#additional-specifications).)

#### RTX 5090: AI and Professional Workloads

The RTX 5090 is well-suited for high-performance AI and professional workloads. Its **680 Tensor cores** and **32GB of GDDR7** memory efficiently accelerate deep learning, 3D rendering, and other compute-intensive tasks. With **98MB of L2 cache** and a **512-bit memory interface**, the RTX 5090 delivers faster data access and reduced latency – crucial for AI training, scientific computing, and content creation.

The **5th-gen Tensor cores** boost AI inference efficiency in frameworks like TensorFlow and PyTorch, while **4th-gen RT cores** enhance real-time ray tracing for animation, visual effects, CAD work, and more.

For demanding AI and video workloads, the RTX 5090 more than meets the challenge. In Procyon's AI XL (FP16) test, the RTX 5090 was **40% faster** than the RTX 4090, and in PugetBench's DaVinci Resolve video processing test, it was **12% faster**.

#### Next-Gen Gaming Features

The biggest leap in technology for this GPU comes from DLSS 4's new **Multi Frame Generation**, an AI-driven frame interpolation technique exclusive to the RTX 50 series. It uses AI to generate up to three additional frames between traditionally rendered ones, significantly boosting frame rates with minimal visual artifacts.

Further, the RTX 5090 introduces a range of AI-driven enhancements in what's called **neural rendering**, designed to push in-game realism to new levels. **RTX Neural Shaders** improve texture compression and deliver film-quality lighting and shading in real time, while **RTX Neural Faces** uses generative AI to create lifelike skin, hair, and facial details.

Building on the power of DLSS 4, these neural rendering techniques integrate directly into the graphics pipeline, leveraging NVIDIA’s Tensor cores to enhance visual fidelity. As game support continues to expand, these technologies demonstrate the potential for deeper shadows, realistic translucency in materials, and an overall richer environment.

Another breakthrough is **Mega Geometry**, a feature that uses RT cores to significantly boost the number of triangles in ray-traced scenes. This allows game engines to maintain full geometric detail without sacrificing performance, resulting in more realistic object depth, shadows, and fine details.

The above features position the RTX 5090 as a powerhouse for next-generation gaming visuals... but there's a catch.

##### Caveat: Performance Claims vs. Real-World Results

The way that NVIDIA presents its performance numbers can be misleading. The company leans heavily on Multi Frame Generation-boosted FPS (frames per second) to compare these new GPUs against older ones.

Frame Generation and Multi Frame Generation (MFG) are powerful tools, but they aren't universal fixes. Their impact varies by game – some titles see massive FPS gains with little visual trade-off, while others may introduce artifacts or latency.

In addition, **not all games will support these features**, as DLSS adoption depends on individual developers. A GPU's raw performance still matters. MFG-inflated benchmarks can make performance gains seem more significant, but Frame Generation still needs a strong base frame rate to be effective. In CPU-limited games, or titles that don't support DLSS 4, the real-world performance gap may be far smaller.

Ultimately, while the RTX 5090 excels at AI-driven rendering, **real-world results will depend on game support and how well these features are implemented.**

### Worldwide RAM Shortage and Price Increases

Due to explosive growth in AI data centers and tech giants continuing to invest heavily in AI accelerators, there's an ongoing worldwide RAM shortage that shows no signs of slowing down. Some forecasts <a href="https://www.theverge.com/ai-artificial-intelligence/914672/the-ram-shortage-could-last-years" target="_blank" rel="noopener noreferrer">predict</a> that shortages could last until 2030.

It's no surprise that high demand and limited supply lead to rising prices. With the RTX 5090, though, there may be another factor at play. DeepSeek's rapid rise in popularity has sparked a huge <a href="https://www.theinformation.com/articles/deepseek-spurs-black-market-demand-for-nvidia-gaming-chips-in-china" target="_blank" rel="noopener noreferrer">surge</a> in demand for NVIDIA's gaming GPUs in China, as these cards can be repurposed to run DeepSeek's AI models – and provide a workaround for U.S. export restrictions on advanced GPUs.

See our guide on [how to run DeepSeek R1 on a GPU cloud](https://vast.ai/article/deepseek-r1-open-source-disruptor-or-overhyped-upstart).

With a global shortage of RAM and both gamers and AI developers vying for the same hardware, the strain on supply could push RTX 5090 prices even higher in the near future. Most buyers currently face inflated market prices ranging from $2,500 to over $4,000. And with NVIDIA set to <a href="https://wccftech.com/nvidia-passes-another-300-hit-to-rtx-5090-board-partners-due-to-vram-shortage/" target="_blank" rel="noopener noreferrer">increase prices</a> further on account of VRAM sourcing costs, those numbers could skyrocket to over $5,000.

### Final Thoughts: The Power of the RTX 5090 – Without the Hassle

The RTX 5090 remains the most powerful consumer GPU available today. For cutting-edge 4K gaming with DLSS 4 and Multi Frame Generation, as well as demanding AI and professional workloads, it delivers unmatched performance in a relatively accessible package.

At the same time, it may only be worth purchasing if you really want the best of the best and you're willing to shell out for it. The high price tag makes it a significant investment for the average user.

In the meantime, why wait? Skip the delays and high costs – [rent an RTX 5090](https://cloud.vast.ai/?gpu_option=RTX%205090) on Vast.ai for instant access to top-tier performance. Given its limited availability and its 575-watt power draw, our affordable rental options are the most practical way to access the RTX 5090's full potential without the upfront investment.

[Try the 5090 GPU](https://cloud.vast.ai/?gpu_option=RTX%205090) for yourself today.

<span id="additional-specifications"></span>

#### Additional Specifications:

<div className="article-table-scroll">
<table className="w-full border-collapse border border-gray-300">
    <thead>
        <tr className="bg-gray-100">
            <th className="border border-gray-300 px-4 py-2 text-left font-semibold">
                Spec
            </th>
            <th className="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 5090
            </th>
            <th className="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 5080
            </th>
            <th className="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 5070 Ti
            </th>
            <th className="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 5070
            </th>
            <th className="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 4090
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">GB202</td>
            <td className="border border-gray-300 px-4 py-2 text-center">GB203</td>
            <td className="border border-gray-300 px-4 py-2 text-center">GB203</td>
            <td className="border border-gray-300 px-4 py-2 text-center">GB205</td>
            <td className="border border-gray-300 px-4 py-2 text-center">AD102</td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">VRAM</td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                32GB GDDR7
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                16GB GDDR7
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                16GB GDDR7
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                12GB GDDR7
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                24GB GDDR6X
            </td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                VRAM Speed
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">28Gbps</td>
            <td className="border border-gray-300 px-4 py-2 text-center">30Gbps</td>
            <td className="border border-gray-300 px-4 py-2 text-center">28Gbps</td>
            <td className="border border-gray-300 px-4 py-2 text-center">28Gbps</td>
            <td className="border border-gray-300 px-4 py-2 text-center">21Gbps</td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                VRAM Bus Width
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">512</td>
            <td className="border border-gray-300 px-4 py-2 text-center">256</td>
            <td className="border border-gray-300 px-4 py-2 text-center">256</td>
            <td className="border border-gray-300 px-4 py-2 text-center">192</td>
            <td className="border border-gray-300 px-4 py-2 text-center">384</td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Bandwidth
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                1792GB/s
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                960GB/s
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                896GB/s
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                672GB/s
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                1.01TB/s
            </td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Streaming Multiprocessors (SMs)
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">170</td>
            <td className="border border-gray-300 px-4 py-2 text-center">84</td>
            <td className="border border-gray-300 px-4 py-2 text-center">70</td>
            <td className="border border-gray-300 px-4 py-2 text-center">48</td>
            <td className="border border-gray-300 px-4 py-2 text-center"></td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Cores
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">680</td>
            <td className="border border-gray-300 px-4 py-2 text-center">336</td>
            <td className="border border-gray-300 px-4 py-2 text-center">280</td>
            <td className="border border-gray-300 px-4 py-2 text-center">192</td>
            <td className="border border-gray-300 px-4 py-2 text-center">512</td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Ray Tracing Cores
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">170</td>
            <td className="border border-gray-300 px-4 py-2 text-center">84</td>
            <td className="border border-gray-300 px-4 py-2 text-center">70</td>
            <td className="border border-gray-300 px-4 py-2 text-center">48</td>
            <td className="border border-gray-300 px-4 py-2 text-center">128</td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Boost Clock
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                2407 MHz
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                2617 MHz
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                2452 MHz
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                2512 MHz
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                2520 MHz
            </td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Total Graphics Power (TGP)
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">575W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">360W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">300W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">250W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">450W</td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Recommended Power Supply
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">1000W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">850W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">750W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">650W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">850W</td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Launch Date
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                Jan. 30, 2025
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                Jan. 30, 2025
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                Feb. 20, 2025
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                Mar. 5, 2025
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                Sept. 20, 2022
            </td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Launch Price
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">$1,999</td>
            <td className="border border-gray-300 px-4 py-2 text-center">$999</td>
            <td className="border border-gray-300 px-4 py-2 text-center">$749</td>
            <td className="border border-gray-300 px-4 py-2 text-center">$549</td>
            <td className="border border-gray-300 px-4 py-2 text-center">$1,599</td>
        </tr>
    </tbody>
</table>
</div>

Post originally written March 7, 2025, updated May 21, 2026.


NVIDIA’s RTX 5090 remains hard to find as AI and gaming demand push prices higher. Review the specs, performance features, and rental options on Vast.ai.

NVIDIA GeForce RTX 5090: Everything You Need to Know


120,000+ AI developers are actively searching for compute. Your hardware is what they're looking for.

Whether you're running a single RTX 4090 in your home office or managing a datacenter full of H100s, your idle compute could be generating revenue for you right now.

So, how much money can you actually make renting out GPUs on Vast.ai? The short answer: it varies.

## What Hosts Are Earning on Vast.ai

Your earnings depend on your specific hardware, uptime, reliability, rental type, and how your prices compare to similar listings. That said, as a host, here's a snapshot of what you might expect:

- **Consumer GPUs** like the RTX 5090 typically earn $0.30-$0.60 per GPU-hour, depending on your rental type and your reliability score. This score is a measure of uptime and connection stability. A four-GPU 5090 rig running at 80% utilization can generate $700-$1,400 per month. Use the [host earnings calculator](https://vast.ai/hosting/calculator) to estimate your specific hardware.
- **Datacenter hardware** commands much higher rates. H100s and H200s bring in $2.15-$4.00+ per GPU-hour. If you operate a GPU farm, you can turn dedicated infrastructure into a profitable compute business. Vast.ai even offers financing and will match you with enterprise clients looking for reserved compute, so you can presell capacity and have paying customers from day one.

For current market rates across the Vast.ai platform, check the public [pricing page](https://vast.ai/pricing). Live GPU prices are typically about 25% above what hosts earn.

Once you log in and register at least one machine with Vast.ai, your host dashboard will show detailed market stats with occupancy rates and more, allowing you to optimize your pricing strategy and better estimate potential earnings.

## How Vast.ai Works for Hosts

From hobbyists at home to datacenters and GPU farms, Vast.ai supports every scale. This flexibility is what makes the platform work for hundreds of providers already earning steady revenue.

Many GPU rental platforms lock you into contracts and dictate pricing structures. Some only work with datacenters, requiring certifications and minimum machine counts before you can even list.

Vast.ai is different. You set your rates, select your rental terms, and decide exactly how much bandwidth and storage to offer, whether you're running one GPU or one hundred. *Note: hosting requires a separate account from your client account; a single account cannot be used for both.*

To accommodate your needs, Vast.ai supports three rental models, each designed for different workloads and revenue strategies.

### 1. On-Demand Instances: Premium Pricing for Guaranteed Access

On-demand is your highest-earning option. Set a fixed hourly rate and an offer end date. When a client rents your machine, those terms lock into a rental contract that cannot be changed -- not by modifying the offer, not by unlisting the machine, not by any other action. Once a rental contract is created, its terms -- price, end date, hardware specs -- are fixed. Unlisting your machine prevents new rentals but does not affect any active contracts. You must keep the machine available through every contract's rental end date.

This option attracts clients who are willing to pay a premium for reliability and guaranteed access, but you don't have to pick just one rental type. You can maximize utilization by offering interruptible capacity alongside on-demand instances.

### 2. Interruptible Instances: Lower Rates, Higher Flexibility

Interruptibles work in a bidding system. As the host, you set a minimum interruptible price for a client to rent your machine. Clients set a bid price for their instance; the current highest bid is the instance that runs, with lower bids paused. On-demand rentals always take priority over all interruptible instances regardless of bid price. Higher bids instantly preempt lower ones, and on-demand rentals always take priority.

Interruptible instances appeal to clients who are willing to trade guaranteed uptime for 50-80% cost savings. You'll earn less per hour with interruptible instances vs. on-demand, but they fill gaps in time that would otherwise generate nothing, so most hosts offer both.

### 3. Reserved Instances: Long-Term Contracts at Discounted Rates

Reserved instances let clients secure long-term rentals at a reduced rate. If you choose to offer reserved instances, you set your maximum possible discount off your normal on-demand price. That discount scales with rental duration -- longer commitments unlock larger discounts, up to your set maximum. You can also set this value to zero to opt out of reserved pricing entirely.

The reserved model works well for predictable, stable revenue if you're willing to trade some margin for higher occupancy. Customers commit upfront to weeks or months of utilization. That means you spend less time managing short-term rentals.

## How to Stand Out as a Host and Maximize Revenue

Once you're listed, a number of strategies can help you stand out and make more money:

- **Price competitively at launch.** New hosts often list their hardware at slightly below market rates to attract initial rentals and build a good reliability score faster. You can always adjust pricing later.
- **Offer reserved instance discounts.** Long-term contracts at discounted rates encourage more stable utilization and recurring revenue.
- **Keep machines online.** Strong uptime maintains your reliability score, which reflects connection stability. Note that reliability and verification are tracked independently; reliability drops don't affect your verification status. To avoid disruption, plan maintenance around active rental contracts. For unplanned maintenance, use the CLI's `schedule maint` command, which notifies affected clients to save their work.
- **Get and stay verified.** Vast.ai's automated verification process evaluates your machine's network stability, operational health, and performance. Verification runs on a randomized schedule. High-end hardware like H100s and 8x4090 rigs are typically tested fastest. Verified machines rank higher in search results and attract more rentals.
- **Apply for data center certification.** If you operate professionally managed infrastructure, you can become a [Certified Data Center Partner](https://docs.vast.ai/host/datacenter-status), giving you the blue trust label, priority placement, [Secure Cloud](https://vast.ai/compliance) classification for enterprise compliance, and increased customer traffic.

## Getting Started as a Host on Vast.ai

Becoming a host on Vast.ai is more straightforward than you might think. Setup complexity varies by hardware. The [Host Setup guide](https://cloud.vast.ai/host/setup/) walks you through Ubuntu installation, NVIDIA drivers, network port configuration, and other requirements. It can take as little as an hour.

But you don't need to figure everything out alone. Vast.ai maintains active host-only [Discord](https://discord.gg/vast) channels where providers share troubleshooting advice, setup guidance, and optimization tips. Get peer support from operators who've already solved the problems you might encounter.

Once your machines are online, hosting becomes less hands-on. Simply maintain your hardware and let the earnings roll in. Or you can take a more active approach and gradually optimize pricing, experiment with different rental models, and expand into multi-GPU systems or dedicated GPU infrastructure over time.

The choice is yours.

## The Bottom Line

On Vast.ai, hosts get a level of operational control and pricing transparency that's rare in the GPU cloud market. Whether you're hosting one GPU or an entire cluster, the platform is designed to let you monetize and grow on your own terms.

Here's what that means for you: 120,000+ developers are actively searching for compute on Vast.ai. That demand continues to grow, and it's waiting for your hardware. The only question is whether you're ready to start earning.

Want to put your hardware to work? Check [current GPU Pricing Rates](https://vast.ai/pricing), explore the [Host Setup guide](https://cloud.vast.ai/host/setup/), or jump straight in and list your first machine today.


Learn how much you can earn renting GPUs on Vast.ai - from RTX 4090s to H100s. Compare rental types, pricing strategies, and how to maximize your hosting revenue.

How Much Money Can You Earn Renting Out Your GPU on Vast.ai?


As AI demand grows, models are expected to perform increasingly complex tasks with pinpoint accuracy, often leaving founders and engineers scratching their heads. Hard-pressed to find ways to improve their systems, fine-tuning provides a novel solution. With tweaks and training, a model can be taught to produce high accuracy, consistent responses for all manner of specialized tasks. This is where tools such as Unsloth come in.

## Why use Unsloth Studio?

Unsloth, an open-source, code-driven framework for training and running AI models, released its no-code/low-code web UI, [Unsloth Studio](https://vast.ai/article/unsloth-studio-now-available-on-vast-ai), in early 2026. It's a convenient solution that makes fine-tuning a breeze, simplifying training, inference, data, and deployment.

Unsloth Studio currently supports over 500 models, training them 2x faster than traditional fine-tuning methods while requiring 70% less VRAM, all with no accuracy loss. It also runs, trains, and exports GGUF and safetensors models locally on Mac, Windows, and Linux. More efficient VRAM usage, faster training speeds, and local model support mean reduced computational needs and, as a result, cloud costs.

## Getting started with Unsloth Studio

The general process required to run and train your models in Unsloth Studio is straightforward:

1. Install Unsloth Studio
2. Choose a base model
3. Create or upload a dataset
4. Configure fine-tuning settings
5. Start training
6. Test your fine-tuned model
7. Export the model

For your convenience, we've built a [comprehensive guide](https://docs.vast.ai/examples/ai-ml-frameworks/unsloth-studio) that includes all of the information you'll need to get Unsloth Studio set up on your Vast.ai GPU instance. Be sure to use our pre-configured [Unsloth Studio template](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Unsloth%20Studio) in conjunction with the guide to make setup even easier.

## Fine-tuning at a glance

### The basics

Fine-tuning uses datasets to improve pre-trained models in key ways:

- **Update and learn:** Incorporate and learn domain-specific knowledge.
- **Customize behavior:** Engineers can adjust AI model tone, personality, or response style.
- **Tailor models for specific tasks:** Enhance accuracy and relevance for specialized workflows.

When compared to Retrieval-Augmented Generation (RAG), fine-tuning replicates all of RAG's capabilities, but not vice versa. There is a common misconception that fine-tuning doesn't perform as well as RAG and that it can't make a model learn new knowledge. This is untrue. A specialized model can train with fine-tuning and reinforcement learning (RL) while RAG only changes what the model sees at inference time.

### How does fine-tuning achieve memory savings?

Unsloth Studio adds two thin matrices to model weights and optimizes those, rather than changing all weights. This means that only a small fraction of weights, often well under 5%, depending on the LoRA rank and which layers are targeted, have to be optimized and quantization from 16-bit (LoRA) to 4-bit (QLoRA) saves 75% memory.

<div className="article-image-light-frame">
  <img
    src="/uploads/unsloth-studio-lora-memory-savings.png"
    alt="Diagram showing a model weight block and low-rank adapter matrices A and B"
  />
</div>

Image source: [Unsloth fine-tuning guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide)

### Choosing your model and method

When starting, it is recommended to use instruct models which require less data than base models. They also allow for direct fine-tuning using convenient conversational chat templates (ChatML, ShareGPT, etc.).

Regarding model training, you will need to choose between multiple methods:

- **Reinforcement learning (RL):** This method is most advantageous for training models to excel at specific behaviors using an environment and reward function instead of labeled data.
- **LoRA:** A parameter efficient training technique that trains a small set of low-rank adapter weights with 16-bit precision.
- **QLoRA:** This method augments LoRA with 4-bit precision to more efficiently handle large models.
- **Fine-tuning:** The Unsloth framework supports full fine-tuning (FFT) and pretraining, though both demand substantially more resources. If configured correctly, LoRA can match the gains of FFT.

Unsloth recommends QLoRA to start, as it is both accessible and highly effective at training models.

### Creating your datasets

Formatted and organized datasets are crucial for model fine-tuning success. Datasets should generally contain two columns, question and answer, and the data must be able to be tokenized.

Data can be generated synthetically and structured using an LLM. Unsloth Studio also provides a free Synthetic Dataset notebook that can parse documents, including PDFs and videos, generate QA pairs, and auto-clean data.

Generally, you should build a well-structured dataset rather than dumping data alone. This ensures learning, understanding, and response accuracy. Keep in mind, though, that if you're fine-tuning for code, dumping all of your code data can indeed bring significant performance improvements. Be sure to carefully consider your use case when building your fine-tuning strategy.

### AI training hyperparameters

Hyperparameters are essentially the settings when tuning AI models. They control how Low-Rank Adaptation fine-tunes LLMs and can be quite involved. Therefore, Unsloth provides a comprehensive [guide to hyperparameters](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) for your convenience.

The key hyperparameter choices one needs to consider are as follows:

- **Learning rate:** Controls how aggressively the model adjusts its weights during training.
- **Epochs:** Specifies how many complete passes the model makes through the training dataset.
- **LoRA or QLoRA:** LoRA fine-tunes models using 16-bit precision, while QLoRA uses 4-bit quantization to reduce VRAM usage and hardware requirements.

## Ready to start improving your models?

We hope that this fine-tuning breakdown has given you the confidence to train and customize your AI models with Unsloth Studio. At Vast.ai, our mission goes beyond providing high-performance cloud GPU infrastructure and support; we also strive to educate so that engineers and founders have the knowledge and resources they need to create, build, and scale effectively.

For additional information on how to fine-tune your LLM with Unsloth Studio, check out Unsloth's [fine-tuning guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide).

## Resources

- [Unsloth Studio Documentation](https://unsloth.ai/docs/new/studio)
- [Unsloth GitHub](https://github.com/unslothai/unsloth)
- [Vast.ai Instance Guide](https://docs.vast.ai/documentation/instances/overview)


Learn how Unsloth Studio helps teams fine-tune AI models faster, use less VRAM, and lower cloud GPU costs on Vast.ai.

How to Use Unsloth Studio for Fast and Affordable AI Fine-Tuning


Creative AI workflows can get pretty complex.

Let's say your project requires image and video generation with 3D modeling and animation, plus you want to include a voiceover track and music in the background. Or maybe you're fine-tuning a custom language model and immediately testing it through an API while generating supporting visuals in [ComfyUI](https://vast.ai/article/getting-started-with-comfy-UI).

Normally, projects like these require juggling multiple GPU instances for different AI workflows, which can get complicated and expensive fast.

Vast.ai's [All-in-One App Studio](https://cloud.vast.ai/template/readme/1d77da4218e4f86b1d2d0fe755d9b953) offers a unified solution that's elegantly simple: you get a complete creative AI studio with eight powerful applications, a GPU-accelerated remote desktop with KDE Plasma, and Blender, all in a single container.

## Everything You Need in One GPU Instance

The All-in-One App Studio template lets you spin up a single GPU instance to generate images and videos, create music, synthesize voices, transcribe audio, train LoRAs, fine-tune and serve LLMs, and work inside a full desktop environment with 3D rendering abilities.

The template includes:

- ComfyUI for node-based image and video workflows.
- SD Forge for Stable Diffusion-based image generation.
- Wan2GP for advanced video generation accessible even to lower-VRAM machines.
- ACE Step 1.5 for commercial-grade AI music generation with optional vocals.
- Voicebox for text-to-speech generation, voice cloning, and audio editing.
- Whisper WebUI for transcribing audio and video files and generating subtitles.
- Ostris AI Toolkit for LoRAs and model training.
- Unsloth Studio for fine-tuning and serving LLMs.
- A full KDE Plasma remote desktop environment with Blender support.

## Built for Multimodal AI Workflows

With the All-in-One App Studio template, everything works together seamlessly. You can make the most of your creative pipelines without constantly spinning up and shutting down separate environments.

ComfyUI and [SD Forge](https://cloud.vast.ai/template/readme/94b6e3bbbc8a968fef0bae6ced1e39f2) share the same model directories, so you can download a checkpoint once and use it in both tools. After you train a LoRA in the AI Toolkit, immediately test it in ComfyUI or Forge through a shared model library.

Fine-tune a language model in Unsloth Studio and then instantly interact with it through the chat interface, or connect tools like Open WebUI and SillyTavern via the built-in OpenAI-compatible API. There's no separate deployment step required.

The desktop service provides a full KDE Plasma remote desktop streamed to your browser. Blender comes fully configured with GPU-accelerated Cycles rendering, so you can create and render 3D scenes on powerful cloud hardware. You also get Google Chrome, LibreOffice, VLC media player, and a complete desktop environment for GUI-based workflows.

Having all of these tools in a single container significantly reduces setup headaches and makes experimentation so much easier.

## Flexible Workloads Without Wasting VRAM

Start what you need, stop what you don't. That's how All-in-One App Studio handles GPU resources. Nothing auto-starts.

Instead, the template's Instance Portal has a Supervisor tab where you click "Start" next to the applications you want to use at any given time. If you want to work in ComfyUI, start it up with a click. Need the desktop for Blender? Start that, too. Done with video generation? Stop Wan2GP and free up VRAM for another task.

If you want more automation, you can set specific services to auto-start on boot or run your own provisioning script to configure your workspace with custom models or software.

The template is adaptable to your budget, as well. Lower-VRAM GPUs with 8-12 GB can comfortably handle lighter workloads like transcription or voice synthesis, while 24-48 GB instances are better suited for FLUX image generation, LLM fine-tuning, training, and video workflows.

## Getting Started with All-in-One App Studio

It only takes a few minutes to get up and running with the [All-in-One App Studio template](https://cloud.vast.ai/template/readme/1d77da4218e4f86b1d2d0fe755d9b953). [Launch an instance](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=All-in-One%20App%20Studio) with the VRAM you need via the Vast.ai console: 24 GB or more for most workflows, or 48 GB or more if you're planning to run multiple applications simultaneously.

Then open the Instance Portal, navigate to the Supervisor tab, and start up the apps you want to use. It's that easy.

One instance. Eight creative AI tools. A full desktop with Blender. Endless possibilities.

Try All-in-One App Studio on Vast.ai today.


Launch eight creative AI tools, a GPU-accelerated KDE desktop, and Blender in one Vast.ai container with the All-in-One App Studio template.

All-in-One App Studio: Your Complete Creative AI Toolkit in a Single Container


This month's updates bring new templates and guides, major upgrades to two-factor authentication, a benchmarking tool to help you pick the right GPU for your workload, and a complete creative AI studio that runs in a single container.

## NVIDIA Cloud GPU Updates

We've introduced several platform-level improvements this month. Two-factor authentication (2FA) now supports authenticator apps like Google Authenticator and 1Password instead of relying solely on SMS. Additional upgrades include CLI 2FA support, automatic backup codes for account recovery, multiple authentication methods per account, and 2FA for team roles and API key permission groups.

> **Note:** Action is required if you use legacy SMS 2FA. Simply log in normally, go to the Settings page, and click "Regenerate" on the backup codes section to migrate to the new 2FA in less than a minute.

For developers, the Vast [pip package](https://pypi.org/project/vastai/) has been upgraded, and the SDK and CLI repositories are now merged into a single, easy-to-grab repo. We've also introduced an agent skill for coding assistants like Claude and Codex to drive our CLI more effectively.

## Highlighted Feature: Benchmark Before You Deploy

Choosing the right production GPU is now faster and easier, without any guesswork involved. Vast's [benchmarking CLI](https://docs.vast.ai/guides/serverless/choosing-gpus) will take your workload and test it on H100s, A100s, 5090s, 4090s, or any selection you define to compare performance and cost efficiency across GPU classes.

The CLI rents one instance of each GPU type you specify, or a default set for a reasonable starting point if you don't specify any, and then runs the template's built-in benchmark workload, reports performance per dollar, and tears down each rental automatically when finished. Multiple GPU types can be tested in parallel, making it much faster to evaluate your options before committing to longer rentals or production traffic.

Because each run uses real instances and consumes account credit, you get measured performance and cost per hour on actual hardware, not estimates. This benchmarking tool is built for the moment you're staring at a pricing page, wondering whether an H100 is actually worth 4x the cost of a 5090. Sometimes it is, and sometimes it isn't. Now you can find out before you deploy.

## New Templates and Guides

We've rolled out another batch of templates and guides to support creative workflows, fine-tuning, experimentation, and multimodal reasoning.

The standout addition this month is the **All-in-One App Studio** template. It packs eight creative AI applications and a GPU-accelerated remote desktop with KDE Plasma and Blender into a single container, giving you a complete AI production environment without having to manage separate instances.

### New Templates

- [All-in-One Creative AI Studio](https://cloud.vast.ai/template/readme/1d77da4218e4f86b1d2d0fe755d9b953) - One GPU instance: image generation, video, music, voice, transcription, LoRA training, LLM fine-tuning, and KDE desktop with Blender, each activated only when you need it.
- [Unsloth Studio](https://cloud.vast.ai/template/readme/6444356291eabeb423440bb946ccc197) - No-code UI for running and fine-tuning 500+ open-source models with 70% less VRAM.
- [Kimi K2.6](https://vast.ai/model/kimi-k2.6) - 1T-parameter MoE, 32B active, multimodal agentic model with 256K context.
- [Qwen3.5 27B](https://vast.ai/model/qwen35-27b) and [Qwen 3.5B-A3B](https://vast.ai/model/qwen36-35b-a3b) - MoE vision-language models with unified multimodal reasoning and 256K context.
- [Gemma 4 26B-A4B](https://vast.ai/model/gemma-4-26b-a4b-it) and [Gemma 31B IT](https://vast.ai/model/gemma-4-31b-it) - Google's vision-language models with thinking mode and 256K context.
- [Autoresearch](https://docs.vast.ai/examples/ai-agents/autoresearch) - Let Claude Code run ML experiments for you, now in a [template](https://cloud.vast.ai/?ref_id=146798&creator_id=146798&name=Autoresearcher%20%28Claude%20Code%29).

### New Guides

- [Unsloth Studio Guide](https://docs.vast.ai/examples/ai-ml-frameworks/unsloth-studio)
- [CLI Hello World](https://docs.vast.ai/cli/hello-world)
- [SDK Hello World](https://docs.vast.ai/sdk/python/quickstart)

## Other Improvements

This update also brings fleet-wide mitigations for the recently disclosed CopyFail exploit. No action is required from users, and running workloads remain unaffected.

## Our Commitment

As other clouds are out of GPU capacity or charging a premium for what's left, we remain focused on making high-performance AI infrastructure available, affordable, and easy to deploy at scale.

Need help? Contact us anytime at [support@vast.ai](mailto:support@vast.ai) or join our [Discord](https://discord.gg/vast) server for tips, community support, and platform updates in real time.

## Change Log

### New Features

- [Benchmarking CLI](https://docs.vast.ai/guides/serverless/choosing-gpus) for GPU performance and cost comparison.
- [All-in-One App Studio](https://cloud.vast.ai/template/readme/1d77da4218e4f86b1d2d0fe755d9b953): eight creative AI tools in one GPU instance.
- Authenticator app support: use any TOTP app, including Google Authenticator, 1Password, and Microsoft Authenticator, instead of SMS.
- CLI 2FA: 2FA now works through the Vast CLI.
- Backup codes: automatically generate one-time recovery codes so you're never locked out.
- Multiple authentication methods: register more than one per account.
- 2FA for team roles and API key permission groups.

Set up and enable 2FA in [Settings](https://cloud.vast.ai/settings/) -> [Full setup guide](https://docs.vast.ai/guides/reference/account-settings#two-factor-authentication).

### Issues Resolved

- CopyFail exploit: fleet-wide mitigations rolled out with no impact on running workloads and no user action required.
- SSH key input validation, error message, and link fixes.
- Cloud copy fixes.
- Mobile UI fixes.
- Notifications fixes.
- General proactive security fixes.

### API Changes

- The Vast [pip package](https://pypi.org/project/vastai/) has been upgraded, and the SDK and CLI repos have been merged into a single, easy-to-grab repo. Use: `pip install vastai`.
- Install our agent skill to help Claude, Codex, etc., drive our CLI: `npx skills add vast-ai/vast-cli`.

### Updated Templates

- Wan2GP - migrated from provisioning installer to Docker.
- Whisper UI - migrated from provisioning to Docker image.
- FluxGym - Dockerized to prevent provisioning failures.
- vLLM - updated build actions for new CU130 image tag.

### New Blog Posts

- [Deploy LLM Inference Using Vast.ai Serverless](https://vast.ai/article/deploy-llm-inference-using-vast-ai-serverless)
- [Mistral Small 4 Just Dropped - Run It on Affordable H200s with Vast.ai](https://vast.ai/article/mistral-small-4-just-dropped-run-it-on-affordable-h200s-with-vast-ai)
- [Unsloth Studio Is Now Available on Vast.ai](https://vast.ai/article/unsloth-studio-now-available-on-vast-ai)


Vast.ai's May 2026 product update includes authenticator app support for 2FA, a benchmarking CLI for GPU selection, new creative AI templates, and platform security improvements.

May 2026 Product Update


## What Is TurboQuant?

TurboQuant has arrived, and already it has decisively bent the trajectory of LLM inference research. Savings of over 5x in memory requirements and faster token generation speed? Yes, please.

When [Google introduced TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) to the wider world by highlighting it on its research blog, it set the LLM community ablaze almost overnight. The claims seemed almost too good to be true. Memory-chip stocks sold off. Researchers began shipping patches and prototype implementations within days. This time, the promise was real. TurboQuant is not just another clever optimization paper. It has redrawn the practical landscape of transformer inference by attacking one of its most expensive pressure points: the KV cache.

For anyone renting GPUs and trying to ship something that actually works, that is the part that matters. This is no longer theoretical hand-waving. [The paper](https://arxiv.org/abs/2504.19874) has been public since April 2025, but Google brought it to a much wider audience in March 2026, and the idea has already spread into the local-inference stack, including [llama.cpp discussions](https://github.com/ggml-org/llama.cpp/discussions/20969), [vLLM feature work](https://github.com/vllm-project/vllm/pull/39890), and [experimental Ollama-adjacent efforts](https://github.com/ollama/ollama/issues/15051).

## Why Long-Context Inference Gets Expensive

People talk about AI cost as if it were mostly a compute problem. Sometimes it is. But for long-context inference, the uglier constraint is often GPU memory.

The culprit is the KV cache. During prefill and generation, the model stores keys and values from earlier tokens so it can attend back to prior context efficiently instead of recomputing everything from scratch. That is what makes long prompts, chat history, and retrieval-heavy workflows feasible. It is also what makes VRAM usage climb as context grows. [Google's writeup](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) explicitly frames KV cache as a central memory bottleneck for LLM inference, and the [paper](https://arxiv.org/abs/2504.19874) reports TurboQuant results specifically for KV-cache quantization.

That matters because once the KV cache swells, the consequences are brutal and immediate. VRAM headroom disappears, concurrency drops, deployment options narrow, and if you push too far, the job can fail outright. You may need a higher-VRAM card than the model weights alone would suggest. You may have to cut concurrency. You may even end up in a multi-GPU setup just to hold context, which complicates setup and can be a latency footgun. This is why long-context jobs can feel disproportionately expensive: the model is not merely "doing more thinking." It is dragging around a bigger and bigger memory footprint until VRAM pressure starts dictating what hardware is viable at all.

## Cracking the Memory Bottleneck

[Google describes TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) as an extreme compression method that can reduce KV-cache storage to roughly the 3-bit range while preserving quality in the reported benchmarks and accelerating attention-logit computation. The [paper](https://arxiv.org/abs/2504.19874) states the result more precisely, reporting absolute quality neutrality at about 3.5 bits per channel and only marginal degradation at about 2.5 bits per channel for KV-cache quantization. It also says the method is suitable for online applications, which is exactly what you want here. The cache is not a file you compress once and forget about. It sits in the hot path of inference.

That "3-bit" headline is why people noticed, but the deeper point is more practical. TurboQuant is not just trying to make the cache smaller on paper. It is trying to keep it small enough that your GPU spends less time moving KV data around and less of its VRAM budget just holding onto context. The market reaction made for drama, but the implementer's version is simpler: more context on the same card, or more headroom on the card you already use.

## Why "Just Quantize It" Is Not Enough

It is easy to say "compress the cache and save memory." It is harder to do that without giving back the win somewhere else. Low-bit schemes can lose fidelity, and some of them rely on enough extra metadata, scaling factors, or correction machinery that the real savings look less impressive once everything is counted. The TurboQuant paper calls this out directly by focusing not only on mean-squared error, but also on inner-product distortion, which matters because attention depends on those inner products.

There is also a runtime angle. A compression scheme is not useful here if it saves VRAM by making generation slower. If encoding, decoding, or correction is too expensive, you have not removed a bottleneck. You have moved it. Google's blog is notable because it does not present TurboQuant as a pure "memory down, speed down a little" trade. It reports substantial speedups in attention-logit computation, which strongly suggests the compressed representation is cheap enough to use in practice.

## What TurboQuant Does Differently

At a high level, TurboQuant is a two-stage scheme. The first stage rewrites the vector into a form that is easier to compress efficiently. Google's public explanation leans on the PolarQuant framing here. The important point is not to turn the article into a geometry lecture. It is that the representation is rearranged so that low-bit quantization behaves unusually well and avoids some of the per-block overhead conventional methods carry around.

The second stage uses a Quantized Johnson-Lindenstrauss transform, or QJL, on the residual. This is where the method gets more interesting than a generic "make the numbers smaller" story. The paper's point is that a quantizer can look decent on ordinary reconstruction error and still distort the inner products on which attention relies. TurboQuant spends a tiny additional correction budget to preserve those inner products more faithfully. In plain English, it is not just compressing the cache. It is compressing it in a way that tries not to confuse attention.

## Does It Reduce the "Math Tax"?

People sometimes describe TurboQuant as reducing generation cost by moving a math tax into prefill. That is not quite wrong, but it is not quite right either. A better description is that TurboQuant pays a small cost when KV entries are created so later attention steps have less data to move around. That includes prefill, but it does not stop there, because new KV entries are also created during generation. The win is not really that generating tokens no longer has to do the math; it is that generation keeps touching a smaller working set. In practice, the more important tax is often memory bandwidth. Google's published results support exactly that framing: they report attention-logit speedups, which points to reduced memory traffic and a friendlier working set rather than a mere reshuffling of arithmetic.

TurboQuant does not make the model attend to fewer tokens. It makes each attention step cheaper by keeping the KV cache compact enough that the GPU spends less time hauling memory around.

## Why Cheap Decompression Matters

A scheme like this only helps if the compressed cache is cheap to use. If the decoding or correction path is expensive, you can save memory and still make generation slower. That is why generic file-compression instincts do not really apply here. Inference is not trying to pack data for archival storage; it is trying to keep a hot data structure small and fast enough to touch constantly.

That is why TurboQuant is more interesting than a raw compression ratio would suggest. The published results indicate the overhead is low enough for the overall attention path to speed up rather than slow down. Saving VRAM is nice, but saving VRAM without sabotaging decoding speed is actually what changes system behavior. Google's blog reports up to 8x speedup in attention-logit computation on H100 for 4-bit TurboQuant, exactly the kind of result people notice.

## What It Means to You on Vast.ai

If you are renting GPUs on Vast.ai, you already know the real tradeoffs: VRAM, throughput, latency, context length, and hourly price. TurboQuant matters because it changes one of the nastiest constraints in that equation.

When these published gains carry over to your stack, a given workload may fit on a smaller card than before. A long-context job that used to push you toward a high-VRAM premium instance might stay viable on something cheaper. You may also be able to preserve more context or support more concurrent requests before memory becomes the thing that ruins your day. [Google reports](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) up to 6x KV-cache compression and substantial attention-logit speedups, making it possible to run larger models on the same hardware without breaking the bank.

## Bottom Line

TurboQuant matters because it attacks one of the fastest-growing costs in long-context inference: the KV cache. If that cost can be cut this aggressively without wrecking quality or slowing generation, then a lot of assumptions about what hardware a workload needs start to break. For people renting GPUs, this is not a side effect. This is **leverage**.

## Useful Resources

- [TurboQuant paper on arXiv](https://arxiv.org/abs/2504.19874)
- [Google Research: TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/)
- [llama.cpp TurboQuant KV-cache discussion](https://github.com/ggml-org/llama.cpp/discussions/20969)
- [vLLM TurboQuant KV-cache PR](https://github.com/vllm-project/vllm/pull/39890)
- [Ollama TurboQuant/RotorQuant issue](https://github.com/ollama/ollama/issues/15051)
- [Reuters coverage of AI memory-chip demand](https://www.investing.com/news/stock-market-news/samsung-flags-eightfold-jump-in-q1-profit-as-ai-chip-demand-drives-up-prices-4599550)


TurboQuant reduces LLM KV-cache memory pressure while speeding attention-logit computation, changing the economics of long-context inference on rented GPUs.

TurboQuant Explained: How It Reduces LLM Memory by 5x and Speeds Up Inference


Teams running inference-heavy workloads almost always run into the same problem: the high cost of compute. They're often forced to pay for capacity they're not using, or struggle to keep up when demand spikes.

What teams really need is scalable LLM deployment without expensive hyperscaler pricing or overprovisioned infrastructure that sits idle between peak loads.

That's where [Vast.ai Serverless](https://vast.ai/products/serverless) comes in. It's a new way to run AI inference on GPUs with automated scaling that solves three problems traditional cloud providers often exacerbate.

## Why Traditional GPU Clouds Aren't Ideal for LLM Inference

LLM inference workloads are fundamentally different from training jobs. They're bursty and latency-sensitive, so they amplify traditional GPU infrastructure challenges, such as:

- Unpredictable costs when compute usage varies but capacity must be provisioned ahead of time.
- Laggy cold starts when scaling up to meet spikes in demand.
- Infrastructure overhead from manual capacity planning, instance management, and vendor lock-in.

These limitations make it difficult to run LLM inference efficiently at scale. However, Vast.ai Serverless solves this problem by routing workloads to the most cost-effective hardware and scaling compute automatically. In doing so, it eliminates the need to manage infrastructure altogether.

## What Makes Vast.ai Serverless Different

The following are a few ways Vast.ai Serverless stands out:

### Predictive Optimization That Anticipates Demand

Instead of reacting to spikes in demand after they occur, Vast.ai Serverless anticipates them.

The platform proactively provisions reserve GPU workers based on historical usage patterns, real-time load, and ongoing market benchmarking. This approach minimizes laggy cold starts for latency-sensitive LLM inference.

### One Endpoint, Powered by a Global GPU Fleet

Vast.ai continuously benchmarks over 17,000 GPUs across 500+ locations worldwide. Our network includes everything from RTX-class consumer GPUs to enterprise-grade machines like A100s, H100s, and B200s.

But rather than locking you into a single hardware profile, Vast.ai Serverless lets you deploy one endpoint backed by our entire globally distributed GPU fleet. [Multiple Workergroups per Endpoint](https://docs.vast.ai/documentation/serverless/architecture) enable requests to be routed dynamically based on workload requirements.

This real-time hardware selection ensures your unique, inference-heavy workloads run on the most cost-effective GPUs available at any given moment while also improving fault tolerance due to reduced single-point risk.

### Lowest-Cost Transparent Serverless Pricing

Vast.ai bills by the second across three pricing options: On-Demand, Interruptible, and Reserved. You can start with just $5, and there are no hidden fees, tiers, or usage caps.

Plus, Vast.ai Serverless delivers AI inference cost savings of up to 75% or more compared to traditional providers with centralized infrastructure. It's the lowest-cost autoscaling GPU cloud on the market today.

How does it work? Let's take a look at the deployment process.

## How to Deploy LLM Inference Serverless on Vast.ai

Running LLM inference on Vast.ai Serverless takes five steps:

### 1. Bring Your Own Large Language Model

Launch open-source LLMs like Llama 3 or DeepSeek, or deploy your own custom fine-tuned models. Vast.ai provides prebuilt images, including vLLM, TGI, and Oobabooga, so you can serve models with minimal setup.

### 2. Send an Inference Request

Call your endpoint via API to send inference requests. Vast.ai handles all orchestration behind the scenes.

### 3. Vast.ai Serverless Routes and Provisions Compute on Ready GPUs

Our platform automatically selects the most efficient hardware based on your performance targets as well as current availability. Reserve workers are already provisioned and ready to handle your request immediately.

### 4. GPU Executes and Returns Results

The selected GPU worker executes the inference request and returns results through the same API endpoint. This entire process happens seamlessly.

### 5. Serverless Predictively Scales Up and Down Based on Usage

As demand fluctuates, Vast.ai Serverless automatically scales capacity up or down to match. Billing stops immediately when resources are released, so you only pay for actual compute time.

## Get Started with Vast.ai Serverless

Deploying LLM inference at scale doesn't have to mean choosing between performance and affordability. Vast.ai Serverless gives you both. You can run production-ready inference that automatically scales when you need it and stops charging when you don't.

Ready to build? Check out our [Serverless Overview](https://docs.vast.ai/documentation/serverless) and our [Model Library](https://vast.ai/model-library), and get started today with just $5.


Learn how Vast.ai Serverless deploys cost-efficient LLM inference with predictive autoscaling, global GPU routing, and per-second billing.

Deploy LLM Inference Using Vast.ai Serverless


# Mistral Small 4 Just Dropped — Run It on Affordable H200s with Vast.ai

[Mistral Small 4](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) is the first Mistral model to unify three previously separate families — instruct, reasoning (Magistral), and coding (Devstral) — into a single set of weights. It scores 78.0 on MMLU-Pro and 71.2 on GPQA Diamond, outperforms GPT-OSS 120B on LiveCodeBench with 20% less output, and delivers 40% lower latency and 3x higher throughput than Mistral Small 3.

Under the hood, it's a 119B-parameter mixture-of-experts model that activates just 6.5B parameters per token. It includes a Pixtral vision encoder, a 256K-token context window, native function calling, and a per-request `reasoning_effort` parameter to toggle between fast responses and deep chain-of-thought. Weights are stored in FP8 and it fits on 2x H200 141 GB GPUs with no quantization required. It's Apache 2.0 licensed.

This guide walks through deploying Mistral Small 4 on [Vast.ai](https://vast.ai) with vLLM.

## Model Overview

| Property | Value |
|---|---|
| Developer | [Mistral AI](https://mistral.ai/news/mistral-small-4) |
| Model | Mistral-Small-4-119B-2603 |
| Architecture | MoE with Multi-head Latent Attention (MLA), 128 experts, 4 active per token |
| Total Parameters | 119B |
| Active Parameters | 6.5B per token |
| Context Length | 256K tokens |
| Vision | Pixtral encoder |
| License | Apache 2.0 |
| HuggingFace | [mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) |

## Deploy Mistral Small 4 on Vast.ai with vLLM

The model weights are ~111 GB in FP8. With KV cache and overhead, you need 2x H200 141 GB (282 GB total) or 4x H100 80 GB. We'll use 2x H200 since they offer more headroom per card.

### Prerequisites

- A [Vast.ai](https://cloud.vast.ai) account with credits
- [`jq`](https://jqlang.org/) for parsing JSON responses
- The Vast CLI and OpenAI Python client installed:

```bash
pip install --upgrade vastai openai
vastai set api-key <YOUR_API_KEY>
```

Generate an API key to secure your model endpoint:

```bash
export VLLM_API_KEY=$(openssl rand -hex 24)
echo $VLLM_API_KEY  # save this somewhere
```

### Find a GPU

```bash
vastai search offers "gpu_name=H200 num_gpus=2 direct_port_count>=1 rentable=true disk_space>=500 cuda_vers>=12.2"
```

### Launch the Instance

Mistral Small 4 requires a [custom vLLM Docker image](https://hub.docker.com/r/mistralllm/vllm-ms4) with fixes for tool calling and reasoning parsing. Upstream support is tracked in [vLLM PR #37081](https://github.com/vllm-project/vllm/pull/37081).

Pick an offer ID from the search results and deploy:

```bash
vastai create instance <OFFER_ID> \
  --image mistralllm/vllm-ms4:latest \
  --env '-p 8000:8000' \
  --disk 500 \
  --onstart-cmd "vllm serve mistralai/Mistral-Small-4-119B-2603 \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 2 \
    --max-model-len 131072 \
    --attention-backend FLASH_ATTN_MLA \
    --tool-call-parser mistral \
    --enable-auto-tool-choice \
    --reasoning-parser mistral \
    --api-key $VLLM_API_KEY"
```

The `--reasoning-parser mistral` flag separates the model's chain-of-thought reasoning from the final answer in API responses. `--max-model-len 131072` sets a 128K context window as a practical default — the model supports up to 256K natively if you have the VRAM headroom.

Check progress with `vastai logs <INSTANCE_ID>`.

## Call the API

Find your instance's IP and port:

```bash
vastai show instance <INSTANCE_ID> --raw | jq -r '"\(.public_ipaddr):\(.ports["8000/tcp"][0].HostPort)"'
```

You can also find this in the [Vast console](https://cloud.vast.ai/instances/).

### Text Generation

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://<IP>:<PORT>/v1",
    api_key="<YOUR_VLLM_API_KEY>",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {"role": "user", "content": "What are three advantages of mixture-of-experts architectures over dense transformers? Be concise."}
    ],
    max_tokens=512,
    temperature=0.1,
)

print(response.choices[0].message.content)
```

### Vision

The built-in Pixtral encoder handles images alongside text — pass image URLs directly in the message content:

```python
response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this chart. What benchmarks are being compared, and which model performs best?"},
                {"type": "image_url", "image_url": {"url": "https://cms.mistral.ai/assets/7d5b181d-776d-4406-aabc-b88414edf567.png"}},
            ],
        }
    ],
    max_tokens=1024,
    temperature=0.1,
)

print(response.choices[0].message.content)
```

### Configurable Reasoning

Toggle between fast responses and step-by-step reasoning on a per-request basis with `reasoning_effort`:

```python
response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {"role": "user", "content": "Find all integers n such that n^2 + 2n + 4 is divisible by 7."}
    ],
    max_tokens=2048,
    temperature=0.7,
    extra_body={"reasoning_effort": "high"},
)

# Reasoning trace is separated from the final answer
data = response.model_dump()
thinking = data["choices"][0]["message"].get("reasoning")
answer = data["choices"][0]["message"]["content"]

print("Thinking:", thinking[:500] if thinking else "N/A")
print("Answer:", answer)
```

Use `reasoning_effort="none"` for fast, everyday tasks. Use `"high"` for math, logic, and complex analysis.

## Cleanup

When you're done, destroy the instance to stop charges:

```bash
vastai destroy instance <INSTANCE_ID>
```

## Conclusion

Mistral Small 4 packs 119B parameters into a model that activates 6.5B per token, unifying instruct, reasoning, and coding capabilities behind a single endpoint. With vision, a 256K context window, and native function calling, it covers a wide range of use cases. The [NVFP4 checkpoint](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4) (~66 GB) fits on 2x H200, and a trained [Eagle speculative decoding head](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle) is available for faster generation.

## Resources

- [Mistral Small 4 on Hugging Face](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)
- [Mistral Small 4 NVFP4 on Hugging Face](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)
- [Mistral Small 4 Announcement](https://mistral.ai/news/mistral-small-4)
- [vLLM Documentation](https://docs.vllm.ai/)
- [Vast.ai Documentation](https://docs.vast.ai/)


Mistral Small 4 is the first Mistral model to unify three previously separate families — instruct, reasoning (Magistral), and coding (Devstral) — into a single set of weights.

Mistral Small 4 Just Dropped — Run It on Affordable H200s with Vast.ai


It's been a busy few months at Vast.ai, with updates spanning performance and reliability improvements, new templates and guides, and a major step forward for serverless deployment workflows.

## NVIDIA Cloud GPU Updates

Managing large numbers of rented instances is now faster and more responsive, thanks to pagination and performance improvements when fetching instance data. Monitoring serverless workloads is easier as well, with improved serverless metric visualization so you can track usage and performance at a glance.

We've also updated the Host Setup page with clearer instructions for new hosts, and extended two-factor authentication via SMS to support additional countries.

## Highlighted Feature: Deploy Serverless GPU Endpoints from Python

You can now create, configure, and manage serverless GPU endpoints directly from code, without relying on the dashboard.

With the open beta of the Vast SDK deployment workflow, you can define Docker images, apt packages, autoscaling settings, and more entirely in Python. The goal is simple: one install, one deployment flow, and a much faster path from local code to a live serverless endpoint.

Where this gets especially powerful is the `@remote` programming model. Instead of building HTTP wrappers and managing infrastructure by hand, you can decorate a Python function and call it like local code while Vast handles the serverless execution layer underneath.

This is the fastest way to spin up serverless GPUs at Vast's price-performance point, and it's available now to all users as part of the open beta.

Everything you need is in the [deployment docs](https://docs.vast.ai/documentation/serverless/deployments) and [Python examples](https://docs.vast.ai/documentation/serverless/deployments/examples/square).

![Python serverless deployment example](/uploads/april-2026-serverless-sdk.png)

## New Templates and Guides

We also shipped a fresh round of templates, model pages, and deployment guides to help you get up and running faster across training, inference, multimodal workflows, and migrations.

### New Templates

- [Unsloth Studio](https://cloud.vast.ai/?ref_id=62897&template_id=2fba903698162c83cca06d926a016764) - LLM fine-tuning and inference environment.
- [GLM 5](https://vast.ai/model/glm-5) - 744B MoE model for agentic reasoning, coding, and tool use.
- [Kimi K2.5](https://vast.ai/model/kimi-k2.5) - Next-generation multimodal agentic model.
- [vLLM Omni](https://cloud.vast.ai/?ref_id=62897&template_id=10d5f684d6da7700379b16a7bcb74faa) - Multimodal inference engine.
- [Voicebox TTS](https://cloud.vast.ai/?ref_id=62897&template_id=c2325adb01ae87b2a3e8f6f68f88fabe) - Multi-model text-to-speech with voice cloning.
- [LTX 2.3 via ComfyUI](https://vast.ai/model/ltx-2-3) - Video generation workflow.

### New Guides

- [Budget-Friendly Alternative to Claude Code - Overnight Ralph Loop](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop)
- [OpenClaw AI Assistant with vLLM on Vast.ai](https://docs.vast.ai/examples/ai-agents/openclaw)
- [Autonomous AI Research with Autoresearch on Vast.ai](https://docs.vast.ai/examples/ai-agents/autoresearch)
- [Migrate from Runpod to Vast.ai](https://docs.vast.ai/examples/migrations/runpod-to-vast)

## Other Improvements

This update also brings a number of platform-level changes, including BitPay support for crypto payments, serverless OpenAI-compatible endpoints, pagination for large rented-instance result sets, and a long list of quality-of-life and reliability fixes.

## Our Commitment

As we continue to improve platform performance and expand support for deployment workflows, our focus remains on making AI infrastructure easier to use at scale.

Need assistance? Reach out anytime at [support@vast.ai](mailto:support@vast.ai) or join our [Discord](https://discord.gg/vast) server for tips, discussion, and the latest updates as they happen.

## Change Log

### New Features

- BitPay payment gateway for crypto payments, including Lightning Network support, flat fees, and refunds across hundreds of wallets including Coinbase.
- Serverless OpenAI API-compatible endpoints released.
- Serverless metric visualization improvements.
- 2FA SMS support expanded to many additional countries.
- Pagination and performance improvements when fetching large numbers of rented instances.
- Host Setup page updated with improved instructions for new hosts.

### Issues Resolved

- Local volume deletion issue resolved.
- Copy error visibility improved.
- Copy progress bar issue resolved.
- Disk quota issues resolved.
- Machines missing geolocation resolved.
- Open button token persistence after instance recreation improved.
- Missing DLPerf on the instance page resolved.
- Various security improvements.
- Various quality-of-life improvements.
- Various serverless edge-case scaling improvements.
- Host CLI schedule maintenance command issue resolved.

### Feature Changes

- Coinbase is sunsetting Coinbase Commerce on March 31, so payment support is moving accordingly.

### API Changes

- Upcoming breaking change: `get instances` will soon return paginated results, with responses limited to a maximum of 25 instances per request. Update API, CLI, and SDK integrations to handle pagination.
- Deprecated: `disable_bundling` for `search offers` will be removed from the API, CLI, and SDK.
- Serverless SDK updated for enhanced security.

### Updated Templates

- PyTorch
- Open WebUI
- Ollama
- Ostris Toolkit
- ComfyUI
- Llama.cpp (ssl)
- SGLang (onstart script fixed)
- Pinokio (v6.x)

### New GPUs

- RTX PRO 5000

### New Blog Posts

- [Affordable Claude Code Alternative: Run Autonomous Coding Agents on Vast.ai](https://vast.ai/article/affordable-claude-code-alternative-ralph-loop)
- [Vast.ai Named Among Fastest Growing Vendors by Ramp and Brex](https://vast.ai/article/ramp-brex-fastest-growing-vendor)
- [Liquid AI's LFM2 Just Dropped - Here's How to Run It on Vast.ai](https://vast.ai/article/liquid-ai-lfm2-run-on-vast-ai)
- [Running Qwen 3.5 Medium Models on Vast.ai](https://vast.ai/article/running-qwen-3-5-medium-models-on-vast-ai)
- [Vast.ai GPUs Can Now Be Rented Through SkyPilot](https://vast.ai/article/vast-ai-gpus-can-now-be-rentend-through-skypilot)
- [7 Popular Model Templates for AI Workloads on Vast.ai Right Now](https://vast.ai/article/popular-templates-for-ai-workloads-on-vast-ai)
- [Running Nemotron-Cascade-2 on Vast.ai](https://vast.ai/article/running-nemotron-cascade-2-on-vast.ai)


Vast.ai's April 2026 product update includes Python-based serverless deployments in open beta, new templates and guides, payment updates, and platform reliability improvements.

April 2026 Product Update


[Unsloth Studio](https://unsloth.ai/docs/new/studio) is an open-source, no-code web UI for running and fine-tuning more than 500 open-source AI models across text, vision, audio, and embeddings. It delivers 2x faster training with 70% less VRAM than standard approaches, which makes it possible to fine-tune larger models on more affordable hardware.

Vast.ai now offers a pre-configured [Unsloth Studio template](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Unsloth%20Studio). Rent a GPU, click Open, select Unsloth Studio, and start running or training models without CLI setup or manual Docker configuration.

## What You Can Do

- **Run models** — Search and download any of 500+ models from Hugging Face. Chat with built-in tool calling, code execution, and web search.
- **Compare models** — Use Model Arena to evaluate two models side by side on the same prompt.
- **Fine-tune** — Train with QLoRA, LoRA, or full fine-tuning. Upload datasets or build them from raw documents using Data Recipes. Monitor training with real-time loss curves and GPU stats.
- **Export** — Save fine-tuned models as merged safetensors, LoRA adapters, or GGUF files. Upload directly to Hugging Face Hub.

## Get Started

Read the full guide: **[Run and Fine-Tune AI Models with Unsloth Studio on Vast.ai](https://docs.vast.ai/examples/ai-ml-frameworks/unsloth-studio)**

Or launch the template directly: **[Open Unsloth Studio Template](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Unsloth%20Studio)**

## Resources

- [Unsloth Studio Documentation](https://unsloth.ai/docs/new/studio)
- [Unsloth GitHub](https://github.com/unslothai/unsloth)
- [Vast.ai Instance Guide](https://docs.vast.ai/documentation/instances/overview)


Run and fine-tune 500+ open-source AI models with the Unsloth Studio template on Vast.ai.

Unsloth Studio Is Now Available on Vast.ai


# Inside GTC 2026: A Bright Future for AI and GPU Computing

NVIDIA's flagship conference, [GTC](https://www.nvidia.com/gtc/), concluded in San Jose with much fanfare. We attended and want to share our key takeaways and what they mean for the future of AI compute. GTC's sessions generated a buzz in the industry, ranging from NVIDIA CEO Jensen Huang's keynote, which introduced the powerful (and controversial) DLSS 5, to announcements of new hardware technologies and the shift from generative to agentic systems. The industry event sparked excitement about the future of AI and GPU computing.

## AI Infrastructure Becomes the Product

The focus has shifted as greater attention is now being paid to AI inference versus yesteryear's training. As AI systems mature and vie for market saturation, the need for sustained computing power grows, and with it, AI infrastructure. NVIDIA and others look to reshape the narrative around AI datacenters, framing them as "AI Factories" where training and inference converge and full-stack systems bring cost efficiency and reliability.

## From Generative to Agentic AI

With the announcement of NemoClaw, a secure, enterprise-grade wrapper for the agentic framework OpenClaw, NVIDIA has broadened the market. NemoClaw was built to address the issue of trust in AI agent adoption, making it safer for companies to utilize agentic systems at scale. This is a crucial development for founders as infrastructure will increasingly need to support long-lived processes with memory and retrieval for 24/7 persistent workloads.

## The Evolution of AI in Gaming

The latest iteration of NVIDIA's gaming tech, DLSS 5, was announced during the show and raised eyebrows in more ways than one. Some critics quickly panned it as "AI slop" but behind the graphical changes that are sure to improve over time lies the "Real-Time Neural Rendering" architecture. The advancement expands possibilities for AI use in creative fields as developers will now be able to precisely control AI rendering parameters. As barriers to AI use in creative projects fall and pressure for workflow efficiency rises, demand for GPU compute power in creative professions will grow.

## Changes in the Global AI Market

Jensen Huang had a lot to say at this year's GTC but some select comments about the broader market and NVIDIA's renewed growth within China stood out. "Our supply chain is getting fired up," Huang said, revealing that NVIDIA is in the process of restarting manufacturing of H200 chips for the Chinese market. After receiving licenses from the U.S. government to once again sell the chips in China, NVIDIA will begin fulfilling orders from Chinese companies. This will have a broader impact on the GPU market as demand for advanced chips expands once again.

## Powering the Next Era of AI Compute

GTC 2026 brought together major and emerging players who showcased new technologies, shared industry developments, and made predictions for the future of AI. What comes next will be defined by how efficiently and broadly AI compute can be deployed.

At Vast.ai, we value the fair distribution of AI computing resources. Our goal is to help you access the infrastructure you need to build, scale, and compete. [Speak to our team](https://vast.ai/contact-sales) today to learn how we can support your AI workloads.


Vast.ai attended NVIDIA GTC 2026 and shares key takeaways on what they mean for the future of AI compute.

Inside GTC 2026: A Bright Future for AI and GPU Computing


Andrej Karpathy recently released [autoresearch](https://github.com/karpathy/autoresearch) -- a framework that lets an AI agent autonomously run ML research experiments while you sleep. Point it at a GPU, give it a training script, and it will iterate on model architecture and hyperparameters for hours, keeping improvements and discarding regressions. No human in the loop.

The idea is simple but powerful: a single H100 running overnight can execute ~100 five-minute training experiments. Each one modifies the model code, trains, evaluates, and decides whether to keep or revert. By morning you have a log of findings from every experiment.

## Why This Matters

ML research has always been bottlenecked by the experiment loop -- tweak, train, evaluate, repeat. Autoresearch collapses that loop by handing it to an AI agent ([Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview)) that can run continuously. It's not replacing the researcher -- it's running the tedious overnight grid searches and ablations so you can focus on the ideas that require human intuition.

Autoresearch includes a simplified single-GPU implementation of [nanochat](https://github.com/karpathy/nanochat) -- a small GPT trained on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) and evaluated on validation bits-per-byte (`val_bpb`). The agent has full freedom to modify the training script -- architecture, optimizer, learning rate schedules, attention patterns -- within a fixed 5-minute time budget per experiment.

## Running It on Vast

We put together a step-by-step guide for running autoresearch on Vast.ai. It covers everything from renting a GPU instance to launching the autonomous research loop:

**[Autonomous AI Research with Autoresearch on Vast.ai ->](https://docs.vast.ai/examples/ai-agents/autoresearch)**

The guide walks through:

- Renting a GPU instance with the Vast CLI
- Installing dependencies (`uv`, Node.js, Claude Code)
- Preparing the training data and running a baseline experiment
- Configuring Claude Code for unattended overnight operation
- Monitoring experiments as they run

All you need is a Vast.ai account and a Claude Code account. The whole setup takes about 15 minutes before you can kick off the research loop and walk away.


Run Andrej Karpathy's autoresearch on Vast.ai H100 GPUs to automate overnight ML experiments and training loops.

How to Run Karpathy's Autoresearch on Vast H100s


If you’ve ever dabbled in machine learning, data science, or coding in general, chances are you’ve heard of Google Colaboratory – better known as **Google Colab**. But what exactly is it, what are its limitations, and how can you make the best use of it?

Let’s take a closer look at these questions and more...

## What Is Google Colab?

In a nutshell, **Google Colab** is a browser-based platform that lets you run and write code in a Jupyter notebook environment without having to set up anything locally on your machine. Offering both free and paid plans, it’s widely used for Python-based projects, especially in machine learning and data analysis.

Colab is an easy starting point for experiments, since you can use the Jupyter notebook interface to write and execute code, visualize results, and keep everything neatly organized in one place. This type of interactive programming allows you to experiment with code in real time, making it ideal for data science and research projects that require immediate feedback and iteration.

Some other benefits include:

- **Accessibility** – The browser-based platform allows you to access your projects from anywhere.
- **Integration with Google Drive** – Save your notebooks and datasets directly to your Google Drive.
- **Collaboration** – Easily share your notebooks with others for real-time collaboration across teams.
- **Wealth of Resources** – Plenty of tutorials, sample notebooks, and community support are available in the Colab ecosystem, making it easier for you to learn and troubleshoot along the way.

These features make Google Colab a convenient tool for a wide range of users. However, it does have some significant limitations that may prompt you to look for other solutions – such as running a Jupyter instance right here on **Vast** and uploading your notebook directly.

## Why Choose Vast Over Google Colab?

While Google Colab is helpful for smaller-scale projects or initial experimentation, **Vast's** cloud GPU platform offers a much more flexible, powerful, and reliable experience through our Jupyter integration. Here’s why you should consider using Vast instead.

### No Resource Limits

With Google Colab, free-tier users face dynamic usage limits with idle timeout periods, very limited access to GPUs, and resource caps that sometimes fluctuate. The paid plans have their own set of restrictions as well.

For example, notebooks can only run for up to 12 hours on a free Colab plan. The highest-tier paid plan supports continuous background execution for at most 24 hours, with output saved to Google Drive even if your browser is closed. Lower-tier paid plans have some extended execution based on availability and may not automatically save output to Drive.

But with **Vast**, there are no imposed time limits, so your projects can run uninterrupted for as long as you need. Whether your project takes hours or days, you won’t be cut off or forced to restart – and your data will stay on the Vast instance, available to be copied at all times.

> **Note:** Google Colab does include the option to pay for a dedicated virtual machine (VM) to get around usage limits, but Vast’s pricing is extremely competitive as we are the market leader in low-cost cloud GPU rental.

### Customizable GPU Power for Every Project

Google Colab offers different VMs based on the plan you choose. The free tier only allows access to VMs with a standard system memory profile, restricting the compute power you can use. Paid versions of Colab provide access to machines with a higher system memory profile, but you still don't get to select specific GPUs unless you opt to manually pay for a dedicated machine.

**Vast** gives you access right from the start to powerful and customizable hardware options that are also highly affordable. You can select the exact GPU setup you need, whether you're running small tasks or large-scale machine learning models. For high-end performance, Vast supports multi-GPU configurations, making it easy to scale up your projects without hitting resource bottlenecks.

### Simplicity and Reliability

**Vast** simplifies the entire process by running your Jupyter notebook directly in the cloud with no need for SSH or complex setups. Jupyter by itself is much more reliable than Google Colab and doesn’t require setting up SSH keys – you can always open a terminal right inside the browser.

And while Google Colab does facilitate sharing and collaboration, **Vast** offers the same, with the added benefit of running on a more stable platform without timeouts or interruptions. Team members can collaborate in real time, sharing notebooks and results without worrying about session limits.

## Making the Switch to Vast

If you’ve been using Google Colab for simple notebooks and want to step up your workflow, switching to Vast is straightforward. Simply download your notebook from Colab as a `.ipynb` file, run a Vast Jupyter instance with the recommended [PyTorch template](https://cloud.vast.ai/?ref_id=43484&template_id=f5540ef1a1398b8499546edb53dae704), and upload your notebook into Jupyter directly.

Check out the step-by-step guide on running [Jupyter notebooks](https://docs.vast.ai/google-colab) on Vast.

While Google Colab is a useful starting point for basic tasks, it often falls short when workflows become more demanding and compute costs start adding up. **Vast** provides a more cost-effective and customizable solution through its Jupyter notebook integration, with greater control, higher performance, and fewer interruptions.

With **Vast.ai**, you’ll enjoy the freedom to push your projects even further. **Explore the possibilities today!**


Google Colab is a browser-based platform that lets you run and write code in a Jupyter notebook environment without having to set up anything locally on your machine.

Google Colab Explained: Simplifying Your Workflow with Cloud Tools


# Running Nemotron-Cascade-2 on Vast.ai

NVIDIA released [Nemotron-Cascade-2-30B-A3B](https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B), a reasoning model that achieves gold-medal-level scores on international math and programming competitions. It does this with 30B total parameters and only 3B active per token - 20x fewer than competing models - outperforming both Qwen3.5-35B-A3B and larger open-weight alternatives.

The model fits on a single 80 GB GPU at about 60 GB in BF16 with no quantization or multi-GPU setup required. It includes built-in chain-of-thought reasoning, tool-integrated reasoning for math and code, and an OpenAI-compatible API.

This guide walks through deploying the model on [Vast.ai](https://vast.ai) with vLLM. Vast.ai offers A100 and H100 GPUs at 3-5x less than traditional cloud providers, with no contracts or minimums.

## Model Overview

| Property | Value |
| --- | --- |
| Developer | [NVIDIA](https://research.nvidia.com/labs/nemotron/nemotron-cascade-2/) |
| Model | Nemotron-Cascade-2-30B-A3B |
| Architecture | Hybrid Mamba-Attention MoE |
| Total Parameters | 30B |
| Active Parameters | 3B per token |
| Training | Cascade RL post-training from Nemotron-3-Nano base |
| License | [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| HuggingFace | [nvidia/Nemotron-Cascade-2-30B-A3B](https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B) |

## Deploy Nemotron-Cascade-2 on Vast.ai with vLLM

The model requires a single 80 GB GPU (A100 or H100) with about 150 GB disk for weights and cache.

### Prerequisites

- A [Vast.ai](https://cloud.vast.ai) account with credits
- The Vast CLI installed and configured:

```bash
pip install --upgrade vastai
vastai set api-key <YOUR_API_KEY>
```

Generate an API key to secure your model endpoint:

```bash
export NEMOTRON_API_KEY=$(openssl rand -hex 24)
echo $NEMOTRON_API_KEY  # save this somewhere
```

### Find a GPU

Search for the cheapest single 80 GB GPU with direct port access:

```bash
# -o dph sorts results by price (dollars per hour, cheapest first)
vastai search offers "gpu_ram>=80 num_gpus=1 direct_port_count>=1 cuda_vers>=12.4 cuda_vers<13" -o dph
```

### Launch the Instance

Pick an instance ID from the search results and deploy with vLLM:

```bash
vastai create instance <INSTANCE_ID> \
  --image vllm/vllm-openai:latest \
  --env '-p 8000:8000' \
  --disk 150 \
  --onstart-cmd "vllm serve nvidia/Nemotron-Cascade-2-30B-A3B \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 32768 \
    --api-key $NEMOTRON_API_KEY \
    --trust-remote-code"
```

This pulls the model from HuggingFace and starts an OpenAI-compatible API server on port 8000. Initial startup takes several minutes while the model downloads.

Once the instance is running, get your connection details:

```bash
vastai show instances --raw | jq '.[] | select(.id == <INSTANCE_ID>) | {id, actual_status, public_ipaddr, ports}'
```

You can also find the IP and port mapping in the [Vast console](https://cloud.vast.ai/instances/).

## Call the API

Send a request to test the model:

```bash
curl http://<VAST_IP>:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $NEMOTRON_API_KEY" \
  -d '{
    "model": "nvidia/Nemotron-Cascade-2-30B-A3B",
    "messages": [{"role": "user", "content": "Find all integers n such that n^2 + 2n + 4 is divisible by 7."}],
    "max_tokens": 2048
  }'
```

The model thinks before it answers - you will see its step-by-step reasoning in the response, followed by the final answer. This chain-of-thought reasoning is what drives its competition-level performance on math and code.

## Cleanup

When you're done, destroy the instance to stop charges:

```bash
vastai destroy instance <INSTANCE_ID>
```

## Conclusion

Nemotron-Cascade-2 delivers reasoning performance that previously required models 20x its size, and it runs on a single GPU. That combination of capability and efficiency makes it a practical option for math, code, and general reasoning workloads. With Vast.ai, you can have it running in minutes at a fraction of what traditional cloud providers charge.

## Resources

- [Nemotron-Cascade-2-30B-A3B on HuggingFace](https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B)
- [Nemotron-Cascade-2 Research Page](https://research.nvidia.com/labs/nemotron/nemotron-cascade-2/)
- [Nemotron-Cascade-2 Paper (arXiv:2603.19220)](https://arxiv.org/abs/2603.19220)
- [Vast.ai Documentation](https://docs.vast.ai/)
- [vLLM Documentation](https://docs.vllm.ai/)


Deploy NVIDIA's Nemotron-Cascade-2-30B-A3B on Vast.ai with vLLM and run a powerful reasoning model on a single 80GB GPU.

Running Nemotron-Cascade-2 on Vast.ai


When you're getting started with an AI project, choosing the right model is only half the battle. You also need to configure environments and install dependencies while connecting the tools that make your workflows run smoothly. Setting things up can quickly become a bottleneck.

Prebuilt templates simplify the process – and **Vast.ai** gives you plenty to choose from!

Our [Model Library](https://vast.ai/model-library) offers a model-first way to deploy AI, with templates that launch ready-to-run environments for some of the most-used AI tools today. Vast.ai also offers [lower-level templates](https://cloud.vast.ai/templates) for frameworks and inference engines like PyTorch and vLLM for more customizable setups.

Each pre-configured template is organized by modality and use case so you can quickly find what you need – and you retain full custom control over the underlying runtime, parameters, and infrastructure.

### Popular Templates in the Vast.ai Model Library

From image, text, and audio-video generation to multimodal AI workflows, the following are a few of the most popular templates being deployed on Vast.ai right now.

#### 1. Flux.2-Dev – Rectified Flow Image Generation

[**FLUX.2-Dev**](https://vast.ai/model/flux.2-dev) is a 32-billion parameter rectified flow transformer built for advanced text-to-image generation and editing. The model processes text prompts alongside optional reference images, so you can modify existing images or combine multiple visual references without additional fine-tuning.

Its ability to maintain consistent characters, objects, and styles across generations makes it ideal for character design, visual storytelling with scene continuity, digital art with precise style control, and design prototyping.

#### 2. LTX-2 – Native Audio-Video Generation in a Single Workflow

[**LTX-2**](https://vast.ai/model/ltx-2) is a 19-billion parameter Diffusion Transformer (DiT) model that generates audio and video together in a single pass. It creates synchronized audiovisual output from text prompts, images, and other inputs – keeping dialogue, motion, and ambient sound aligned throughout the clip.

One of LTX-2's biggest advantages is its light and iterative workflow. Since the model generates video in a compressed latent space before converting it to full resolution, it supports faster iteration and more efficient memory usage than many traditional video models.

#### 3. WAN 2.2 – Efficient Text-to-Video and Image-to-Video Generation

[**WAN 2.2**](https://vast.ai/article/wan-2-2-explained-new-approach-ai-video-generation) is a video generation model that produces short cinematic clips from either text prompts or static images. What sets it apart is its Mixture-of-Experts (MoE) architecture. The model switches between specialized experts that handle early scene structure and later visual refinement. This results in smoother motion and higher visual fidelity along with more efficient use of compute.

Available on Vast.ai in both text-to-video ([T2V](https://vast.ai/model/wan-2.2-t2v-a14b)) and image-to-video ([I2V](https://vast.ai/model/wan-2.2-i2v-a14b)) variants – with the latter offering optional guidance via text prompts – WAN 2.2 gives you fine control over motion, lighting, and scene composition. Popular use cases include rapid prototyping of video concepts based on text descriptions and product animations generated from still images.

> Not sure whether **WAN 2.2** or **LTX-2** fits your workflow best? Our detailed [comparison post](https://vast.ai/article/wan-2-2-vs-ltx-2-which-ai-video-model-should-you-use) explores the strengths of each model and which one to choose for your specific needs!

#### 4. ACE-Step V1 – Fast Text-to-Music Generation

[**ACE-Step V1**](https://vast.ai/model/ace-step-v1-3.5b) is an open-source music generation model that creates full musical compositions from text prompts in **17 languages**. Thanks to its diffusion-based architecture and efficient audio compression pipeline, it generates music about **15x faster** than comparable alternatives.

Musical quality is high, with controllable duration (up to five minutes works best) and coherent output across melody, harmony, and rhythm. You can even edit lyrics and manipulate vocals. Overall, ACE-Step V1 is a practical tool for fast experimentation with musical ideas, as well as voice cloning applications and music remixing and style transfer.

#### 5. Dia 1.6B – Realistic Dialogue Generation from Text

[**Dia 1.6B**](https://vast.ai/model/dia-1.6b) is a text-to-speech model that produces highly realistic dialogue in English from given transcripts. Using [S1] and [S2] speaker tags, it creates multi-speaker conversations that can also include nonverbal expressions like laughter, coughing, and gasps.

Audio conditioning enables precise control over emotion and tone, and the model even supports voice cloning functionality. Dia 1.6B's applications include conversational audio content, accessibility tools, and voice synthesis applications that require consistent, natural-sounding speech.

#### 6. DeepSeek V3.2 Exp – Long-Context Sparse Attention LLM

[**DeepSeek V3.2 Exp**](https://vast.ai/model/deepseek-v3.2-exp) is an open-source experimental large language model designed for long-context reasoning and advanced text generation. It introduces DeepSeek Sparse Attention, a novel mechanism that enables efficient processing of extended documents and conversations without sacrificing output quality.

The model is well suited for long-form document analysis and generation, research that requires extended reasoning, code generation and debugging, and even customer support with context-aware responses. Notably, it has a license that offers full commercial use permissions.

#### 7. Qwen3.5 397B A17B – Multimodal Reasoning and Agentic Workflows

[**Qwen3.5 397B A17B**](https://vast.ai/model/qwen35-397b-a17b) is the open-weight of the first model in the new Qwen3.5 series. It's a frontier-level native vision-language model with outstanding results across numerous benchmarks.

Built with a sparse mixture-of-experts (MoE) architecture and hybrid attention mechanisms, Qwen3.5 397B A17B supports highly complex tasks in areas like mathematical reasoning and problem solving, multi-turn agentic workflows, code generation and debugging, document analysis, video understanding, visual question answering, and multilingual interaction across **200+** languages and dialects.

### Launch Templates in Minutes on Vast.ai

Today's AI workloads span an incredibly wide range of modalities. The templates above represent just a small snapshot of what developers are building with on Vast.ai right now.

With the **Vast.ai** [Model Library](https://vast.ai/model-library), getting started on your own AI projects is easy. Simply browse templates by use case, select the recommended GPU hardware, and launch a production-ready environment in minutes!


Discover the most popular prebuilt templates on Vast.ai for image, video, audio, and text AI workloads — from Flux.2-Dev to Qwen3.5.

7 Popular Model Templates for AI Workloads on Vast.ai Right Now


[SkyPilot](https://skypilot.co/) is an open-source framework from UC Berkeley that abstracts away cloud infrastructure, letting you define AI workloads once and run them on any provider. It handles resource discovery, provisioning, cost optimization, and failover across 25+ clouds, including Vast.ai.

With Vast.ai as a SkyPilot backend, you get access to Vast's GPU marketplace, often the lowest-cost option for on-demand GPU compute, through SkyPilot's unified interface. SkyPilot automatically finds the cheapest available instance matching your requirements, provisions it, and manages the full lifecycle.

This guide covers setup, configuration, deploying a model, and the advanced Vast-specific options available in SkyPilot.

## Prerequisites

- A [Vast.ai account](https://cloud.vast.ai/) with credits
- Python 3.7-3.13 (on Python 3.12+, install `skypilot[vast]` explicitly - it is excluded from `skypilot[all]`)
- Terminal access

## Installation

Install SkyPilot with Vast.ai support:

```bash
pip install -U "skypilot[vast]"
```

If you use `uv`, you can also install via:

```bash
uv pip install "skypilot[vast]"
```

This pulls in all required dependencies, including the Vast SDK.

## Configuring Credentials

Get your API key from the [Vast.ai Account page](https://cloud.vast.ai/account/), then store it where SkyPilot expects:

```bash
mkdir -p ~/.config/vastai
echo "<YOUR_VAST_API_KEY>" > ~/.config/vastai/vast_api_key
```

Verify that SkyPilot detects your credentials:

```bash
sky check
```

You should see `Vast: enabled` in the output.

## Deploying a Model: DeepSeek R1 on vLLM

This section walks through deploying DeepSeek R1-0528-Qwen3-8B, an 8B parameter model with step-by-step reasoning capabilities, on a Vast.ai H100 GPU using vLLM.

### Create the YAML Configuration

Create a file named `deepseek-r1-inference.yaml`:

```yaml
resources:
  accelerators: H100:1
  cloud: vast
  image_id: docker:vllm/vllm-openai:latest
  disk_size: 100

run: |
  vllm serve deepseek-ai/DeepSeek-R1-0528-Qwen3-8B \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.9 \
    --reasoning-parser deepseek_r1
```

| Parameter | Purpose |
|-----------|---------|
| `accelerators: H100:1` | Request 1x H100 GPU |
| `cloud: vast` | Use Vast.ai as the cloud provider |
| `image_id: docker:vllm/vllm-openai:latest` | Pre-built vLLM Docker image |
| `disk_size: 100` | 100GB storage for model weights and container |
| `--reasoning-parser deepseek_r1` | Enable DeepSeek's chain-of-thought reasoning output |
| `--gpu-memory-utilization 0.9` | Use 90% of GPU VRAM for model serving |

### Launch the Cluster

Deploy the service with SkyPilot:

```bash
sky launch deepseek-r1-inference.yaml --cluster deepseek-r1
```

SkyPilot will find the cheapest available H100 on Vast.ai, provision it, download the model (~15GB), and start the vLLM server. This takes 5-10 minutes.

Monitor progress in real time:

```bash
sky logs deepseek-r1 --follow
```

Confirm the cluster is ready:

```bash
sky status deepseek-r1
```

Look for `Status: UP`.

### Access the Service via SSH Tunnel

Vast.ai instances provisioned through SkyPilot don't expose ports directly. Use SSH tunneling to access the vLLM server on your local machine:

```bash
ssh -L 8000:localhost:8000 deepseek-r1
```

Keep this terminal open - the tunnel must remain active for API access.

### Test the Deployment

In a new terminal, verify the server is running:

```bash
curl http://localhost:8000/health
curl http://localhost:8000/v1/models
```

Send a test inference request:

```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    "messages": [
      {"role": "user", "content": "What is 15 * 8? Show your reasoning step by step."}
    ],
    "max_tokens": 300,
    "temperature": 0.1
  }'
```

### Python Client Example

The vLLM server exposes an OpenAI-compatible API, so you can use the standard OpenAI Python client:

```python
import openai

client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="sk-fake-key"  # vLLM doesn't require a real API key
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{
        "role": "user",
        "content": "Write a brief history of Los Angeles in one paragraph."
    }],
    max_tokens=300,
    temperature=0.7
)

print(response.choices[0].message.content)
```

## Managing Your Cluster

```bash
# Check status
sky status deepseek-r1

# View logs
sky logs deepseek-r1

# SSH into the instance
ssh deepseek-r1

# Stop the cluster (preserves data, stops billing)
sky stop deepseek-r1

# Restart a stopped cluster
sky start deepseek-r1

# Auto-stop after 30 minutes of idle time
sky autostop deepseek-r1 --idle-minutes 30

# View all clusters and costs
sky status --all

# Tear down completely
sky down deepseek-r1
```

Always stop or tear down clusters when you're done to avoid unnecessary charges.

## Advanced: Vast Configuration in SkyPilot

SkyPilot supports Vast-specific configuration in `~/.sky/config.yaml`. These options let you control instance selection and pass parameters directly to the Vast API.

### Datacenter-Only Instances

Filter to professional datacenter-hosted machines only. This excludes consumer-grade or home-hosted GPUs, which can improve reliability:

```yaml
vast:
  datacenter_only: true
```

Note that some GPU types may only be available on non-datacenter offers, so enabling this may reduce availability.

### Custom Instance Parameters

The `create_instance_kwargs` block passes parameters directly to the Vast API when creating instances. This gives you access to Vast features beyond what SkyPilot's standard YAML exposes:

```yaml
vast:
  datacenter_only: true
  create_instance_kwargs:
    python_utf8: true
    lang_utf8: true
    extra: "--shm-size=16g"
    onstart_cmd: "echo 'Instance started'"
```

#### Supported Parameters

| Parameter | Description |
|-----------|-------------|
| `image` | Docker image override (e.g., `vastai/base-image:@vastai-automatic-tag`) |
| `env` | Environment variables and port mappings (e.g., `"-e KEY=value -p 8080:8080"`) |
| `price` / `bid_price` | Maximum bid price for preemptible (spot) instances |
| `disk` | Disk size in GB (overrides `disk_size` in task YAML) |
| `label` | Custom instance label |
| `extra` | Extra `docker run` arguments (e.g., `"--shm-size=16g"`) |
| `onstart_cmd` | Shell command to run on instance start |
| `onstart` | Path to a local script to run on instance start |
| `login` / `image_login` | Docker registry credentials for private images |
| `python_utf8` | Enable Python UTF-8 mode |
| `lang_utf8` | Enable system UTF-8 locale |
| `jupyter_lab` | Start JupyterLab on the instance |
| `jupyter_dir` | Jupyter notebook directory path |
| `template_hash_id` | Use a pre-configured Vast template by hash ID |
| `args` | Custom Docker command arguments |
| `user` | Run the container as a specific user |
| `vm` | Use VM mode instead of container mode |

### Using Vast Templates

If you have a saved Vast template, you can reference it by hash ID. When using a template, `image` and `disk` are inherited from the template:

```yaml
vast:
  create_instance_kwargs:
    template_hash_id: "abc123def456"
    price: 0.50
```

### Spot (Preemptible) Instances

Request spot instances for lower costs. SkyPilot handles preemption recovery automatically when using managed jobs:

```yaml
resources:
  accelerators: H100:1
  cloud: vast
  use_spot: true
```

You can set a maximum bid price via `create_instance_kwargs`:

```yaml
vast:
  create_instance_kwargs:
    bid_price: 0.50
```

## Current Limitations

Vast.ai support in SkyPilot does not currently include:

- **Multi-node clusters** - interconnection between nodes is non-trivial on Vast
- **Object store mounting** - no direct mounting of S3, GCS, etc.
- **Port opening after launch** - ports must be configured at launch time
- **Custom disk or network tiers** - Vast does not expose these options

## Troubleshooting

### "Vast: not enabled" After `sky check`

Verify your API key file exists and is readable:

```bash
cat ~/.config/vastai/vast_api_key
```

If the file is missing, re-run the credential setup steps above. If you previously stored the key at `~/.vast_api_key`, note that SkyPilot now expects it at `~/.config/vastai/vast_api_key`.

### SSH Tunnel Connection Refused

Confirm the cluster is running before attempting the tunnel:

```bash
sky status deepseek-r1
```

If the status is `UP`, recreate the tunnel:

```bash
ssh -L 8000:localhost:8000 deepseek-r1
```

### Model Not Loading or Out of Memory

SSH into the instance and check GPU utilization:

```bash
ssh deepseek-r1
nvidia-smi
```

If the model exceeds available VRAM, reduce `--gpu-memory-utilization` to `0.7` or `--max-model-len` to a smaller value in your YAML, then redeploy.

### Slow Startup

Large Docker images (vLLM is ~15GB) take time to download. Model weights add another 5-30GB depending on the model. First launch on a new instance can take 10+ minutes. Use `sky logs <cluster> --follow` to monitor progress.

## Cleanup

Tear down the cluster when you're done to stop all charges:

```bash
sky down deepseek-r1
```

## Next Steps

- **Try other models**: Swap the model in the YAML to serve Llama 3, Mistral, or Qwen
- **Use managed jobs**: Run `sky jobs launch` instead of `sky launch` for automatic recovery from preemptions
- **Explore spot instances**: Add `use_spot: true` to your resource block for lower costs
- **Datacenter filtering**: Set `datacenter_only: true` in `~/.sky/config.yaml` for production workloads
- **SkyPilot docs**: Full reference at [docs.skypilot.co](https://docs.skypilot.co/)

## Additional Resources

- [SkyPilot Documentation](https://docs.skypilot.co/)
- [SkyPilot GitHub](https://github.com/skypilot-org/skypilot)
- [SkyPilot Vast Configuration Reference](https://docs.skypilot.co/en/latest/reference/config.html#vast)
- [Vast.ai Console](https://cloud.vast.ai/)
- [Vast.ai Documentation](https://vast.ai/docs/)
- [DeepSeek R1 on HuggingFace](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B)


Learn how to use SkyPilot with Vast.ai to provision low-cost GPUs, deploy vLLM workloads, tune Vast-specific settings, and troubleshoot common issues.

Vast.ai GPUs Can Now Be Rented Through SkyPilot


Qwen 3.5 is Alibaba's latest model family, featuring a hybrid architecture that combines Gated DeltaNet (linear attention) with standard attention in a 3:1 ratio. This design enables dramatically faster inference — up to 8.6x at 32K context and 19x at 256K — while maintaining strong reasoning capabilities. The medium models released February 24, 2026 include:

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Total Params</th>
      <th>Active Params</th>
      <th>Architecture</th>
      <th>VRAM (BF16)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Qwen3.5-122B-A10B</td>
      <td>122B</td>
      <td>10B</td>
      <td>MoE</td>
      <td>~244 GB</td>
    </tr>
    <tr>
      <td>Qwen3.5-35B-A3B</td>
      <td>35B</td>
      <td>3B</td>
      <td>MoE</td>
      <td>~66 GB</td>
    </tr>
    <tr>
      <td>Qwen3.5-27B</td>
      <td>27B</td>
      <td>27B</td>
      <td>Dense</td>
      <td>~54 GB</td>
    </tr>
  </tbody>
</table>

The MoE models route each token through 8 of 256 experts plus 1 shared expert, so only a fraction of the total parameters are active per token. The 35B-A3B model activates just 3B parameters per token while having 35B total — giving you large-model quality at small-model inference cost. All three models are Apache 2.0 licensed, so no HuggingFace token is needed.

In this guide, we'll deploy Qwen3.5-35B-A3B on Vast.ai using SGLang, which has day-0 Qwen3.5 support as of v0.5.9, on a single A100 80GB GPU.

## Deploying on Vast.ai

Install the Vast.ai CLI and set your API key:

```bash
pip install --upgrade vastai
export VAST_API_KEY="YOUR_KEY_HERE"
vastai set api-key $VAST_API_KEY
```

Search for a suitable GPU. The model needs ~66 GB VRAM in BF16, so an 80 GB card works:

```bash
vastai search offers "gpu_ram>=80 num_gpus=1 direct_port_count>=1 rentable=true disk_space>=200 cuda_vers>=12.2" -o dph
```

Deploy the model using SGLang's Docker image. Replace `YOUR_INSTANCE_ID` with an offer ID from the search results:

```bash
vastai create instance YOUR_INSTANCE_ID \
    --image lmsysorg/sglang:latest \
    --env '-p 8000:8000' \
    --disk 200 \
    --onstart-cmd "python3 -m sglang.launch_server \
        --model-path Qwen/Qwen3.5-35B-A3B \
        --host 0.0.0.0 --port 8000 \
        --tp-size 1 \
        --context-length 32768 \
        --reasoning-parser qwen3 \
        --mem-fraction-static 0.85"
```

Key flags:
- `--reasoning-parser qwen3` enables the model's built-in thinking mode, separating reasoning from the final answer
- `--mem-fraction-static 0.85` allocates 85% of GPU memory for KV cache
- `--context-length 32768` sets a 32K context window (the model supports up to 262K natively)

The model will download (~70 GB) and load automatically. Check `vastai logs YOUR_INSTANCE_ID` — you'll see "The server is fired up and ready to roll!" when it's serving.

## Calling the Model

Find your instance's IP and port in the [Instances tab](https://cloud.vast.ai/instances/) of the Vast.ai console. Click the IP address button to see the port mapping for 8000/tcp.

Install the OpenAI SDK:

```bash
pip install --upgrade openai
```

Then call the model:

```python
from openai import OpenAI

VAST_IP_ADDRESS = "YOUR_IP"
VAST_PORT = "YOUR_PORT"

client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B",
    messages=[
        {"role": "user", "content": "What are three benefits of mixture-of-experts models? Be concise."}
    ],
    max_tokens=1024,
    temperature=0.7
)

print(response.choices[0].message.content)
```

### Response

```
1. **Computational Efficiency:** Only a subset of parameters is activated per input,
   reducing inference costs while maintaining a massive total parameter count.
2. **Enhanced Performance:** The larger effective model capacity allows for better
   accuracy on complex and diverse tasks compared to dense models of similar compute.
3. **Specialization:** Individual experts can learn to handle specific input patterns
   or domains, improving the model's robustness and adaptability.
```

With `--reasoning-parser qwen3`, the model's thinking process is automatically separated from the final answer. The thinking tokens appear in `reasoning_content` in the API response, while the clean answer appears in `content`.

## Other Models

You can swap `Qwen/Qwen3.5-35B-A3B` for the other medium models:

- **Qwen3.5-122B-A10B**: Larger MoE model, 122B total / 10B active. At BF16 the full weights need ~244 GB, so you'll need multiple GPUs (e.g., 4× A100 80GB with `--tp-size 4`). Alternatively, GGUF quantizations are available — the Q4_K_M variant is ~70 GB and fits on a single 80GB card.
- **Qwen3.5-27B**: Dense model, all 27B parameters active. Needs ~54 GB VRAM. Fits on a single A100 or H100.

## Conclusion

Qwen3.5-35B-A3B packs 35B parameters into a model that only activates 3B per token, and its hybrid DeltaNet architecture delivers substantially faster inference than standard transformers. With SGLang and a single A100 on Vast.ai, you can have it serving in under 10 minutes.

For production use, consider increasing `--context-length` — the model supports up to 262K natively, with 128K+ recommended for best reasoning performance. GGUF quantizations (Q4_K_M at ~22 GB) are also available for running on consumer GPUs like the RTX 4090.

## Useful Resources

- [Qwen3.5-35B-A3B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)
- [Qwen3.5-122B-A10B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)
- [Qwen3.5-27B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-27B)
- [SGLang Documentation](https://docs.sglang.ai/)
- [Vast.ai Documentation](https://docs.vast.ai/)


Deploy Qwen 3.5 medium models on Vast.ai using SGLang, including setup, launch commands, and API usage.

Running Qwen 3.5 Medium Models on Vast.ai


Liquid AI just released [LFM2-24B-A2B](https://huggingface.co/LiquidAI/LFM2-24B-A2B), a 24 billion parameter foundation model that only activates 2.3 billion parameters per token. That ratio — less than 10% of total parameters active at inference time — makes it one of the most aggressively sparse mixture-of-experts models available today.

What makes LFM2 different from other MoE models isn't just the sparsity. The architecture is a hybrid design that replaces most of the attention layers with gated short convolution blocks. Out of 40 total layers, only 10 use grouped query attention — the other 30 are convolutional. This is a fundamentally different approach from transformer-only models, and it's what allows Liquid AI to push the active parameter count so low while maintaining competitive quality across benchmarks like MMLU-Pro, GPQA Diamond, IFEval, and MATH-500.

Liquid AI, an MIT spinoff, has scaled this architecture from 350M to 24B parameters following log-linear scaling laws. LFM2 is the first model in the family large enough to be broadly useful — and with day-one support for vLLM, SGLang, and llama.cpp, it's ready to deploy today.

> **Note**: LFM2-24B-A2B is an early checkpoint trained on 17 trillion tokens without reinforcement learning or extensive post-training. A post-trained version (LFM2.5-24B-A2B) is coming.

## Model Overview

<table>
  <thead>
    <tr>
      <th>Property</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Developer</td>
      <td><a href="https://www.liquid.ai">Liquid AI</a></td>
    </tr>
    <tr>
      <td>Model</td>
      <td>LFM2-24B-A2B</td>
    </tr>
    <tr>
      <td>Architecture</td>
      <td>Hybrid MoE: 30 convolutional + 10 GQA attention layers, 64 experts, top-4 routing</td>
    </tr>
    <tr>
      <td>Total Parameters</td>
      <td>24B</td>
    </tr>
    <tr>
      <td>Active Parameters</td>
      <td>2.3B per token</td>
    </tr>
    <tr>
      <td>Context Length</td>
      <td>32,768 tokens</td>
    </tr>
    <tr>
      <td>Training Data</td>
      <td>17T tokens (mixed BF16/FP8)</td>
    </tr>
    <tr>
      <td>License</td>
      <td><a href="https://www.liquid.ai/lfm-open-license-v1-0">LFM Open License v1.0</a></td>
    </tr>
    <tr>
      <td>HuggingFace</td>
      <td><a href="https://huggingface.co/LiquidAI/LFM2-24B-A2B">LiquidAI/LFM2-24B-A2B</a></td>
    </tr>
  </tbody>
</table>

## Deploy LFM2-24B-A2B on Vast.ai with vLLM

At BF16, the model weights are ~48 GB — it fits on a single A100-80GB with room for a 32K context KV cache. No quantization or multi-GPU setup needed.

### Prerequisites

- A [Vast.ai](https://cloud.vast.ai) account with credits
- The Vast CLI installed and configured:

```bash
pip install --upgrade vastai  # make sure you're on v0.5.0+
vastai set api-key <YOUR_API_KEY>
```

### Find a GPU

Search for a single GPU with at least 80 GB VRAM and direct port access:

```bash
vastai search offers "gpu_ram>=80 num_gpus=1 direct_port_count>=1 cuda_vers>=12.4 cuda_vers<13" -o dph
```

> **Important**: Use a machine with **CUDA 12.x** (not 13.x). The vLLM nightly image is built against CUDA 12 and will fail with a driver mismatch on CUDA 13 machines.

### Launch the Instance

Pick an instance ID from the search results and deploy with vLLM. LFM2 requires a newer version of the `transformers` library than what ships in the vLLM image, so we upgrade it at startup:

```bash
vastai create instance <INSTANCE_ID> \
  --image vllm/vllm-openai:nightly \
  --env "-p 8000:8000" \
  --disk 100 \
  --onstart-cmd "pip install 'transformers>=5.0.0' && \
    vllm serve LiquidAI/LFM2-24B-A2B \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 32768 --dtype auto \
    --trust-remote-code"
```

This upgrades the tokenizer library, pulls the model from HuggingFace, and starts an OpenAI-compatible API server on port 8000. Initial startup takes several minutes while the model downloads (~48 GB for the BF16 weights).

Once the instance is running, find your IP and port in the [Vast console](https://cloud.vast.ai/instances/) — click the IP button on your instance to see the port mapping.

> **Why `nightly` and `transformers>=5`?** LFM2's tokenizer uses the `TokenizersBackend` class introduced in `transformers 5.0`. The stable vLLM image pins `transformers<5`, so the tokenizer fails to load. Using `nightly` with a transformers upgrade resolves this. This was tested with vLLM nightly build `v0.16.0rc2`. Once a stable vLLM release ships with `transformers>=5.0.0` support, you can switch to `vllm/vllm-openai:latest` and drop the `pip install` step.

## Call the API

```bash
curl http://<VAST_IP>:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "LiquidAI/LFM2-24B-A2B",
    "messages": [{"role": "user", "content": "What makes mixture-of-experts models efficient?"}],
    "max_tokens": 300
  }'
```

## Cleanup

When you're done, destroy the instance to stop charges:

```bash
vastai destroy instance <INSTANCE_ID>
```

## Resources

- [LFM2-24B-A2B on HuggingFace](https://huggingface.co/LiquidAI/LFM2-24B-A2B)
- [LFM2-24B-A2B GGUF Quantizations](https://huggingface.co/LiquidAI/LFM2-24B-A2B-GGUF)
- [Liquid AI Blog: LFM2-24B-A2B](https://www.liquid.ai/blog/lfm2-24b-a2b)
- [Liquid AI Models](https://www.liquid.ai/models)
- [Vast.ai Documentation](https://docs.vast.ai/)
- [vLLM Documentation](https://docs.vllm.ai/)


Liquid AI released LFM2-24B-A2B, a sparse 24B MoE model with 2.3B active parameters per token.

Liquid AI's LFM2 Just Dropped — Here's How to Run It on Vast.ai


Claude Code is one of the most capable agentic AI tools for software development – but it comes with a practical constraint: usage limits and spend. It's easy to run into daily or weekly quotas, and API costs can add up quickly.

Fortunately, there's a workaround.

### Ralph Loop: Running an Open-Source Coding Agent on Vast.ai

Instead of relying on API-limited services or local hardware, you can run open-source coding agents on Vast.ai for about **$1.50 per hour** – even continuously overnight! One example of this is Ralph, an agentic coding loop designed to implement software projects from a Product Requirements Document (PRD).

Ralph picks a user story, writes the code, runs tests, fixes any failures, and moves on to the next story. This process repeats until the project is complete and everything passes.

To jump straight in, follow our [Overnight Ralph Loop guide](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop) to get started. Otherwise, let's explore some more about what you can accomplish with Ralph and why.

### Why Run the Ralph Loop on Vast.ai?

There are several practical reasons to run the Ralph loop on a self-hosted model using our dedicated GPU infrastructure.

First, cost becomes more manageable. On Vast.ai, you pay for GPU time directly, at cost-effective rates that make overnight runs both predictable and affordable. Ralph is built for sustained workflows, so you can run it continuously without the usage caps of API-based tools like Claude Code.

Running Ralph on Vast.ai also gives you direct control over your development environment and supports stricter security and [compliance](https://vast.ai/compliance) needs. At the same time, you can match model size and GPU compute to the complexity of the task, ensuring your resources scale up only when the workload requires it.

### The Ralph Loop in Practice: Model & Use Cases

On Vast.ai, you can run the Ralph loop on Qwen3-Coder-Next, a large Mixture-of-Experts (MoE) model trained specifically for agentic coding tools like Claude Code, Aider, and Cline. It boasts a 256K context length and strong performance across multi-step coding tasks involving large codebases.

Once deployed, Ralph operates continuously and autonomously on real software projects, making it well suited for overnight development workflows. Common use cases include:

- Building a full CLI application
- Implementing REST APIs with authentication and validation
- Generating complete test suites for existing codebases
- Creating a web scraper with multiple site adapters, rate limiting, and data export

You simply define the requirements and allow the agent to carry out the work unattended.

### Getting Started with the Overnight Ralph Loop

With Vast.ai, a typical setup using Qwen3-Coder-Next on a 4x RTX 4090 instance costs approximately $1.50 per hour, with an overnight run often coming in **under $18** .

Our [Overnight Ralph Loop guide](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop) walks you through deploying the model, connecting the agent, and running the Ralph loop on Vast.ai. Start your first overnight run today!


Run autonomous coding agents on Vast.ai for about $1.50/hour with the Ralph Loop workflow.

Affordable Claude Code Alternative: Run Autonomous Coding Agents on Vast.ai


Welcome back to another round of updates at Vast.ai. We've kicked off the new year with improvements focused on expanding orchestration support, refining billing and serverless workflows, and introducing new templates and guides to help you deploy faster and more securely.

Another exciting update: we recently launched the [Vast.ai Startup Program](https://vast.ai/startup), designed to support early-stage teams building on our platform.

Startups can now get up to **$2,500 in free GPU credits** when they move their workloads to Vast.ai. With 24/7 priority support and access to Vast's full range of GPUs, the program delivers high-performance compute to power the next wave of AI innovation without long-term commitments. For more info, see our blog post [here](https://vast.ai/article/vast-ai-startup-program).

Not to mention we have a comprehensive new [Model Library](https://vast.ai/model-library), giving you a faster path from model selection to deployment.

## NVIDIA Cloud GPU Updates

### SkyPilot Enhancements

For teams using [SkyPilot](https://vast.ai/article/vast-ai-gpus-can-now-be-rentend-through-skypilot) to manage deployments, we've made updates that expand configuration options on Vast.ai. SkyPilot users can now filter capacity by Secure Cloud to ensure that workloads run only on verified datacenter-hosted GPUs.

We've also added full support for instance creation parameters via SkyPilot, enabling precise control over images, environment variables, disk size, pricing, startup commands, and other instance-level settings at launch. See [SkyPilot docs](https://docs.skypilot.co/en/latest/reference/config.html#vast) for full details.

### Fresh New Templates and Guides

Our [Model Library](https://vast.ai/model-library) introduces a model-first approach to deploying AI. Templates are organized by modality and use case and include recommended hardware presets and clear dedicated GPU pricing.

Here are some of our newest templates:

- [Kimi K2.5](https://vast.ai/model/kimi-k2.5) - An open-source, native multimodal agentic model that combines vision and language understanding with agentic capabilities.
- [LTX-2](https://vast.ai/model/ltx-2) - A DiT-based audio-video generation model that produces synchronized video and sound.
- [SGLang](https://cloud.vast.ai/template/readme/1fce7860e11071cae9c357fb1c8b7560) - A high-performance LLM inference framework with a hosted API server template.

And our latest guides:

- [dstack + vLLM](https://docs.vast.ai/examples/serving-infrastructure/dstack-vllm): Deploy language models using vLLM with dstack for automated provisioning and cost control.
- [DR-Tulu](https://docs.vast.ai/examples/mcp/dr-tulu): Deploy a research agent trained with an MCP server and web tools.
- [GLiNER2](https://docs.vast.ai/examples/ner/gliner2): Run GLiNER2 for named entity recognition with custom entity types.
- [BrowseSafe](https://docs.vast.ai/examples/ai-agents/browsesafe): Deploy BrowseSafe to detect prompt injection attacks in HTML.
- [Overnight Ralph Loop](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop): Budget-friendly alternative workflow to Claude Code.

### Other Improvements

If you missed it late last year, we also launched [Vast.ai Serverless](https://vast.ai/article/vast-ai-serverless-automated-gpu-scaling). It enables automatic, pay-per-use scaling for AI workloads through a fully serverless API on Vast's globally distributed GPU cloud.

## Our Commitment

With new SkyPilot enhancements, startup opportunities, a model-first deployment experience, and improved inference workflows, we're continuing to build a platform that gives you more control with less friction.

Need help? Contact our customer support team anytime at [support@vast.ai](mailto:support@vast.ai) or join our [Discord](https://discord.gg/vast) server for tips, real-time updates, and ongoing community discussion.

## Change Log

### SkyPilot Enhancements

- Advanced configuration options for SkyPilot on Vast.ai.
- Support for filtering by Secure Cloud (`datacenter_only`).
- Full support for instance creation parameters via SkyPilot.
- See [SkyPilot docs](https://docs.skypilot.co/en/latest/reference/config.html#vast).

### New Templates

- [Kimi K2.5](https://vast.ai/model/kimi-k2.5): Multimodal agentic model.
- [LTX-2](https://vast.ai/model/ltx-2): Video and audio generation model.
- [SGLang](https://cloud.vast.ai/template/readme/1fce7860e11071cae9c357fb1c8b7560): High-performance LLM inference engine.
- Find them in the [Model Library](https://vast.ai/model-library).

### Template Updates

- Base Image and PyTorch now support **CUDA 13.1** and updated NVML libraries.
- Ostris AI Toolkit updates.
- ComfyUI updates.
- Llama.cpp (ssl) updates.
- SGLang onstart script fix.
- Pinokio v6.x updates.

### New Guides

- [dstack](https://docs.vast.ai/examples/serving-infrastructure/dstack-vllm): Serving infrastructure.
- [DR-Tulu](https://docs.vast.ai/examples/mcp/dr-tulu): MCP.
- [GLiNER2](https://docs.vast.ai/examples/ner/gliner2): Named Entity Recognition.
- [BrowseSafe](https://docs.vast.ai/examples/ai-agents/browsesafe): AI agents and security.
- [Overnight Ralph Loop](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop): Budget-friendly alternative to Claude Code.

### Platform Improvements

- Fixed serverless GPU filter UI.
- Fixed serverless template selection UI.
- Billing Transaction History Invoice PDF Export now excludes failed and pending payments and other non-payment activity.
- Billing Transaction History now supports easy individual invoice download.
- Security improvements.
- Local volume deletion issue resolved.
- Copy progress bar issue resolved.
- Disk quota issues resolved.
- Machines missing geolocation resolved.
- Host CLI schedule maintenance command issue resolved.


Vast.ai's January and February 2026 updates include new SkyPilot controls, Startup Program credits, Model Library additions, fresh guides, and major platform fixes.

January & February 2026 Product Updates


### Introducing the Vast.ai Startup Program: AI Infrastructure Without the Burn Rate

Startups can now get [$2,500 in free GPU credits](https://vast.ai/startup) when they move workloads to Vast.ai.

To apply, get in touch with our team to share a bit about your project and compute needs. From there, we'll finalize your credit offer, help you configure the right setup, and get you up and running quickly. You'll also have access to **24/7 priority support** throughout the credit period.

Compute credits apply across Vast.ai's global GPU marketplace, with access to everything from consumer-grade cards to high-end accelerators like A100s and H200s.

The Vast.ai Startup Program gives startups an on-ramp to the same platform that teams rely on well beyond the early stage. That's because Vast.ai is designed to scale alongside you - without long-term commitments or runaway costs.

### Built for Real AI Workloads

From day one, Vast.ai supports the tools and workflows that teams already use. Launch preconfigured environments with Jupyter, Docker, RAPIDS, and Hugging Face to train deep learning models, run large-scale experiments, and process massive datasets on demand.

We offer a growing selection of ready-to-run AI models in our [Model Library](https://vast.ai/model-library), plus thousands of [templates](https://cloud.vast.ai/templates) to get started immediately - or deploy your own custom model anytime. Your team can even collaborate across departments and campuses, avoiding duplicated infrastructure and fragmented workflows.

Most importantly, with Vast.ai, your team can cut costs without cutting capability. Reduce infrastructure expenses by **up to 81%** compared to major cloud GPU providers, while paying only for what you need, when you need it.

Plus, most startups can migrate existing workloads with minimal changes and begin running jobs shortly after approval.

### Secure, Compliant, and Production-Ready

Vast.ai is **SOC 2 Type II** certified and designed to support all of your [security and compliance](https://vast.ai/compliance) needs from the start.

Run workloads in isolated instances with end-to-end encryption and full control over compute environments to keep your sensitive data protected. Our [Secure Cloud](https://cloud.vast.ai/?secure_cloud=true) option lets you filter for GPUs from our vetted datacenter partners for even higher security needs.

Notably, these safeguards support reproducible research via consistent environments, as well as compliance-friendly configurations aligned with GDPR and HIPAA requirements.

With [Vast.ai Serverless](https://vast.ai/article/vast-ai-serverless-automated-gpu-scaling), moving from development to production is even easier. Inference workloads run through a fully serverless API that automatically routes to the most cost-effective GPUs available, according to your performance requirements. This eliminates manual instance management and capacity planning - so you can leverage our distributed GPU fleet with minimal overhead.

### Get Started with Vast.ai Today

For early-stage teams, access to affordable GPU compute is crucial. But there's more to it than that. You also want infrastructure that scales as you grow and supports evolving security and compliance needs - all without locking you into long-term contracts.

The Vast.ai Startup Program helps you get there faster.

[Apply today](https://vast.ai/startup) to get up to 2,500 free compute hours and start building on Vast.ai on your own terms.


Startups can get up to $2,500 in free GPU credits when moving workloads to Vast.ai, with priority support and production-ready infrastructure.

Introducing the Vast.ai Startup Program


If you've ever tried generating AI video and thought, *This looks cool, but it's not what I asked for*, you're not alone!

**Wan 2.2** was designed to address scaling and efficiency challenges in video generation. It is among the first open-source video generation models to leverage a Mixture-of-Experts (MoE) architecture, enabling more efficient training. Created by Alibaba Tongyi Lab, the model gives creators more control over how their ideas turn into motion – following prompts more closely, with smoother movement and cinematic-quality results – without drastically increasing compute requirements.

On Vast.ai, Wan 2.2 is available in two variants in our [Model Library](https://vast.ai/article/vast-ai-model-library-choosing-best-ai-models):

- **[Text-to-video (T2V)](https://vast.ai/model/wan-2.2-t2v-a14b)** – Generating short videos directly from written prompts.
- **[Image-to-video (I2V)](https://vast.ai/model/wan-2.2-i2v-a14b)** – Transforming a static image into a dynamic video sequence.

Both models share the same underlying architecture, which is where things get interesting...

## What Makes Wan 2.2 Different

Most diffusion-based video models use a single neural network to handle the entire denoising process. Wan 2.2's MoE design, on the other hand, uses two specialized "experts" that strategically separate the video generation workflow:

- A **high-noise expert** that handles the early denoising stages, focusing on layout, composition, and overall scene structure and motion patterns.
- A **low-noise expert** that takes over later, refining video details and enhancing cinematic elements like lighting, contrast, and color tone.

This separation helps reduce common issues with video generation like unstable camera motion and inconsistent frames. It also makes the process more efficient. Even though each model has approximately **27 billion parameters** in total, only about **14 billion are active** per inference step.

The models automatically switch between experts via signal-to-noise ratio (SNR) thresholds – a measure of how much meaningful visual information remains versus how much noise is still present at each stage of generation. By activating the right expert at the right moment, Wan 2.2 avoids unnecessary computation. This keeps performance much more efficient, comparable to smaller single-expert models.

In short, you get more natural motion dynamics, superior visual fidelity, reduced artifacts, and less wasted compute.

## One Architecture, Two Workflows

While their underlying architecture is the same, the two Wan 2.2 variants have different creative entry points.

### Wan2.2 T2V A14B (FP8)

The text-to-video (T2V) model uses text prompts to generate **5-second clips at 480P or 720P** resolution, with granular control over lighting, composition, contrast, and color tone.

A few other capabilities of the T2V variant include:

- Superior performance on Wan-Bench 2.0 evaluations compared to leading commercial video generation systems
- Integration with Qwen models or DashScope API for enhanced prompt elaboration
- Support for parameter-type conversion, leading to improved inference speed

Overall, the text-to-video model is well suited for storyboarding, concept visualization, marketing clips, and rapid prototyping of video ideas from plain language.

### Wan2.2 I2V A14B (FP8)

The image-to-video (I2V) model begins with a static image input and turns it into a short video sequence. It offers both **480P and 720P output**, and can optionally support text prompts to guide content and motion in detail.

Some strengths of the I2V variant include:

- Handling a wide range of aesthetic preferences, from photorealistic to stylized
- Processing at **24 frames per second** (FPS) for smooth, high-definition output
- Automatic prompt derivation that makes it possible to generate videos from an image alone *without* text

The image-to-video model is especially useful for turning concept art, product photos, and illustrations into animated content.

## From Idea to Output: Designed for Real-World Use

Whether you start from a text prompt or from an image, Wan 2.2 is built for real-world creative and production workflows.

You don't even need enterprise-grade hardware to use the model. Wan 2.2's MoE architecture keeps things so efficient, you can achieve high-quality results on consumer GPUs like the RTX 4090. For larger workloads, multi-GPU inference is also an option. The model supports FSDP and DeepSpeed Ulysses for accelerated generation.

Wan 2.2 pairs particularly well with Vast.ai's flexible GPU infrastructure. On Vast, you can spin up exactly the hardware you need and run the model on a single GPU, or scale across multiple GPUs for faster generation as your video projects grow.

Both the [T2V](https://vast.ai/model/wan-2.2-t2v-a14b) and [I2V](https://vast.ai/model/wan-2.2-i2v-a14b) models are compatible with [ComfyUI](https://docs.vast.ai/documentation/serverless/comfyui-wan-2.2) workflows, making them easy to integrate into tools you may already be using – which are also available directly on Vast.ai.

Ready to start creating with Wan 2.2? Launch it on [Vast.ai](https://vast.ai/model-library) in just a few clicks and start generating cinematic video today!


Wan 2.2 uses a Mixture-of-Experts architecture to deliver efficient, high-quality AI video generation. Learn how it works and why it matters.

Wan 2.2 Explained: A New Approach to AI Video Generation


Wouldn't it be great if you could just think of a scene and instantly turn it into a video the way it appears in your head? Technology isn't quite there yet, but today’s most advanced AI video generation models are getting us closer.

Today we’re taking a look at **WAN 2.2** and **LTX-2**, two open-source/open-weights models that transform text and images into short-form video.

## What WAN 2.2 and LTX-2 Are — and How They Differ

From the outside, WAN 2.2 and LTX-2 are pretty similar tools. They’re both open-source/open-weights diffusion-based video generation models designed to turn images or text prompts into short video clips. Their underlying architecture, however, is very different.

### WAN 2.2: Prompt Fidelity and Cinematic Control

Developed by Alibaba Tongyi Lab, WAN 2.2 is built around a Mixture-of-Experts (MoE) architecture. Instead of using a single neural network to manage the entire denoising process, it employs two specialized “experts”: a **high-noise expert** for overall structure and layout, and a **low-noise expert** for refining textures and details like lighting and color tone.

Switching between these two experts means the model can allocate compute depending on what it needs to do at any given moment—focusing on broader structure first and finer details later. It also boosts efficiency by avoiding unnecessary computation.

WAN 2.2 comes in three main variants, each designed for different workflows:

- **Text-to-Video (T2V):** Generates 5-second video clips at 480P to 720P from a text prompt written in plain language. This is a flexible option for scenes where everything needs to be synthesized from scratch.
- **Image-to-Video (I2V):** Begins with a single image and turns it into a short video. It uses automatic prompt derivation to generate video from an image without text input at all, but can also support text prompts for more directed results.
- **Hybrid:** A compact model with 5 billion parameters that handles both text-to-video and image-to-video generation. It delivers high-definition results at up to 720P and 24 FPS, but is designed for users with lower VRAM.

The base WAN 2.2 models generate video only, without native audio output. However, there is a specialized speech-to-video version (WAN 2.2 S2V) that transforms static images and audio inputs into synchronized videos.

### LTX-2: Native Audio-Video Generation

Created by Lightricks, LTX-2 is a DiT-based (Diffusion Transformer) audio-video generation model. It produces audio and visuals together in one pass, keeping dialogue, lip movements, and ambient sound aligned coherently.

Its architecture is based on latent diffusion, which means the model works in a compressed version of the video first, before converting it into full resolution. This makes it more memory efficient and enables faster iteration, translating to quicker experimentation and lower hardware overhead.

LTX-2 can generate up to **~20 seconds of synchronized audio and video**, with support for high resolutions and high frame rates depending on configuration and available compute. The model offers fine-grained control options—such as LoRA-based customization and multimodal inputs including text, image, video, and audio—for precise creative direction.

This makes LTX-2 a highly flexible model. In short, it supports text-to-video, image-to-video, and native audio-visual generation, along with cross-modal workflows like audio-to-video, text-to-audio, and video-to-audio—all within a single model.

## Choosing the Right Model for Your Workflow

How the two models are designed directly affects what you experience as a user.

For instance, WAN 2.2's MoE design prioritizes structured generation and motion consistency. It boasts strong prompt adherence with high-fidelity output and is more likely to preserve scene intent across frames, sticking closely to what you asked for—albeit at the cost of slightly longer generation times.

LTX-2's latent diffusion approach emphasizes speed and accessibility. It’s faster to iterate with, easier to experiment on, and even offers native audio-video sync. However, it may require more prompt tuning to get exactly what you want.

### Choose **WAN 2.2** if you want:

- Cinematic or narrative-style clips where composition and camera motion are critical
- Strong prompt fidelity for complex scenes with multiple elements
- More deliberate, production-oriented outputs and professional video content

### Choose **LTX-2** if you prefer:

- Rapid prototyping of video concepts and creative exploration for lengthier scenes
- Visual storytelling or character-driven video with synchronized dialogue or sound
- A lighter and more iterative workflow where speed matters more than precision

Both models also integrate with ComfyUI, so you can jump right into testing them out with an intuitive node-based visual workflow.

## Final Thoughts

Neither WAN 2.2 nor LTX-2 is objectively superior to the other. The two open-source/open-weights models are designed for different kinds of workflows and creative goals. The best way to get a feel for them is to actually try them out.

The good news is that both models run well on high-end consumer GPUs, making them far more accessible than many people might expect. With Vast.ai, it’s even easier: you can spin up the right hardware on demand and experiment on your own terms, paying only for the compute you need—and save up to 80% over traditional clouds.

Try [WAN 2.2 T2V](https://vast.ai/model/wan-2.2-t2v-a14b) and [WAN 2.2 I2V](https://vast.ai/model/wan-2.2-i2v-a14b), or [LTX-2](https://vast.ai/model/ltx-2) (or both!) in our [Model Library](https://vast.ai/model-library), and build your own creative pipeline on Vast.ai today.


A practical comparison of WAN 2.2 and LTX-2—two open-source/open-weights AI video models—covering architecture, strengths, and use cases.

WAN 2.2 vs. LTX-2: Which AI Video Model Should You Use?


dstack is an open-source GPU orchestration platform that automates instance provisioning and lifecycle management across cloud providers. This guide walks through using dstack with Vast.ai as the backend, combining declarative infrastructure with competitive GPU marketplace pricing.

## What Makes dstack Special?

Key features include:
- **Infrastructure as Code**: Define GPU requirements, pricing limits, and workloads in YAML
- **Automatic Provisioning**: dstack finds and provisions the best available instances
- **Cost Controls**: Set `max_price` to cap hourly costs automatically
- **Built-in Proxy**: Access services through dstack's authenticated endpoint

## What's in the New Guide

Our latest documentation walks you through deploying language models with dstack and vLLM on Vast.ai.

**Complete Setup Instructions**
- Installing and configuring dstack with your Vast.ai API key
- Starting the dstack server and CLI
- Creating service configurations for vLLM deployments

**Working Service Configurations**
- Ready-to-use YAML for deploying Qwen3-30B-A3B
- GPU memory and pricing parameters
- Real deployment outputs showing what to expect

**API Integration Examples**
- Python code using the OpenAI SDK
- cURL examples for testing
- Streaming response implementations

## Why dstack + Vast.ai?

The combination gives you orchestration on top of a GPU marketplace:

- **Simplified Workflow**: No manual instance hunting or environment setup
- **Cost Optimization**: dstack finds the cheapest instance meeting your requirements
- **Flexible Pricing**: Access Vast.ai's competitive rates with automatic cost caps
- **Production-Ready API**: vLLM provides OpenAI-compatible endpoints

The guide demonstrates deploying Qwen3-30B-A3B on an H100 80GB—provisioned with a single command.

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Teams** wanting reproducible, version-controlled GPU deployments
- **Developers** building LLM applications who want simple infrastructure management
- **Anyone** tired of manually provisioning and configuring GPU instances

## Get Started

The complete guide is now available in our documentation:

**[Read: Deploy LLMs with dstack and vLLM on Vast.ai →](https://docs.vast.ai/examples/serving-infrastructure/dstack-vllm)**

Whether you're new to GPU orchestration or looking for a better way to manage LLM deployments, this guide provides everything you need to get started with dstack on Vast.ai.

Ready to try it? [Sign up for Vast.ai](https://vast.ai) and follow the guide to deploy your first model.


Learn how to deploy language models with dstack and vLLM on Vast.ai, combining declarative infrastructure with competitive GPU marketplace pricing.

Deploy LLMs with dstack on Vast.ai


DR-Tulu is AI2's open-source research agent—an 8B model that autonomously plans research strategies, searches the web, reads pages, and synthesizes comprehensive answers with citations. Unlike typical LLMs with tools bolted on, DR-Tulu was trained end-to-end with its MCP tools, making it a compelling open alternative to proprietary research APIs. This guide walks through deploying DR-Tulu on Vast.ai, where you can run the full agent stack on affordable GPU instances and integrate it into your own agentic applications.

## What Makes DR-Tulu Special?

Key features include:
- **Native MCP Integration**: Trained with web search and page reading tools from the start
- **Autonomous Research**: Plans strategies, gathers sources, and synthesizes cited answers
- **Efficient Architecture**: 8B parameters with strong research capabilities

## What's in the New Guide

Our latest documentation walks you through deploying DR-Tulu on Vast.ai with a split architecture: vLLM for GPU inference on Vast.ai, MCP backend running locally for tool execution.

**Complete Deployment Instructions**
- Instance selection with the Vast.ai CLI
- vLLM configuration with the correct version (critical for compatibility)
- MCP backend setup with required API keys
- Workflow configuration to connect the components

**Three Ways to Use DR-Tulu**
- Interactive chat for exploratory research
- Batch evaluation against built-in datasets
- Python API for programmatic integration

## Why This Architecture?

The split deployment keeps GPU-intensive inference on Vast.ai while running tool orchestration locally. This means:

- Your API keys (Serper, Jina) stay on your machine (or wherever you deploy the MCP backend)
- Model inference doesn't require sending data to third-party APIs
- You can modify the MCP backend without redeploying the model
- Lower Vast.ai costs since you only need GPU for inference

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Researchers** who need cited, source-backed answers at scale
- **Developers** building applications that require web research capabilities
- **Teams** evaluating open alternatives to proprietary research APIs

## Get Started

The complete guide is now available in our documentation:

**[Read: Running DR-Tulu on Vast.ai →](https://docs.vast.ai/examples/mcp/dr-tulu)**

Whether you're exploring agentic AI architectures or ready to deploy your own research assistant, this guide provides everything you need to get DR-Tulu running on Vast.ai infrastructure.

Ready to try it? [Sign up for Vast.ai](https://vast.ai) and follow the guide to deploy your first instance.


Learn how to deploy DR-Tulu, AI2's open-source research agent, on Vast.ai for autonomous web research with citations.

Deploy Deep Search with DR-Tulu on Vast.ai


As 2025 comes to a close, we're looking back at the milestones and updates that shaped the past year for us here at Vast.ai.

What a year it's been! From new hardware to major security achievements and product launches, here are our top moments of 2025.

## Expanding Our Global GPU Fleet

The past year saw some of the biggest hardware expansions in Vast.ai's history.

In 2025, we added NVIDIA's latest RTX 5000-series GPUs – including the RTX 5090 – followed by high-performance machines like the RTX PRO 6000 WS and the H200 and B200. These additions supported higher-throughput workloads and more efficient AI inference across a [wide range](https://cloud.vast.ai/) of hardware profiles.

Another highlight: our RTX 5090 inventory surpassed 1,000 machines this year!

## Security and Compliance Milestones

This year, Vast.ai achieved both **SOC 2 Type I** and **Type II** certification. These independent audits validated that the security infrastructure we've built over 6+ years of trusted service continues to meet rigorous compliance standards.

We take this responsibility seriously. Data privacy and integrity remain core priorities in everything we build. As we continue to advance our [security and compliance](https://vast.ai/compliance) roadmap, we're committed to upholding the trust you place in us.

## New Templates and Tools

We introduced a number of ready-to-use templates this year, to help you move from idea to execution quickly and seamlessly. Some notable ones included:

- [Langflow + Ollama](https://docs.vast.ai/langflow-ollama) for visual AI workflow design
- [Ostris AI Toolkit](https://cloud.vast.ai/?ref_id=62897&template_id=81e6e9dd266d9bdbcbb0fd48b9afe357) for custom Stable Diffusion and LoRA training
- [Open-Sora](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Open-Sora) and [LTX Video](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%2520%252B%2520LTX%2520Video) (ComfyUI) for video generation workflows
- [ComfyUI ACE Step](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ACE%2520Step%2520Music%2520Generation) for AI-assisted music creation
- [InvokeAI](https://cloud.vast.ai/?ref_id=62897&template_id=c309df384010cf9d3dc9abb9e0a5a6fd) for image generation in a professional web interface
- [Dia 1.6b TTS](https://cloud.vast.ai/?ref_id=62897&template_id=6ba071acb5f590913c5eaf6c2dc3d0db) for lifelike text-to-speech

But the biggest update of the year was still ahead...

## Introducing Vast.ai Serverless

The headline story of 2025 was the [recent launch](https://vast.ai/article/vast-ai-serverless-automated-gpu-scaling) of **Vast.ai Serverless** – a fully automated GPU compute layer designed for AI inference at scale.

With our Serverless offering, you can easily run workloads without managing instances on your own or overpaying for capacity. You simply send inference requests and the platform automatically selects the right hardware, routes workloads across regions as needed, and predictively scales up and down based on usage.

In short, Vast.ai Serverless gives you access to a diverse, globally distributed GPU fleet with a serverless model that applies predictive optimization and flexible scaling – with none of the overhead. We're thrilled to finally make truly serverless AI inference a reality on Vast.ai!

## Looking Ahead

To all of our users and the wider Vast.ai community, thank you for being part of our journey this year. It's been an incredible year of growth, and we're grateful to everyone building and innovating with us.

Here's to an even stronger year ahead!


As 2025 comes to a close, we're looking back at the milestones and updates that shaped the past year for us here at Vast.ai.

Vast.ai's 2025 Year in Review


With so many AI model libraries to choose from, making the right decision requires research and testing. Luckily, [Vast.ai](http://vast.ai) partners with some of the leading innovators across a variety of generative and assistive models. Along with our GPU rentals, we empower businesses and individuals to produce their best work without latency issues or breaking the bank. Not only are these AI models reliable, they are also very affordable. You can scale production and workload with ease. Whether looking for an image generator, video production, audio, text or computer vision research, these are some of the best AI models available.

Model Library is Vast's new, model-first way to deploy AI. Instead of starting from the runtime-level (like vLLM, ComfyUI, or PyTorch), you can browse and discover model-level templates, sorted by modality and use case, then launch a running instance in seconds with full custom control over the underlying runtime, parameters, and infrastructure.

Every template includes recommended hardware presets and clear dedicated GPU rental pricing, so you can quickly get started with the right cost/performance setup and deploy with confidence, or tweak to fit your needs.

Coming soon: more runtimes, more model variations, and Vast Serverless model templates.

## The Best AI Image Generation Models

With text-based prompts, you can produce images from a range of popular AI models. From hyper-realistic renderings to cartoonish depictions, these models are among your best bets for generating images.

- **[HiDream I1 Full](https://vast.ai/model/hidream-i1-full)**: Released in May 2025, HiDream produces images in a variety of styles. Whether you want realistic photo imagery or stylized cartoon graphics, HiDream can deliver. With speed and precision, it can produce images without sacrificing quality.

- **[Qwen Image (FP8)](https://vast.ai/model/qwen-image)**: With a specific focus on generating images containing text, Qwen Image can produce high-quality images that provide a strong eye for detail. This is true not only of images with English characters, but even intricate characters for multilingual projects. It has a particular focus on generating images containing Chinese characters.

- **[FLUX.1 [dev]](https://vast.ai/model/flux.1-dev)**: Whether prototyping concepts, marketing, or digital art, FLUX.1 is an open-source tool that rivals many closed-source competitors in its functionality and precision. Developed by Black Forest Labs, this 12-billion-parameter image generator serves for a range of use cases from concepting to marketing materials and scientific visualization.

- **[Juggernaut XI v11](https://vast.ai/model/juggernaut-xi-v11)**: With a particular emphasis on photorealistic images, Juggernaut XI v11 is trained to provide exceptional detail on challenging human features like hands and facial expressions. It also provides exceptional detail for landscapes and other detailed imagery that have historically fallen into the realm of "uncanny valley."

- **[RealVisXL V5.0](https://vast.ai/model/realvisxl-v5.0)**: Developed by Evgeny, RealVisXL renders photorealistic images and produces anatomically accurate and high-fidelity visuals thanks to inference parameters like DPM++ SDE Karras sampling. This makes it equally adept at producing architectural designs and photorealistic images.

- **[Stable Diffusion XL Base 1.0](https://vast.ai/model/stable-diffusion-xl-base-1.0)**: Developed by Stability AI, Stable Diffusion XL Base improves image quality by providing two models. With a brainstorming mode in its base model, you can parse through a variety of images and then hone in on specific imagery with the more detailed refinement module. This holistic approach provides you with everything you need to craft the specific images you need.

## The Best AI Video Generation Models

[Vast.ai](http://vast.ai) offers some of the best video generation options available on the market. With options from a variety of AI engines and developers, you can produce stunning videos that rival cost-and-labor-intensive CGI and video productions.

- **[LTX Video](https://vast.ai/model/ltx-video)**: The first Diffusion Transformer (DiT)-based video generation model, LTX Video, produces high quality videos at resolutions and framerates that exceed much of the competition. Videos are available at 30 FPS and a 1216×704 resolution. This gives it an advantage over many of its competitors.

- **[Mochi 1 Preview](https://vast.ai/model/mochi-1-preview)**: Specializing in photorealistic video generation, Mochi 1 has 10 billion parameters that are built on an innovative Asymmetric Diffusion Transformer architecture. This helps it process over 44,000 visual tokens to produce high-quality videos.

- **[Wan2.2 I2V A14B (FP8)](https://vast.ai/model/wan-2.2-i2v-a14b)**: Wan2.2 I2V generates 720p videos from static image prompts. With a high-noise expert option for background imagery and a low-noise expert option for movement, you can make still images come to life with striking detail.

- **[Wan2.2 T2V A14B (FP8)](https://vast.ai/model/wan-2.2-t2v-a14b)**: Released in July 2025, Wan2.2 T2V generates 5-second videos at both 480P and 720P resolutions. With cinematic aesthetics and complex motion capabilities, it provides greater detail than many previous open-source and commercial models.

## The Best Audio Generation AI Models

Whether you need to produce a podcast, narrate an audiobook, engineer music production, or just explore the possibilities of ai-generated sounds, [Vast.AI](http://vast.ai) provides affordable audio generation options from some of the best-known models available.

- **[ACE Step V1 3.5B](https://vast.ai/model/ace-step-v1-3.5b)**: ACE Step V1 produces realistic audio thanks to its 3.5-billion-parameter model. It can be leveraged for a range of use cases: podcasts, audiobooks, and a wide range of music production prompts.

- **[Dia 1.6B](https://vast.ai/model/dia-1.6b)**: Developed by Nari Labs, Dia 1.6B is an audio generation model that specializes in multi-speaker production. This works great for podcasts and other text-to-audio prompts and excels at inflecting speakers' voices and producing sighs, laughter, and other non-verbal flourishes.

## The Best Text Generation AI Models Available

Whether you need agentic support, assistance on scientific research, or multilingual text translation, there are many incredible choices available. These are the best text generation AI models available from [Vast.AI](http://vast.ai).

- **[GLM 4.6](https://vast.ai/model/glm-4.6)**: This model provides support for language models that work on multi-step workflows. With over 200,000 tokens and sophisticated reasoning logic, it can excel at anything from coding applications to agentic capabilities.

- **[Kimi K2 Thinking](https://vast.ai/model/kimi-k2-thinking)**: Released under a Modified MIT License, Kimi K2 Thinking supports both commercial and research applications. Its chain-of-thought reasoning allows it to operate through hundreds of steps without degrading its response quality.

- **[DeepSeek V3.2 Exp](https://vast.ai/model/deepseek-v3.2-exp)**: Built for complex research and debugging scenarios, DeepSeek V3.2 Exp is an incredible assistant for very labor-intensive processes. With the ability to parse and analyze large bodies of text, you can automate some of the most tedious parts of your day in seconds.

- **[Kimi K2 Instruct 0905](https://vast.ai/model/kimi-k2-instruct-0905)**: With over a trillion parameters, Kimi K2 Instruct excels at language models including agentic intelligence and software development. Its massive 256k token window provides advanced decision-making capabilities for a range of issues and use cases.

- **[Qwen3 235B A22B Thinking 2507](https://vast.ai/model/qwen3-235b-a22b-thinking-2507)**: Developed by Alibaba, this 235-billion-parameter Mixture-of-Experts (MoE) language model is specifically engineered for extended reasoning tasks by activating 8 experts and 22 billion parameters per token. This makes it ideal for complex projects including mathematics, scientific research, and code generation.

- **[Qwen3 Coder 480B A35B Instruct](https://vast.ai/model/qwen3-coder-480b-a35b-instruct)**: This 480-billion-parameter Mixture-of-Experts (MoE) model developed by Alibaba is designed for advanced code generation and is optimized for actionable code output by operating exclusively in an efficient non-thinking mode.

- **[DeepSeek R1 0528](https://vast.ai/model/deepseek-r1-0528)**: This advanced reasoning language model developed by DeepSeek AI utilizes advanced chain-of-thought logic for complex mathematical and logical problems. This makes it a great choice for tool and API integrations.

- **[DeepSeek V3.1](https://vast.ai/model/deepseek-v3.1)**: This hybrid language model operates in both thinking and non-thinking modes. This dual-mode architecture is equally suited for deep reasoning with visible thought processes or generating fast responses without intermediate reasoning.

- **[GPT-OSS-120b](https://vast.ai/model/gpt-oss-120b)**: GPT-OSS-120b is an open-weight reasoning model from OpenAI designed for production use cases. With a range of low, medium, and high reasoning settings it balances speed and accuracy for agentic functions.

- **[GPT OSS 20b](https://vast.ai/model/gpt-oss-20b)**: Built for lower latency and specialized use cases. For everything from web browsing to function calling and API integration tasks, GPT OSS 20b is a great choice for agentic support.

- **[Llama 4 Maverick 17B 128E Instruct](https://vast.ai/model/llama-4-maverick-17b-128e-instruct)**: With seamless processing of both text and visual inputs, Llama 4 Maverick 17B supports up to a 10 million token context length and can process up to 5 input images simultaneously. It supports 12 languages for a wide array of agentic functions including English, Spanish, Tagalog, French, German, Hindi, Indonesian, Italian, Portuguese, Arabic, Thai, and Vietnamese.

- **[Llama 4 Scout 17B 16E Instruct](https://vast.ai/model/llama-4-scout-17b-16e-instruct)**: Released in April 2025, Scout 17B 16E Instruct employs a mixture-of-experts (MoE) and has a total of 109 billion parameters. It functions as a great assistant for analyzing data, parsing large quantities of data and is compatible with 12 languages including English, Spanish, Tagalog, French, German, Hindi, Indonesian, Italian, Portuguese, Arabic, Thai, and Vietnamese.

- **[Qwen3 VL 235B A22B Instruct](https://vast.ai/model/qwen3-vl-235b-a22b-instruct)**: The most powerful vision-language model in the Qwen series, VL 235B A22B Instruct combines 236 billion parameters across visual understanding, agent applications, context processing, and multimodal reasoning tasks. It excels at recognizing a range of items from landmarks, to plants and animals, to works of art and celebrities. This makes it a great research tool for imaging, robotics, and even technical documentation.

## The Best Computer Vision Models Available

When it comes to choosing an AI model that can interpret visual elements from real-world scenarios, these are among the best:

- **[DeepSeek OCR](https://vast.ai/model/deepseek-ocr)**: This vision-language model specializes in optical character recognition and document understanding. This makes it great for extracting text from images or scanned documents, processing PDFs, and much more.

- **[GLM 4.5V](https://vast.ai/model/glm-4.5v)**: Whether summarizing longform videos, analyzing data, or scanning documents for text, GLM 4.5V provides support for advanced reasoning tasks. With so many use cases, you can even toggle between quick responses or deeper analysis of data to make it work for you the way you want.

- **[InternVL3 78B](https://vast.ai/model/internvl3-78b)**: Whether analyzing technical documentation, interpreting results from medical images, or providing agentic support, InternVL3 78B has many use cases from text-only applications to a mixture of visual applications.

- **[Llama 4 Maverick 17B 128E Instruct](https://vast.ai/model/llama-4-maverick-17b-128e-instruct)**: Featuring 17 billion activated parameters distributed across 128 total experts, this model combines text and image understanding capabilities within a unified architecture. This model provides everything from an assistant-like conversational experience to image captioning, document analysis, and multilingual support.

- **[Llama 4 Scout 17B 16E Instruct](https://vast.ai/model/llama-4-scout-17b-16e-instruct)**: With efficiency being front and center, this model runs much more leanly than Maverick. This gives it a more conversational tone while still providing agentic support, robust analysis of charts, multilingual functionality.

- **[Qwen3 VL 235B A22B Instruct](https://vast.ai/model/qwen3-vl-235b-a22b-instruct)**: Qwen3's most powerful vision-language model, it provides incredible PC and mobile GUI navigation for automation tasks, visual coding support across a range of coding languages, and agentic support.

---

If you're ready to take your AI workflows to the next level, [Vast.ai](http://vast.ai) is here to help. From an [AI model library](https://vast.ai/model-library) with a host of tools and use cases to [GPU servers](https://cloud.vast.ai/) that can accelerate your research, we're here to help empower and scale your discovery every step of the way.

---


Explore Vast.AI's comprehensive model library featuring the best AI models across image generation, video production, audio, text, and computer vision.

Vast.AI's Model Library: Choosing from the Best AI Models Available


# Running ComfyUI on Vast.ai Serverless: Simple, Scalable Image Generation

ComfyUI has become one of the most popular open-source AI tools for image generation, for good reason.

Rather than typing a prompt and hoping for the best, you build the entire process yourself through a **node-based interface that gives you full, visual control** over how images are created. Every part of the workflow – from models, prompts, and parameters – can be adjusted or branched however you like, and workflows themselves are also shareable and reusable.

While ComfyUI runs well locally for smaller projects, it's a whole other challenge to use it for heavier custom workflows or at production scale. Complex ComfyUI graphs can exceed consumer GPU memory or demand more throughput than a single machine can handle – or simply take too long to iterate on locally.

That's where **Vast.ai Serverless** comes in.

### ComfyUI Meets Vast.ai Serverless

Vast.ai Serverless makes it easy to run your ComfyUI workflows at scale, thanks to a ready-to-use template that handles all of the setup for you – and no need to manage GPUs or monitor capacity.

Simply send requests to ComfyUI and have your generated assets automatically uploaded to S3-compatible storage. The template will return pre-signed URLs in response to requests, as well as detailed process updates while the workflow runs.

It's a straightforward way to integrate ComfyUI into your applications or pipelines without worrying about infrastructure overhead.

### What Comes in the ComfyUI Serverless Template

Our ComfyUI Serverless template provides [everything you need](https://www.google.com/url?q=https://docs.vast.ai/documentation/serverless/comfy-ui&sa=D&source=editors&ust=1766188232265630&usg=AOvVaw0k4v0rLcf_0MltEhly3_ZW) right out of the box:

- **ComfyUI** and its **API wrapper** (which manages asset uploads)
- **Stable Diffusion 1.5** for benchmarking
- **PyWorker** for processing JSON workflows
- **Provisioning script** for custom configuration

The template saves you from building or maintaining your own Docker image and ensures you get a consistent environment every time a worker spins up. Once the template is in place, several capabilities come built into the workflow by default.

#### 1. Built-In Asset Handling with S3 Storage

Generated assets are automatically uploaded to your private S3-compatible storage buckets, and pre-signed URLs provide secure, temporary access to download results directly. This keeps storage and retrieval simple, even as workloads scale.

Just add your credentials once in your account settings, and every worker can use them. If you need different buckets for different jobs, you can override these variables on a per-request basis.

#### 2. Flexible Configuration When You Need It

With ComfyUI Serverless, you can keep things simple or tailor your setup with optional configurations. For instance:

- Add a webhook to be notified after generation completion or failure
- Use a custom PyWorker git repository URL
- Adjust benchmark settings to better reflect your workload

With these configuration tools, you're free to adapt the template to your unique needs.

#### 3. Distributing Work Through Benchmarking

Each worker runs a quick benchmark when it initializes, to validate GPU performance and calculate a performance score. This score is what determines how requests are distributed across workers.

The template includes a default Stable Diffusion 1.5 benchmark for this purpose, but you can configure it to match your actual workload complexity or even supply your own custom benchmark.

### Try It Out Interactively Before Going Serverless

The ComfyUI Serverless template is designed for serverless execution, of course, but you can test it as an interactive instance first. Interactive testing helps confirm that everything works as expected before you start sending requests through Serverless.

You'll have to enable port forwarding to access applications over the public interface, but the benchmarking process will be visible in the instance logs either way.

### A Smoother Path to Production-Scale ComfyUI

Vast.ai Serverless and our ComfyUI template lets you go from a local workflow to a scalable, on-demand system with minimal setup. The environment is preconfigured – with customizable adjustments as needed – and assets are handled automatically while workloads are routed to the right GPUs based on real performance.

For creative work beyond images, we offer Serverless templates for [text-to-video](https://www.google.com/url?q=https://docs.vast.ai/documentation/serverless/comfyui-wan-2.2&sa=D&source=editors&ust=1766188232275066&usg=AOvVaw33TRQTBXX80tZ040XucSdk) and [text-to-music](https://www.google.com/url?q=https://docs.vast.ai/documentation/serverless/comfyui-acestep&sa=D&source=editors&ust=1766188232275242&usg=AOvVaw3Dnzq-bbQYJjA76I3vHcU3) workflows, as well.

Ready to see ComfyUI in action? Just launch the [ComfyUI Serverless template](https://www.google.com/url?q=https://cloud.vast.ai/?ref_id%3D62897%26creator_id%3D62897%26name%3DComfyUI%2520(Serverless)&sa=D&source=editors&ust=1766188232275749&usg=AOvVaw0clIXR2u6vdhxqRdqoPmHE) on Vast.ai and start running workflows at scale. It's that easy.


ComfyUI has become one of the most popular open-source AI tools for image generation, for good reason.

Running ComfyUI on Vast.ai Serverless: Simple, Scalable Image Generation


Vast.ai is thrilled to announce the launch of our Serverless offering for GPU workloads, enabling automatic, pay-per-use scaling for AI inference without hyperscaler pricing or fixed infrastructure constraints.

With [Vast.ai Serverless](https://vast.ai/products/serverless), users run inference workloads through a fully serverless API on Vast's globally distributed GPU cloud. That means no manual instance management and no capacity planning. Instead, the platform automatically applies predictive optimization and flexible scaling, with access to a diverse GPU fleet -- so teams can deploy production AI systems that scale when needed and yet stay cost-efficient at all times.

How does this serverless model work in practice? Let's take a closer look!

<div style={{ margin: "48px auto", maxWidth: "960px", width: "100%" }}>
  <div style={{ position: "relative", paddingBottom: "56.25%", height: 0, overflow: "hidden", borderRadius: "12px", background: "#000" }}>
    <iframe
      src="https://www.youtube-nocookie.com/embed/0PAPzSZa3tA?rel=0"
      title="Vast.ai Serverless overview"
      allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
      allowFullScreen
      referrerPolicy="strict-origin-when-cross-origin"
      loading="lazy"
      style={{ position: "absolute", top: 0, left: 0, width: "100%", height: "100%", border: 0 }}
    ></iframe>
  </div>
</div>

### Where GPU Cloud Meets Serverless AI

Vast.ai Serverless provides access to a wide range of GPUs, from RTX-class consumer GPUs to enterprise-grade machines like A100s, H100s, and B200s. Rather than managing individual instances, all you have to do is define performance targets, and Vast handles the rest.

Behind the scenes, Vast.ai continuously benchmarks GPUs across a global network of more than **17,000 GPUs hosted by 1,400+ providers in over 500 locations worldwide**. As workloads run, the platform dynamically selects, provisions, and routes jobs to the most efficient hardware available at that moment.

The key is predictive optimization.

### Predictive Optimization Built for Real-World Workloads

Most serverless platforms react to demand after it appears. Vast.ai Serverless goes further.

Our predictive optimization feature analyzes historical usage patterns, real-time load, and ongoing market benchmarking to anticipate demand before it peaks. Based on those signals, the platform **proactively provisions GPU workers** that balance cost and latency -- ready to activate on demand rather than scrambling to scale once performance has already degraded.

These reserve workers help avoid the laggy cold starts and unpredictable costs common with other GPU serverless offerings. Not to mention having to pay for excess GPU capacity that isn't actively being used.

### Multiple Workergroups per Endpoint

Unlike traditional serverless systems that restrict endpoints to a single hardware profile, Vast.ai Serverless [supports multiple Workergroups per Endpoint](https://docs.vast.ai/documentation/serverless/architecture). A Workergroup is a collection of GPU workers that are managed as a logical unit to automatically adjust capacity to meet demand.

In our Serverless offering, each Workergroup in an Endpoint can specify different GPU types or hardware configurations. This lets a single API endpoint serve workloads using whichever Workergroup is most cost-effective or the best fit for your performance targets. For example, lighter requests might route to consumer GPUs, while heavier inference jobs scale onto H100s -- with no manual intervention needed.

In practice, this makes it possible to optimize performance and cost-efficiency in real time within a single deployment. And that brings us to pricing.

### Lowest-Cost Transparent Serverless Pricing

Vast.ai Serverless is the **lowest-cost autoscaling GPU cloud** on the market today.

Our Serverless workloads are billed per second, with support for On-Demand, Interruptible, and Reserved pricing. There are [no tiers and no limits](https://docs.vast.ai/documentation/serverless/pricing) -- and all you need is $5 to get started.

Because Vast draws from a competitive global market of GPU providers, pricing reflects real supply and demand rather than preset SKUs. As demand scales up, Vast.ai Serverless automatically selects the most cost-effective GPUs available. When demand drops, resources are released immediately, and billing stops.

As a result, teams can run production workloads at a fraction of the cost compared to traditional providers with centralized infrastructure. However, lower costs don't mean lower standards. Security and compliance remain core to our platform.

### Enterprise-Ready Security and Compliance

Vast.ai is built with security and compliance as foundational principles. Our platform is backed by **SOC 2 Type II** [certification](https://vast.ai/article/vast-soc2-typeII-certification), with ongoing audits every 12 months to ensure continual coverage.

For customers with the highest security requirements, Vast.ai offers a tiered compliance structure with our [Secure Cloud](https://cloud.vast.ai/?secure_cloud=true) offering. In the Secure Cloud, workloads run on isolated instances with direct SSH, CLI, and API access, hosted by our vetted datacenter partners that meet **ISO 27001** standards at minimum. Many partners maintain additional certifications. (Full details are available on our [compliance page](https://vast.ai/compliance).)

Other enterprise security features -- including private VPN access, optional audit trails, and enterprise-grade compliance support -- can be enabled as needed.

In all cases, data sovereignty remains fully in your control. Models, data, and workloads persist only as long as you choose and can be deleted at any time.

### How to Automatically Scale GPU Compute for AI Inference

With Vast.ai Serverless, you can go from [zero to compute](https://docs.vast.ai/documentation/serverless) in seconds. In short, here's how the process works:

1. Bring your own machine learning model
2. Send an inference request
3. Vast.ai Serverless routes and provisions compute on ready GPUs
4. GPU executes and returns results
5. Serverless predictively scales up and down based on usage

That's all there is to it! From request to response, Vast.ai handles all of the heavy lifting. You also gain access to ample metrics and debugging tools, including logs and Jupyter/SSH.

### Get Started with Vast.ai Serverless

Our Serverless offering makes it easy to run AI inference without managing infrastructure or overpaying for capacity. Vast.ai Serverless gives you a simple and straightforward path from experiment to production -- and it's ready when you are.

Leverage Vast's distributed GPU fleet on your own terms. Check out our [Serverless Overview](https://docs.vast.ai/documentation/serverless) and start building today!


Announcing Vast.ai Serverless for GPU inference with predictive optimization, multi-workergroup endpoints, pay-per-second pricing, and enterprise-grade compliance.

Vast.ai Serverless: Automated GPU Scaling for AI Inference - Without the Overhead


All startups rely on infrastructure that can grow with them, but AI startups in particular face an added challenge: supporting compute-heavy workloads and iterating fast without slowing progress or overspending. 

Whether you're bootstrapping or venture-funded, it's always best to maximize the return on every GPU hour and dollar spent. But how do you do that? 

For AI startups, the right GPU infrastructure is absolutely critical. Training large models and running compute-heavy experiments can quickly consume budgets if you're not careful. Traditional cloud providers often charge a premium for the level of compute that AI workloads demand, while locking users into long-term contracts or inflexible pricing models. 

Buying your own dedicated hardware rarely solves the problem. The high upfront cost can be prohibitive for early-stage teams, and delays can stall progress. You also risk being tied to specific hardware that may be inadequate for your startup's next phase of development.

At Vast.ai, we take a different approach. Our cloud GPU rental platform offers high-performance machines with flexible spot pricing, so you can get the compute you need, when you need it, as affordably as possible.

Let's take a closer look at what makes Vast.ai's [spot GPU rental](https://cloud.vast.ai/) an ideal choice for AI startups.

## The Cost Advantage of Spot GPUs for AI Startups

To keep cloud costs under control as an AI startup, it's important to carefully consider which [pricing model](https://docs.vast.ai/documentation/instances/choosing/instance-types) is right for you. Vast.ai offers on-demand GPU instances as well as spot instances – each balancing cost and reliability in different ways.

**On-demand instances** offer guaranteed uptime at a fixed price, so you pay a slightly higher rate to access a GPU without the risk of sudden interruption. This option is well suited for continuous training jobs or production inference, or basically any workload where you need to prioritize stability over cost.

**Spot instances**, also known as interruptible instances on [Vast.ai](https://vast.ai), let you rent unused GPU capacity at lower rates determined by bidding. The highest bid wins the instance. The trade-off is the possibility that your instance may be interrupted or preempted with little to no notice, requiring you to restart the job later (when you regain priority) or move it to another GPU. This option is nevertheless ideal for workloads that can tolerate occasional interruptions in exchange for significant cost savings. 

For AI startups, the main draw of spot instances is simple: they deliver the same high-performance GPUs at a drastic discount. On Vast.ai, interruptible instances can be **up to 80% cheaper** than traditional cloud rates.

These savings can give early-stage AI startups the momentum to scale faster. The next move is scaling smarter.

## How Spot GPUs Help AI Startups Scale Smarter

As a startup, you want to accelerate your AI development thoughtfully and efficiently. Lowering your cloud GPU costs is the first step, and then you can reinvest those savings to accelerate growth.

With spot pricing, access to high-performance GPUs remains cost-effective. As a result, larger-scale training becomes affordable enough for early-stage AI startups to run more jobs and test new ideas efficiently – ultimately iterating faster. 

Additionally, because spot instances are billed on a pay-as-you-go basis, you can run workloads on cutting-edge GPUs that may otherwise be out of reach – like the H200 – for limited periods as needed, and then scale back down quickly.

Lower costs and operational agility translate into a competitive edge that can make all the difference for an early-stage startup. 

From there, how can you make spot instances work best for you? Here are a few quick tips:

**Use spot GPUs for flexible, restartable jobs.** Batch data processing and any experiments or workloads that can resume from checkpoints are excellent options when you're using interruptible instances on Vast.ai. 

**Make sure your workflows are tolerant of interruptions, and if you can, automate recovery.** With spot or interruptible instances, disruptions will almost certainly happen on occasion, and you'll want to be ready for that.

**Switch to on-demand instances for critical workloads.** Live inference and customer-facing applications are good candidates for on-demand. Anytime you want to minimize productivity loss from an interruption (for example, when a member of your team is actively debugging or deploying code), guaranteed uptime may be worth the higher cost.

## The Bottom Line

Interruptible instances on Vast give AI startups a cost-effective way to iterate quickly and scale smarter. By delivering the same high-performance hardware at a fraction of the cost, spot GPUs help extend your runway and free up budget to reinvest in growth. 

To learn more about why startups of all stripes choose Vast.ai, check out our previous post about how our platform is tailored to meet your needs from day one.

Ready to put the Vast.ai advantage to work for you? Explore our [spot GPU options](https://cloud.vast.ai/) and see how much faster – and smarter – your AI startup can scale!


For AI startups working with limited budgets and evolving product requirements, spot (interruptible) GPUs offer a flexible, cost-effective path to building and scaling AI infrastructure without breaking the bank.

Starting Smart: Why Spot GPUs Are Ideal for AI Startups


When you're training a custom AI model, access to the right GPU compute resources is crucial. That can be a tall order – but you don't have to drain your budget or get locked into long-term contracts to get the compute power you need. With **Vast.ai**, there's a better way.

Our cloud GPU rental platform gives you direct access to high-performance GPUs through a transparent, pay-as-you-go model that lets you train on your own terms: anytime, anywhere, and at the right price.

In this post, we'll take a closer look at how Vast.ai helps you train custom AI models faster and more affordably, without compromising performance or scalability.

## 1. Flexible, Budget-Friendly Pricing

Training AI models can get expensive. Vast.ai offers different [pricing options](https://docs.vast.ai/instances/rental-types) to help you keep cloud costs under control:

**Interruptible instances** let you rent GPUs at lower prices through interruptible or bid-based access. While these instances can be interrupted with little notice, they're a great fit for workloads that can handle brief disruptions in exchange for major cost savings – **up to 80% cheaper** than traditional cloud rates. For many users, that tradeoff is worth it for the reduction in training costs.

**On-demand instances** give you uninterrupted GPU access at a fixed rate. This setup is best for continuous training, real-time inference, or workloads where stability is crucial.

Of course, price is just one part of the equation: the next step is choosing the right GPU for the job. Vast.ai gives you that flexibility, as well.

## 2. GPU Options for Every AI Workload

There's no one-size-fits-all solution when you're training custom AI models. That's why our wide range of cloud GPU options is such an advantage. You can choose exactly what your workload needs, whether that's an enterprise-grade accelerator like the **NVIDIA [A100](https://vast.ai/article/H100-vs-A100-Comparing-two-Powerhouse-GPUs)** or **[H200](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights)**, an ultra-high-performance machine like the **[B200](https://vast.ai/article/nvidia-h200-vs-b200-comparing-datacenter-grade-accelerators)**, or a top-tier consumer card like the **[RTX 4090](https://vast.ai/article/nvidia-rtx-4090-vs-a100-two-powerhouses-two-purposes)**, **[5090](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know)**, or **[L40S](https://vast.ai/article/comparing-nvidia-l40-vs-l40s-and-more)**.

Each listing on Vast.ai displays **transparent instance specs**, making it easy to compare GPUs by type, RAM, storage, location, and more. Simply filter and select the best fit for you!

Most importantly, you'll never have to worry about overpaying for capacity you don't need or being held back by hardware limitations. When your workloads grow, scaling up is seamless. Vast.ai lets you spin up multiple GPU instances and customizable, on-demand [GPU/CPU clusters](https://vast.ai/clusters), so you can expand training capacity instantly – and only pay for what you use.

## 3. Supporting Real-World Use Cases

Our platform is ideal for anyone training, fine-tuning, or experimenting with custom AI models. Whether you're an AI researcher or developer, an enterprise team, or a [startup](https://vast.ai/article/7-reason-why-startups-uses-vast-ai), Vast.ai provides the compute power you need to train models for use cases like automating customer inquiries, identifying sales trends and other business insights, and analyzing massive research datasets.

You can build and refine domain-specific models, fine-tune foundation models like Llama and Mistral on proprietary data, and retrain as your datasets evolve. With support for LLMs and diffusion models, our platform is **compatible with all major machine learning (ML) frameworks** – and you can even run custom containers and tooling.

## 4. Quick Start with Pre-Built Templates

You don't have to start from a blank slate. Vast.ai offers pre-built [templates](https://docs.vast.ai/templates) for popular frameworks – **PyTorch, TensorFlow, vLLM, ComfyUI, and more** – that are designed to help you launch a training environment in minutes.

Each template includes the dependencies and configurations you need for model training or fine-tuning, so you can spend less time setting up and more time improving your results. Customize templates with your own datasets, scripts, or containers. You have complete flexibility to tailor them to your project needs!

Finally, Vast.ai also ensures that you can train your AI models in a trusted environment [with control over data exposure](https://vast.ai/article/running-private-ai-models-without-the-risk-of-data-exposure).

## 5. Enterprise-Grade Security

With Vast.ai, your workloads run on a secure, trusted infrastructure designed to [protect your data](https://vast.ai/compliance) – crucial when training custom AI models. Choose either **verified hosts** with strong reliability metrics or **vetted datacenter partners** who maintain third-party compliance certifications and are bound by comprehensive Data Processing Agreements (DPAs) with Vast.ai.

Our **Secure Cloud** offering gives you the option to select GPU infrastructure only from our vetted datacenter partners. Each of these providers is located in a certified facility that's current with ISO 27001 and/or Tier 3/4 standards at minimum.

We've also [completed](https://vast.ai/article/vast-soc2-typeII-certification) our own SOC 2 Type II certification, affirming our adherence to industry-leading compliance frameworks.

## Get Started with Vast.ai

Training custom AI models shouldn't be limited to those with deep pockets or powerful infrastructure. **Vast.ai** levels the playing field.

With high-performance GPUs, flexible pricing, and pre-configured templates to help you get started, you can move from idea to implementation faster and more affordably than ever with Vast.ai. You'll also enjoy peace of mind knowing that your data and models are protected by enterprise-grade security every step of the way.

Ready to train your own model? Spin up your first instance today at Vast.ai!


When you're training a custom AI model, access to the right GPU compute resources is crucial. Vast.ai gives you direct access to high-performance GPUs through a transparent, pay-as-you-go model that lets you train on your own terms.

Why Choose Vast.ai to Train Custom AI Models


We're excited to announce a new deployment guide for MiniMax-M2, the breakthrough 230 billion parameter language model that's making waves in the open-source AI community. If you've been looking for a cost-effective way to run state-of-the-art LLMs, this guide is for you.

## What Makes MiniMax-M2 Special?

MiniMax-M2 achieves the #1 composite score among open-source models while maintaining incredible efficiency. Despite its 230B parameter size, it activates only 10B parameters per inference, delivering fast responses without the typical computational overhead of large models.

Key features include:
- **Interleaved Thinking**: Transparent reasoning via `<think>...</think>` tags
- **131K Token Context**: Handle large documents and complex conversations
- **OpenAI Compatible**: Drop-in replacement for existing applications
- **MIT Licensed**: No restrictions on commercial use

## What's in the New Guide

Our latest documentation example walks you through deploying MiniMax-M2 on Vast.ai from start to finish. The guide includes:

**Complete Deployment Instructions**
- Hardware requirements and instance selection
- Step-by-step provisioning with the Vast.ai CLI
- API key authentication for secure deployments
- Configuration parameters validated through real deployments
- Expected timelines and what to watch for

**API Integration Examples**
- Python code using the OpenAI SDK with authentication
- cURL examples for quick testing
- Streaming response implementations
- Best practices for multi-turn conversations

**Troubleshooting Section**
Based on real deployment testing, we've documented the four critical issues you might encounter and their solutions:
- Docker image version requirements
- Disk space allocation
- GPU memory optimization
- CUDA driver compatibility

**Performance Data**
All metrics in the guide come from actual deployments on Vast.ai infrastructure:
- 30-minute initial deployment time
- ~7 seconds for 100-token responses
- 131K token context window
- Production-ready performance on 4x H100 GPUs

## Why Deploy on Vast.ai?

Vast.ai's GPU marketplace gives you access to enterprise hardware at competitive rates, making it economical to run powerful models like MiniMax-M2 for production workloads. Our guide uses a 4x H100 (80GB) configuration with the vLLM nightly build. You get:

- Enterprise GPU access (H100 recommended for best compatibility)
- Predictable costs with no usage-based pricing
- Full control over your infrastructure
- OpenAI-compatible API for easy integration

The guide shows you how to leverage these advantages for cost-effective LLM inference that rivals cloud API quality.

## Scaling for Production

While the 4x H100 configuration demonstrated in our guide provides an excellent starting point for deploying and testing MiniMax-M2 on Vast.ai, production deployments typically require larger GPU configurations to support longer context lengths and higher concurrent request volumes. For production use cases, consider configurations such as 8x H100, 4x H200, or 8x H200, which provide substantially more GPU memory for handling concurrent requests and extended context windows.

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Developers** building AI-powered applications who want cost control and flexibility
- **Research teams** running experiments that need high-volume inference
- **Startups** prototyping AI features without committing to expensive cloud APIs
- **Anyone** curious about deploying open-source LLMs at scale

## Get Started

The complete guide is now available in our documentation:

**[Read: Running MiniMax-M2 on Vast.ai →](https://docs.vast.ai/examples/text-generation/minimax-m2#minimax-m2)**

Whether you're exploring open-source AI options or ready to deploy your first large language model, this guide provides everything you need to get MiniMax-M2 running on Vast.ai infrastructure.

Ready to try it? [Sign up for Vast.ai](https://cloud.vast.ai) and follow the guide to deploy your first instance.

---

*All performance metrics and cost estimates in this article are based on actual deployment testing on Vast.ai infrastructure. Results may vary based on instance availability and configuration.*


Learn how to deploy MiniMax-M2, the 230B parameter language model with state-of-the-art performance, on Vast.ai infrastructure.

New Guide: Deploy MiniMax-M2 on Vast.ai

	
Choosing an RTX 6000 GPU isn't quite as straightforward as it might seem. The name "RTX 6000" now spans multiple cards across two architectures – the NVIDIA RTX 6000 Ada, as well as three newer Blackwell-based variants built for very different workloads. The four GPUs share some impressive fundamentals, but their performance profiles and a few important design differences set them apart.

Today we're looking at these NVIDIA RTX 6000 GPU options, what they're built for, and how to decide which one best fits your needs. So let's get into it!

### The RTX 6000 Family at a Glance

For starters, here's a brief overview of the four GPU models in the RTX 6000 lineup:

-   [RTX 6000 Ada Generation](https://cloud.vast.ai/?gpuModelNames=rtx6000AdaGeneration&instanceDiskSizeMin=700&instanceDurationMin=604800&instanceType=onDemand&isAvxSupported=false&isHostSecure=false&isMachineIpStatic=false&isOfferAvailable=true&isOfferCompatible=true&isOfferVerified=true&isQueryInverted=false&machineReliabilityMin=0.9&sorts=score-desc) – An enthusiast-class professional graphics card with solid all-around performance.

-   [RTX PRO 6000 Blackwell Workstation Edition](https://cloud.vast.ai/?gpuModelNames=rtxPro6000BlackwellWorkstation&instanceDiskSizeMin=700&instanceDurationMin=604800&instanceType=onDemand&isAvxSupported=false&isHostSecure=false&isMachineIpStatic=false&isOfferAvailable=true&isOfferCompatible=true&isOfferVerified=true&isQueryInverted=false&machineReliabilityMin=0.9&sorts=score-desc) – A full-power 600W desktop card, best used for single-GPU high-throughput workflows.

-   **RTX PRO 6000 Blackwell Max-Q Workstation Edition** – Basically the same hardware as the standard Workstation Edition above, but with a few design differences that make it better suited for multi-GPU setups.

-   [RTX PRO 6000 Blackwell Server Edition](https://cloud.vast.ai/?gpuModelNames=rtxPro6000BlackwellServer&instanceDiskSizeMin=700&instanceDurationMin=604800&instanceType=onDemand&isAvxSupported=false&isHostSecure=false&isMachineIpStatic=false&isOfferAvailable=true&isOfferCompatible=true&isOfferVerified=true&isQueryInverted=false&machineReliabilityMin=0.9&sorts=score-desc) – A passive-cooled variant designed for rackmount systems and high-density environments.

Next, we'll break down each GPU in more detail, then wrap up with some benchmarks and a side-by-side comparison table to help guide your decision.

### RTX 6000 Ada Generation: Solid and Dependable

The RTX 6000 Ada is the previous-generation flagship graphics card, and it still holds its own as a solid, versatile workstation GPU. Built on the Ada Lovelace architecture, it delivers strong performance for enhanced content creation and light-to-moderate AI training.

Overall, it's a well-rounded option. Featuring 48GB of GDDR6 ECC memory, a respectable 18,176 CUDA cores, up to 91.1 TFLOPS of single-precision performance, and 300W TDP, the RTX 6000 Ada can handle complex tasks and multi-application workflows without the more extreme price or power demands of a Blackwell-class GPU. It remains dependable for many professional use cases, especially if your projects don't require cutting-edge AI acceleration.

The RTX 6000 Ada Generation may be the right choice for you if:

Your workloads span creative applications, 3D tools, and mixed GPU tasks.

You don't need 96GB of VRAM but still want professional-grade stability and performance.

You're looking for a more cost-effective alternative to Blackwell for workstation use.

### RTX PRO 6000 Blackwell Workstation: Top-Tier Desktop Performance

The RTX PRO 6000 Blackwell Workstation Edition is the top-performing desktop card in this group. Built on NVIDIA's Blackwell architecture, it boasts 96GB of GDDR7 memory, a wide 512-bit memory bus, and extremely high memory bandwidth at nearly 1.8 TB/s. Plus, it has a staggering 125 TFLOPS of single-precision performance. This combination makes it well suited for the most demanding AI workloads as well as complex visualization, simulation, and neural rendering tasks.

The Workstation Edition can sustain peak performance under a 600W power load, without thermal throttling – but at the same time, it's not recommended for dense multi-GPU configurations due to its higher heat output and flow-through dual-fan design. Instead, it works best as a single GPU in a workstation setup.

The RTX PRO 6000 Blackwell Workstation Edition makes sense for you if:

You want the highest sustained throughput the RTX PRO 6000 series can deliver in a workstation, for the most demanding workloads.

You don't need to run multiple GPUs in the same system.

Your workstation has the power and cooling to support a 600W GPU.

### RTX PRO 6000 Blackwell Max-Q Workstation: Multi-GPU Friendly

RTX PRO 6000 Blackwell Max-Q Workstation Edition is ideal for anyone who wants Blackwell-level performance in a cooler, more efficient card that works well in multi-GPU configurations. For simplicity's sake, let's call it the Max-Q variant as opposed to the standard Workstation Edition.

The Max-Q features the same basic hardware as the full-power standard Workstation Edition – including 96GB of GDDR7 memory, a 512-bit bus, and 1.8 TB/s memory bandwidth – but it runs at a reduced 300W TDP and uses a blower-style cooler. Its enclosed single-fan design pulls air in and then exhausts it out the rear of the chassis. With lower heat output and superior management of airflow, the Max-Q is a much better fit for multi-GPU configurations than the standard Workstation Edition.

You may want to go with the Max-Q Workstation model if:

You're looking for Blackwell-class performance and memory capacity in a cooler, more efficient card.

You can't support a 600W GPU but still need 96GB of VRAM and high bandwidth.

You plan to run 2-4 GPUs and your workloads benefit from multi-GPU scaling.

### RTX PRO 6000 Blackwell Server Edition: Scalable and Server-Ready

The RTX PRO 6000 Blackwell Server Edition is built specifically for datacenter and rackmount environments. It lacks an active cooling fan, instead relying entirely on external airflow with a passive heatsink. It can also be configured up to the full 600W power envelope – as long as your airflow solution can handle the cooling required, of course!

Like the other Blackwell variants, the Server Edition includes 96GB of GDDR7 memory and a 512-bit bus. Its memory bandwidth is slightly lower at 1.6 TB/s, compared to 1.8 TB/s on the workstation cards, but the Server Edition is still more than capable of handling large models, massive datasets, and multi-GPU workloads. What really sets it apart is its ability to deliver stable, scalable performance in high-density environments.

Choose the RTX PRO 6000 Blackwell Server Edition if:

You need stable performance for continuous workloads across large-scale multi-GPU setups.

You want Blackwell-level memory and bandwidth in a form factor designed for dense compute.

You're deploying into rackmount servers where the chassis provides adequate high-pressure airflow.

### RTX 6000 Series: Performance Benchmarks

Now that we've covered design differences and use cases, let's check out how these GPUs actually perform. The benchmarks below show how each RTX 6000 variant handles real workloads, so you can get a clearer picture of what to expect in practice.

![RTX 6000 benchmark comparison](/uploads/vast_blog_6000_comparison_benchmarks.png)

And finally, for reference, here's a side-by-side specs comparison table showing how the four GPUs stack up against each other:

<table class="w-full border-collapse border border-gray-300">
  <thead>
    <tr class="bg-gray-100">
      <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
        Spec
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX 6000 Ada
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX PRO 6000 Blackwell Workstation
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX PRO 6000 Max-Q
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX PRO Blackwell Server Edition
      </th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Architecture
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Ada Lovelace</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Blackwell</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Blackwell</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Blackwell</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        VRAM
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">48GB GDDR6</td>
      <td class="border border-gray-300 px-4 py-2 text-center">96GB GDDR7</td>
      <td class="border border-gray-300 px-4 py-2 text-center">96GB GDDR7</td>
      <td class="border border-gray-300 px-4 py-2 text-center">96 GB GDDR7</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        ECC
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Memory Bandwidth
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">960 GB/s</td>
      <td class="border border-gray-300 px-4 py-2 text-center">1,792 GB/s</td>
      <td class="border border-gray-300 px-4 py-2 text-center">1,792 GB/s</td>
      <td class="border border-gray-300 px-4 py-2 text-center">1,597 GB/s</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Memory Interface
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">384-bit</td>
      <td class="border border-gray-300 px-4 py-2 text-center">512-bit</td>
      <td class="border border-gray-300 px-4 py-2 text-center">512-bit</td>
      <td class="border border-gray-300 px-4 py-2 text-center">512-bit</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        CUDA Cores
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">18,176</td>
      <td class="border border-gray-300 px-4 py-2 text-center">24,064</td>
      <td class="border border-gray-300 px-4 py-2 text-center">24,064</td>
      <td class="border border-gray-300 px-4 py-2 text-center">24,064</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Tensor Cores
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">568 (4th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">752 (5th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">752 (5th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">752 (5th gen)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Ray Tracing Cores
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">142 (3rd gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">188 (4th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">188 (4th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">188 (4th gen)</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Single-Precision Performance
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">91.1 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">125 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">110 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">120 TFLOPS (FP32)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        RT Core Performance
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">210.6 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">380 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">333 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">355 TFLOPS</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Total Board Power
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">300W</td>
      <td class="border border-gray-300 px-4 py-2 text-center">600W</td>
      <td class="border border-gray-300 px-4 py-2 text-center">300W (power-optimized)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">400–600W (configurable)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Form Factor
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">4.4" H x 10.5" L, dual slot</td>
      <td class="border border-gray-300 px-4 py-2 text-center">5.4" H × 12" L, dual-slot, extended height</td>
      <td class="border border-gray-300 px-4 py-2 text-center">4.4" H × 10.5" L, dual-slot</td>
      <td class="border border-gray-300 px-4 py-2 text-center">4.4" H × 10.5" L, dual-slot</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Thermal (Fan Configuration)
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Single-fan active</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Dual-fan active</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Single-fan active</td>
      <td class="border border-gray-300 px-4 py-2 text-center">No-fan passive (server chassis)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Date Released
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Dec. 2022</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Mar. 2025</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Mar. 2025</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Mar. 2025</td>
    </tr>
  </tbody>
</table>

### Final Thoughts

Each RTX 6000 variant brings something different to the table. As we explored above, the right choice ultimately comes down to how you plan to use it.

To sum things up, the Ada model is solid and dependable for a broad range of workloads, while the full-power Blackwell Workstation card offers the strongest single-GPU performance. The Max-Q variant is the more practical choice for lower-power or multi-GPU setups, and the Server Edition is designed for dense deployments and continuous workloads.

At the same time, these GPUs aren't exactly budget-friendly. That's where Vast.ai comes in. Our cloud GPU rental platform allows you to access this level of performance on demand, eliminating the upfront cost of purchasing the hardware yourself.

With Vast, you can easily match your compute to your workload and only pay for what you use. Best of all, you'll save 5-6X on average compared to traditional cloud providers.

Ready to see what Vast.ai can do for you? Spin up an RTX 6000 GPU instance today, or explore other powerful GPU options like the A100, H200, and B200 – available right now!


A practical guide to every RTX 6000 option, from Ada to the newest Blackwell workstation and server GPUs.

Which NVIDIA RTX 6000 Is Right for You?


When you think of a large language model (LLM), you probably picture the big household names like ChatGPT, Claude, Gemini, and Llama. They're dominant players for a reason – delivering cutting-edge performance that sets the standard for general-purpose AI – but they're also overwhelmingly centered around the English language and Western culture.

If you look further, there are plenty of other models gaining traction globally. Fueled by government investment as well as tech giants, startups, and research labs, these regional LLMs are trained on local languages and optimized for diverse needs.

With that in mind, let's take a look at some examples from around the world.

### Middle East: Arabic-First AI

Across the Middle East, government partnerships and large-scale investments are driving the rapid expansion of the region's AI sector. Much of this effort is focused on Arabic-first and bilingual LLMs designed for regional sovereignty and cultural awareness. For instance, the following are some of the leading projects in the region:

*   Jais offers open bilingual models up to 30B parameters, trained on large Arabic-English datasets, with the ability to switch between Arabic dialects. Instruction-tuned "Chat" variants, optimized for conversational applications, have also been released.

*   ALLaM powers Saudi Arabia's popular HUMAIN Chat conversational assistant and emphasizes Arabic fluency and cultural nuance. Its series of models is designed to support the ecosystem of Arabic Language Technologies (ALT).

*   Falcon, Abu Dhabi's flagship open LLM series, ranges from 7B to 180B parameters and combines commercially permissive licensing with strong performance that prioritizes resource efficiency – making it popular for academic, commercial, and government uses.

### Asia-Pacific: Scale, Localization, and Linguistic Diversity

The AI ecosystem in the Asia-Pacific region spans the largest and most linguistically diverse populations in the world. Both government-backed initiatives and major tech firms are creating models that serve dozens of languages and dialects while delivering performance that in many cases stands on par with the most advanced models worldwide.

*   [DeepSeek](https://vast.ai/article/serving-deepseek-models-on-vast-ai-with-vllm-and-langchain), one of the most capable open-source models to emerge from China, uses a Mixture-of-Experts (MoE) architecture to balance accuracy and computational efficiency. It offers more fluent support for Asian languages, with cultural nuance that may be missed by Western-centric models.

*   [Qwen](https://vast.ai/article/alibabas-qwen-an-open-source-ai-model-that-surpasses-deepseek), developed by Alibaba, covers 119 languages and dialects, with specialized multimodal variants and built-in support for Model Context Protocol. It features hybrid "thinking" and "non-thinking" modes for flexible reasoning – and you can even set a "thinking budget" to control how much computational effort it devotes to reasoning versus how quickly it replies.

*   tzusumi is a lightweight Japanese LLM (0.6B and 7B parameters) designed for on-prem use, offering strong Japanese-language processing and multimodal support with minimal hardware requirements.

*   Solar Pro 2 offers frontier performance in a comparatively compact 31B model, with multilingual capabilities and standout fluency in Korean. It's designed for real-world agent-like workflows across domains.

*   SeaLLM is a multilingual model built for Southeast Asia that outperforms most open-source LLMs across domains like science, physics, and economics in the languages of the region. Notably, it supports non-Latin scripts and low-resource languages like Lao and Khmer.

### Europe: Responsible AI and Data Sovereignty

In Europe, AI development tends to emphasize data sovereignty and compliance. The region has produced a number of LLMs that serve multilingual users and align with the EU AI Act and GDPR while competing with U.S. and Asian counterparts in raw performance.

*   Mistral, developed in France, offers a suite of open and proprietary models for reasoning, multimodal, coding, and other applications, including its flagship Mistral Large 2 and the open-source Devstral family – and even a multilingual chatbot called Le Chat that boasts deep research and image editing capabilities.

*   Luminous by Aleph Alpha is a multilingual model family (13B–70B parameters) built for enterprise and sovereign-AI use. It emphasizes compliance and transparency and provides support for English, German, French, Italian, and Spanish. The company also offers Pharia, a newer generation of models designed for concise, length-controlled responses.

*   BLOOM is a 176 billion-parameter open-access model produced through a year-long volunteer research effort to provide a transparent alternative to proprietary AI models. It can generate text in 46 natural languages and 13 programming languages.

### Other Regions: Emerging Innovation

Outside the major AI centers, new regional efforts are gaining momentum. For instance, Latam-GPT is an open initiative by Chile's nonprofit CENIA to develop a collaborative AI model completely within and for Latin America – trained on regional data and languages. The project's goal is to advance technological independence and cultural representation across the region.

In Africa, InkubaLM (named after the IsiZulu term for a dung beetle) addresses the need for lightweight African-language models. Trained on English and French as well as five African languages – Hausa, Swahili, IsiXhosa, IsiZulu, and Yoruba – the compact 0.4B-parameter model delivers efficient natural-language capabilities without heavy compute demands.

These projects reflect how emerging regional players are similarly building language models that prioritize inclusion and linguistic diversity, broadening the reach of global AI innovation.

### How to Choose a Regional LLM

Choosing a regional model depends on your goals and infrastructure. Here are a few considerations to keep in mind:

*   Look for open-weight models if you're prioritizing sovereignty or on-prem use.

*   Pick the right model size for your infrastructure! Many regional LLMs offer compact variants.

*   Consider dialect coverage and code-switching capabilities (e.g., training on colloquialisms) – along with nuanced cultural awareness – for social and chat applications.

*   If your use case requires meeting strict data protection and compliance standards, EU-built models developed with these priorities in mind may be ideal.

Once you've found the right fit, the next step is putting these models to work!

### The Bottom Line

Regional LLMs reflect how AI is evolving across a global spectrum of languages, cultural norms, regulatory priorities, and specialized technical needs.

Ultimately, they represent a more inclusive approach to model development. In turn, this helps bridge the AI gap for underrepresented groups worldwide. For developers and organizations, these LLMs provide context-aware and culturally fluent performance that's both versatile and cost-efficient.

With platforms like [Vast.ai](https://vast.ai), you don't need massive in-house infrastructure to take full advantage of these models. Our affordable cloud GPU rental makes it easy to train, fine-tune, and deploy regional LLMs at scale – helping you find just the right fit for your use case and bring AI closer to the communities and markets you serve.

> **Start experimenting with regional LLMs on [Vast.ai](https://vast.ai) today!**


From Jais to Qwen, Falcon to BLOOM - a look at how regional large language models are reshaping AI across languages, cultures, and continents.

Global Growth of Regional Large Language Models (LLMs)


The world’s smallest desktop AI supercomputer is now available in the form of the NVIDIA DGX™ Spark. Originally [announced in January 2025](https://www.theverge.com/news/798775/nvidia-spark-personal-ai-supercomputer) under the name Digits, NVIDIA CEO Jensen Huang praised Spark as a tool for data scientists, AI researchers, and students “to engage and shape the age of AI.”

Powered by a 240 watt power supply that uses a standard wall outlet and with a golden asteroid-like textured casing, it looks more like a high-end PC than an AI supercomputer. Only an inch wider and longer than the M4 Mac Mini, Spark measures just 6” x 6” x 2” and weighs just 2.6 pounds. This compact size makes it all the more impressive that it can fuel enterprise AI development with ease. If you’re looking for a mini AI companion to run local AI models, the NVIDIA DGX Spark is the best-looking, most-talked-about option on the market. But does it deliver on the hype?

### **Inside the NVIDIA DGX Spark**

Spark runs on a 20-core ARM64 CPU and a NVIDIA GB10 [Grace Blackwell](https://vast.ai/article/report-nvidia-developing-new-blackwell-based-ai-chip-for-china) GPU. It’s capable of handling AI models with up to 200 billion parameters at a million billion calculations per second. Its 128GB of unified memory can be seamlessly allocated to both its CPU and GPU without static partitioning. Plus, its stock 4TB NVME.M2 SSD hard drive provides more storage than many of its closest competitors.  
With four USB-C ports and an HDMI port, you can plug it into an array of peripheries, monitors, or even external GPUs. You can also connect two Sparks together and run them simultaneously to double your performance.

#### NVIDIA DGX Spark Technical Specifications

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Specification
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                NVIDIA DGX Spark
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2">
                NVIDIA Grace Blackwell
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">GPU</td>
            <td class="border border-gray-300 px-4 py-2">
                Blackwell Architecture
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">CPU</td>
            <td class="border border-gray-300 px-4 py-2">
                20 core Arm, 10 Cortex-X925 + 10 Cortex-A725 Arm
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                CUDA Cores
            </td>
            <td class="border border-gray-300 px-4 py-2">
                Blackwell Generation
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Performance<sup>1</sup>
            </td>
            <td class="border border-gray-300 px-4 py-2">1 PFLOP</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                System Memory
            </td>
            <td class="border border-gray-300 px-4 py-2">
                128 GB LPDDR5x, coherent unified system memory
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">OS</td>
            <td class="border border-gray-300 px-4 py-2">NVIDIA DGX™ OS</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                System Dimensions
            </td>
            <td class="border border-gray-300 px-4 py-2">
                150 mm L x 150 mm W x 50.5 mm H
            </td>
        </tr>
    </tbody>
</table>

See full specs at [NVIDIA.com](https://www.nvidia.com/en-us/products/workstations/dgx-spark/).

### **Should You Buy the NVIDIA DGX Spark?**

Following a delayed launch, Spark was released earlier this month. It is [available on](https://www.nvidia.com/en-us/products/workstations/dgx-spark/) [NVIDIA.com](http://NVIDIA.co) and other retailers for $3,999. However, Spark was initially slated to retail for only $2,999 — the price of the similarly spec’d ASUS Ascent GX10. Though the Ascent comes with only a 1TB hard drive and is far less inspired design-wise. Beyond its delayed release and initial sticker shock, early reviews have been mixed.

Many early adopters praise Spark’s ability to run larger AI models and champion the GP10’s ability to outperform many PC GPUs. Additionally, remote access through tools like Tailscale have made Spark even more accessible. Using it as a companion to developers’ existing PCs and other devices, you can work with your Spark from anywhere on earth.

However, Spark is being criticized for not reaching its listed capabilities. Though rated for 240 watts, noted programmer and video game developer [John Carmack reported](https://www.pcgamer.com/hardware/graphics-cards/og-doom-legend-john-carmack-says-his-nvidia-dgx-spark-box-has-thermal-throttling-issues-raising-doubts-over-the-viability-of-nvidias-ai-apu-for-laptop-pcs/) less than half of that. As a result, he suggested it only delivers about half of its advertised performance. With wide reports of latency, throttling, and reboot issues, some shoppers have voiced reservations about buying it for more data-intensive projects.

For now, Spark’s full range of ideal use cases have yet to be explored and its ecosystem is still evolving. Recent [news that open source platforms](https://blogs.nvidia.com/blog/open-source-aerial-ai-native-6g/) like NVIDIA Sionna Research Kit and Aerial Testbed are available for the DGX network could provide Spark with native tools to accelerate its capabilities. It has shined brightest for agentic AI and reasoning models. For a mini AI supercomputer, it does a lot — it may just not be enough for everyone.

### **Bringing Powerful Compute Within Reach**

Spark's steep price tag makes it an expense that may be hard to justify. While on-premise, local AI development sounds great, Spark may be too niche for wide adoption at its price point. But the tiny supercomputer does nonetheless reflect a push toward making advanced AI computing more widely available.

Here at Vast.ai, we share that goal of making AI development accessible. Our mission is to democratize AI by making powerful GPUs available to anyone, anywhere — without the upfront cost of purchasing and maintaining costly hardware. With our cloud GPU rental platform, you can leverage machines that exceed DGX Spark with affordable, on-demand compute power designed to scale with your needs.

After all, democratizing AI starts with access. On Vast, you'll save 5–6X compared to traditional cloud providers. Find your compute and [get started](https://vast.ai/pricing) today\!


Explore the NVIDIA DGX Spark — the world’s smallest desktop AI supercomputer — its specs, performance, reviews, and how Vast.ai offers a more scalable alternative.

The NVIDIA DGX Spark: The New Desktop AI Supercomputer That Fits in Your Hand


What if your Integrated Development Environment (IDE) could talk back to you, suggest solutions, and write half your code for you? That's essentially what vibe coding does.

Coined earlier this year, "vibe coding" is an interactive approach where developers partner with AI to generate code. Instead of typing every line manually, they use natural language to collaborate with an AI model – guiding and prompting the model to generate and refine code through an iterative "vibing" process.

So why is vibe coding taking off? What are some of its benefits and challenges? Let's take a closer look at this very new approach to coding.

### Why Vibe Coding Is Growing in Popularity

Vibe coding has lowered the barrier to entry for software development, allowing non-technical individuals and hobbyists to build applications by simply describing what they want. Suddenly just about anyone can create their own custom tools and prototypes.

It's no wonder vibe coding is catching on. The following are some key reasons why:

#### 1\. Context-aware Assistance

In vibe coding, the AI model understands the broader context of the project. It can interpret the developer's intent and suggest code that aligns with the project's purpose, while also tracking variables, dependencies, comments, and recent edits in order to maintain a seamless workflow.

#### 2\. Productivity Gains

Research has [found](https://itrevolution.com/articles/new-research-reveals-ai-coding-assistants-boost-developer-productivity-by-26-what-it-leaders-need-to-know/) that AI coding assistants can boost the productivity of software developers by 26% in real-world business settings. AI models automate mundane and repetitive tasks like writing boilerplate code and generating tests, increasing the speed of iteration and time-to-prototype.

#### 3\. Cognitive Offloading

Instead of focusing on syntax and structure, developers can spend more time thinking about architecture and design. Delegating low-level tasks to the AI model enables them to focus on higher-level decisions like what the software should do, rather than how to make it happen.

### Benefits of Vibe Coding

Beyond speed and efficiency, here are a few benefits of vibe coding:

-   **On-demand collaboration**: The AI model is always available to collaborate as a coding partner. For those new to coding, it can effectively act as a built-in teacher, and for seasoned developers, it helps brainstorm and validate ideas.

-   **Creative workflow**: By handling the mundane, repetitive parts of coding, AI lets developers concentrate on the big picture, which encourages deeper thinking and creative problem-solving.

-   **Democratizing software development**: With vibe coding, non-technical users can build and prototype their own tools. Programming expertise is no longer required to turn ideas into working applications, opening the door to greater creativity and inclusion in tech.

Vibe coding isn't without drawbacks, however.

### Potential Pitfalls of Vibe Coding

Despite its advantages, vibe coding does introduce real challenges that must be thoughtfully dealt with, such as:

-   **Hallucination or incorrect code**: AI models can generate convincing but incorrect code, which could include security flaws. Developers should review and test everything that comes from the AI, essentially treating it as an assistant that requires careful supervision.

-   **Version drift**: Over time, inconsistencies may emerge between human-written and AI-generated sections of code, and the code may increasingly deviate from the intended architectural design. Careful integration pipelines and regular "re-anchoring" to the original goals are crucial to enforce version control.

-   **Governance and compliance issues**: With AI contributing portions of code, organizations must ensure that all of the code they use is auditable and properly licensed while also being safe to deploy. Human oversight at every step is vital.

Overall, vibe coding is a pretty exciting new phase in software development. It just needs to be used carefully, at the right time and in the right place.

### Final Thoughts

Most vibe coding happens on public platforms powered by cloud-hosted AI models. This is perfectly fine for experimentation but may not be ideal for projects involving sensitive code or proprietary data. For individuals and teams that need privacy or operate under strict compliance requirements, a closed and controlled environment may be a better fit.

With **Vast.ai**, it's easy to spin up secure, isolated GPU instances on demand – so you can host open-source models like Code Llama or DeepSeek Coder and get started vibe coding while keeping your data fully contained. Save up to **5–6X** compared to traditional cloud providers and enjoy both privacy and performance without the heavy price tag.

Our platform offers a practical way to experiment with vibe coding on your own terms – securely, affordably, and under your control. Explore Vast.ai today\!


Discover vibe coding — an AI-driven way to code faster and smarter. Collaborate with AI to generate, refine, and prototype code securely on Vast.ai.

Less Coding, More Creating: The Rise of Vibe Coding


One of the most important considerations when you're running private AI models on a cloud platform is data security. You want to protect your training data and the outputs your models generate – and even the models themselves.

At Vast.ai, we recognize that there's always some level of risk when sensitive data moves through cloud infrastructure. That's why we've built safeguards into every layer of our platform.

From containerized environments to our Secure Cloud offering and ongoing compliance efforts, here's how we help ensure you can run private AI models with confidence on Vast.

### Docker Container Isolation

When you rent GPUs on a cloud platform like Vast.ai, your workloads run inside Linux Docker instances. Each of these instances is a container created from a Docker image, where the container packages your processes and dependencies into a self-contained unit. This container runs in its own isolated environment, separate from the host system and from other clients' containers.

Why is this important? On a multi-tenant platform, multiple users may be running workloads on the same physical server. But with the isolation provided by a containerized environment, **each client only has access to their own data.**

Additionally, when you delete your instance, all data stored on it is also deleted. If you stop your instance, your data will still be stored on the disk and able to be copied in and out, but it's crucial to destroy your instances when you no longer need them. This helps protect your data while also avoiding unnecessary disk storage charges.

If you require the extra assurance of enterprise-grade data protection, you're in luck. Our Secure Cloud option may be just what you're looking for.

### Vast.ai's Secure Cloud Offering

Our **Secure Cloud offering** is designed for clients with the strictest data protection requirements – particularly well suited for when you're running your own private AI models or working with proprietary or sensitive datasets. It's our highest security tier, giving you the option to select GPU infrastructure only from our vetted datacenter partners.

The datacenter providers in our Secure Cloud all hold a **minimum of ISO 27001 certification**. Many are also HIPAA, NIST, PCI, and/or SOC 1-3 certified, as well as GDPR compliant. Each datacenter provider signs comprehensive Data Processing Agreements (DPAs) with Vast.ai and undergoes both internal and third-party audits on a regular basis.

Furthermore, we compile documentation on the identity of each provider and ensure they follow industry-recognized best practices for incident response, continuous monitoring systems, and security and compliance training for employees.

Our Secure Cloud offering provides the strongest safeguards available on Vast.ai. However, we also maintain our own ongoing efforts to reinforce data protection across our platform.

### Ongoing Compliance Efforts

At Vast.ai, we understand that compliance is an ongoing responsibility. Our integrated approach to security and trust weaves robust compliance measures throughout our platform to meet the unique needs of every client.

Some of our ongoing efforts include:

-   [**SOC 2 Type I and II and SOC 3 Certification**](https://vast.ai/article/vast-soc2-typeII-certification)s – Independent audits of our internal controls, validating that your sensitive data is secure, available, and privately stored across all of our systems (along with a standard 12-month Type II verification cycle for continuous coverage year round).

-   [**Vulnerability Bounty Program**](https://vast.ai/article/announcing-the-vast-ai-vulnerability-bounty-program) – A structured way to report bugs and responsibly disclose any security issues on our platform, as well as suggest new features.

Our [**Trust Center**](https://trust.vast.ai/) provides a detailed overview of our controls, compliance stance, FAQs and Subprocessors, and details about our other certifications and security practices.

### Securing Your Peace of Mind

Running private AI models on a cloud GPU platform requires careful attention to security. You must be able to protect not only the data you train on, but also the models you've built and the outputs they produce. Addressing these concerns is a top priority for us.

At Vast.ai, our goal is to give you the flexibility to scale workloads while maintaining the privacy and integrity of your data. As a result, you can be confident that your sensitive assets stay secure. As we continue to advance the security infrastructure we've built over 6+ years of trusted service, we remain focused on making powerful GPU compute accessible and secure for everyone.

Explore how **Vast.ai** can support your private AI workloads at scale, today\!


Learn how Vast.ai protects your sensitive data while running private AI models in the cloud. Discover our Secure Cloud offering, Docker container isolation, and ongoing compliance measures that ensure your AI workloads stay private and secure.

Running Private AI Models Without the Risk of Data Exposure


We're back with a new round of updates here at Vast.ai\! These past few months, we've expanded our offerings so you can get the most out of your time and budget – with new hardware options, fresh templates, and continued enhancements to elevate your experience.

To that end, we're pleased to offer **NVIDIA B200 GPUs** for rent\! These incredibly powerful machines deliver exceptional performance for develop-to-deploy pipelines and all of your most demanding enterprise AI tasks.

But that's not all. Here are the details on what's new\!

### NVIDIA Cloud GPU Updates

As mentioned, the NVIDIA B200 joins our growing GPU lineup, bringing you more power and options than ever.

Featuring **192 GB** of HBM3e memory with extremely high bandwidth, the B200 is ideal for high-throughput AI workloads like fine-tuning, Retrieval-Augmented Generation (RAG), and LLM inference. For additional information and pricing details, learn more [here](https://vast.ai/pricing/gpu/B200).

### Fresh New Templates

We've also rolled out several templates that expand what you can build and run on Vast. Jump-start your creative and technical projects with templates such as:

-   [**InvokeAI**](https://cloud.vast.ai/?ref_id=62897&template_id=c309df384010cf9d3dc9abb9e0a5a6fd) – A creative engine providing a professional web interface for AI image generation using Stable Diffusion models.

-   [**Langflow \+ Ollama**](https://docs.vast.ai/langflow-ollama) – A visual AI workflow platform with an intuitive drag-and-drop interface in Langflow and a hosted Ollama API server with models like Llama, Mistral, and more.

-   [**Ostris AI Toolkit**](https://cloud.vast.ai/?ref_id=62897&template_id=81e6e9dd266d9bdbcbb0fd48b9afe357) – A complete environment for training custom Stable Diffusion models, LoRAs, and other AI image generation models using your own datasets.

-   [**ACE Step**](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ACE%20Step%20Music%20Generation) – An AI-powered music generation system that lets you create original music through a simple web interface.

### Other Improvements

We've recently updated our Discord server to make conversations more focused, relevant, and helpful for you. Didn't know we had a Discord? [**Join us**](https://discord.gg/vast) there today for direct access to updates, technical guidance, and our supportive user community\!

We've also made an important update to how API keys are handled. To strengthen security, **API keys are now one-time viewable and not readable anywhere on Vast.** Be sure to store your key off of Vast when it's created, as it won't be visible again afterward. Vast.ai API endpoints will no longer return API keys as part of their response (see details in change log below).

That wraps up the updates for October 2025\!

### Our Commitment

From the new B200 GPUs to convenient templates that help you move from idea to execution, this month's updates continue our mission to make high-performance compute accessible to everyone. We're here to help you build faster and create more, with a platform that grows with your needs.

Need assistance? Reach out to our customer support team at [support@vast.ai](mailto:support@vast.ai) or join our [Discord](https://discord.gg/vast) server anytime for tips, troubleshooting, and community insights.

### Change Log

#### **InvokeAI Template now available\!**

InvokeAI is a leading creative engine for Stable Diffusion models that provides a professional web interface for AI image generation. It's perfect for artists, designers, content creators, or anyone wanting to generate stunning AI artwork. Find the InvokeAI template on Vast.ai [here](https://cloud.vast.ai/?ref_id=62897&template_id=c309df384010cf9d3dc9abb9e0a5a6fd).

#### **Langflow Template & Guide are now live\!**

This template gives you a complete visual AI workflow platform with Langflow and a hosted Ollama API server running in Docker containers. Langflow provides an intuitive drag-and-drop interface for building complex AI applications, while Ollama serves as your private LLM backend with models like Llama, Mistral, and more. Learn how to launch Langflow on Vast.ai [here](https://docs.vast.ai/langflow-ollama).

#### **Ostris AI Toolkit Template now available\!**

This template gives you a complete Stable Diffusion training environment with the Ostris AI Toolkit running in a Docker container. The toolkit provides an intuitive web interface for training custom Stable Diffusion models, LoRAs, and other AI image generation models with your own datasets. Find the Ostris template on Vast.ai [here](https://cloud.vast.ai/?ref_id=62897&template_id=81e6e9dd266d9bdbcbb0fd48b9afe357).

#### **ACE Step Template now available\!**

This template gives you a hosted ACE Step music generation server running in a Docker container. ACE Step is an AI-powered music generation system that lets you create original music through a web interface. It's perfect for musicians, content creators, developers, or anyone who wants to experiment with AI music generation. Find the ACE-Step template on Vast.ai [here](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ACE%20Step%20Music%20Generation).

# **Vast Discord Update: Unlock Your Channels\!**

We’ve streamlined our Discord to make it cleaner, faster, and more relevant to you. Here’s what’s changed.

## **Connect your Vast account to instantly unlock channels designed for your role:**

-   **Hosts** → Access to \#Hosting, \#Hosting-rigs, \#Support-hosting. (More host channels coming soon\!)
-   **Clients** → Unlock support rooms for templates, feature requests, and more.

# **API Key Security Update \+ API Breaking Changes**

**API keys are now one-time viewable and not readable anywhere on Vast.** You will need to store your API key off of Vast at the moment it is created, as it will only be visible this one time. If you need access to your existing API key, and did not store it off of Vast, you will need to delete it and create a new one. **Vast.ai API endpoints will no longer return API keys as part of their response.** However, we've added **key_id** to responses specifically for cases like this. The newly added key_id can be used to verify a user instead of api_key in certain scenarios. The key_id associated with a user's API key is returned from the show api keys endpoint / command.

## **key_id is now available for these API calls:**

-   create_user \- POST /api/v0/users
-   log in (regular and oauth and tfa) \- PUT /api/v0/users/current
-   update_user \- PUT /api/v0/users
-   get_current_user \- GET /api/v0/users/current
-   get_root_user \- GET /api/v0/users/root
-   connect_discord \- GET /api/v0/oauth/connect-discord/
-   disconnect_discord \- DELETE /api/v0/admin/disconnect-discord/
-   get_subaccounts \- GET /api/v0/subaccounts/

## **The following API key endpoints are now deprecated:**

-   GET /api/v0/auth/apikeys/&#123;id&#125;/
-   PUT /api/v0/commands/reset_apikey/

## **And the following will now have null for the key values:**

-   GET /api/v0/auth/apikeys/
-   GET /api/v0/team/apikeys/

## **New NVIDIA B200s added\!**

B200 GPUs are ideal for high-throughput AI workloads like fine-tuning, Retrieval-Augmented Generation, and LLM inference. These GPUs feature 192 GB of HBM3e memory with extremely high bandwidth.

-   [Learn more](https://vast.ai/pricing/gpu/B200)
-   [Rent now](https://cloud.vast.ai/?gpu_option=B200)


Discover what's new at Vast.ai! From the powerful NVIDIA B200 GPUs to fresh AI templates like InvokeAI, Langflow + Ollama, and Ostris, plus security and API updates — explore how we're making high-performance compute faster, safer, and more creative for everyone.

October 2025 Product Update


By now, just about everyone has either heard of large language models (LLMs) or has used them before. Not as many people are familiar with small language models (SLMs), however.

The names offer a clue – large and small – but size isn't the only thing that sets them apart.

So what are the differences between LLMs and SLMs? When do you actually need a large model, and when might a small one be the better choice?

In this post, we'll answer these questions and more. Here's what you need to know.

### Large Language Models vs. Small Language Models

An AI language model is a type of artificial intelligence that is trained on huge text datasets to **understand and generate human language.** These models use probabilistic machine learning to predict which words are most likely to appear next in a phrase sequence – and, in doing so, are able to generate text that resembles how people actually write and speak.

In practice, this ability is what underpins natural language processing (NLP) systems that let machines interact with us in useful ways.

When it comes to LLMs and SLMs specifically, though, let's look at how the two compare.

#### What Are LLMs?

With **massive parameter counts** in the billions and even trillions, most modern LLMs rely on the transformer architecture, which uses a **self-attention mechanism** to make sense of relationships between words across a sequence. This allows them to model incredibly complex patterns and long-range dependencies in language.

Since they're **trained on extensive datasets over a variety of domains**, LLMs are highly versatile. They have broad general knowledge and can generalize well across different tasks – even multitasking effectively across domains.

The downside is that training and running LLMs requires **significant computational power.** It's a resource-intensive process that often involves costly hardware and large-scale distributed computing infrastructure.

#### What Are SLMs?

At the basic level, an SLM is a smaller version of an LLM. It has **fewer parameters** – in the millions to low billions – and its architecture is focused on efficiency.

For instance, some SLMs use a **sliding window attention** mechanism (where the model focuses on a fixed-length "window" that slides across the text), and others rely on techniques like **grouped-query attention** (GQA), **sparse attention patterns**, and **low-rank adaptation** (LoRA) to improve efficiency and lower compute costs.

Unlike LLMs, SLMs are **trained on domain-specific data** or smaller datasets tailored to particular tasks. Although they may lack broad general knowledge, they do well in their specific domains and can even be fine-tuned for niche or regulated industries like finance and healthcare.

Because of their smaller size, SLMs need **less computational power** and can often be trained and deployed on more modest hardware.

### LLMs vs. SLMs: Strengts and Limitations

Both LLMs and SLMs have advantages and drawbacks. Here's a quick side-by-side comparison of their respective pros and cons:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Model Type
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Strengths
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Limitations
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">LLM</td>
            <td class="border border-gray-300 px-4 py-2">
                Broad general knowledge across different domains
                <br />
                Can handle complex, open-ended tasks with more contextual
                understanding
                <br />
                Greater multimodal potential
            </td>
            <td class="border border-gray-300 px-4 py-2">
                Requires extensive compute resources and costly hardware
                <br />
                Slower inference and higher latency
                <br />
                Not optimized for specific tasks, and higher risk of bias due to
                unfiltered training data
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">SLM</td>
            <td class="border border-gray-300 px-4 py-2">
                Easier to fine-tune for specific domains
                <br />
                Uses less compute resources
                <br />
                Faster inference and lower latency
                <br />
                Stronger data control and easier on-prem deployment
            </td>
            <td class="border border-gray-300 px-4 py-2">
                Narrower focus and less general knowledge; may require
                retraining for new tasks
                <br />
                Struggles with highly complex or long-context tasks
                <br />
                Limited multitasking capability
            </td>
        </tr>
    </tbody>
</table>

So how do you decide which type of model is right for your needs?

### Choosing the Right Model

As we've covered, neither model type is universally superior. The value of each one depends on what you're trying to accomplish and the resources you have available. That said, here are some guidelines to keep in mind.

#### An LLM may be the best choice if:

-   You need a model for **open-ended, multi-domain applications** such as general-purpose chatbots or creative content generation.  

-   Your use case involves **long-range context** across different subject areas.  

-   You have access to **powerful compute resources** and can manage the costs involved.

#### An SLM may be right for you if:

-   You prefer a model that is **lightweight and efficient**, suited for **resource-constrained environments.**  

-   Your application is **narrow or domain-specific**, such as FAQ bots or translation from one language to another.  

-   You want to **fine-tune with proprietary data** or maintain stricter control in regulated industries.

At the same time, you don't necessarily even have to choose one or the other; a hybrid approach could be the way to go. An SLM can take on routine tasks while an LLM handles more nuanced issues. For instance, you could use an SLM for standard customer queries and escalate more complex problems to an LLM.

### Final Thoughts

Both LLMs and SLMs offer unique advantages. Getting the most out of them often depends on having scalable, affordable compute – and that's where **Vast.ai** comes in.

Our cloud GPU platform gives you the flexibility to run a lightweight SLM on a single GPU or to train a massive LLM across [distributed clusters](https://vast.ai/products/clusters), all at a fraction of the typical cost. With Vast.ai, you can **save up to 5–6X** compared to traditional cloud providers – and train, fine-tune, and deploy AI models on your own terms.

Ready to get started? Spin up GPUs on demand with [Vast.ai](http://Vast.ai) today\!


Learn the key differences between Large Language Models (LLMs) and Small Language Models (SLMs), their strengths, limitations, and when to use each.

LLMs vs. SLMs: What's the Difference, and Why Does It Matter?


Are you looking for a faster way to turn your AI application ideas into working prototypes?

Langflow may be just the solution you need. Its visual editor simplifies the process, allowing you to build and test a functional AI application workflow quickly and easily. You can develop anything from chatbots and document analysis systems to content generators and agentic applications – and much more.

### What Is Langflow?

Put simply, [Langflow](https://docs.langflow.org/about-langflow) is an **open-source, node-based agent builder** that you can use right from your web browser. In minutes, you can create and serve flows: that is, functional representations of application workflows. All you have to do is connect and configure component nodes using the drag-and-drop visual editor, where each component represents one step in the workflow.

Then you can test your flows in real time using the Playground. This is a dynamic interface that lets you examine and experiment with different inputs, outputs, and memories – and get feedback about flow logic and response generation – before you move forward with more formal app development. Flows can even be embedded into your application code using the Langflow API.

Plus, Langflow is model-agnostic. It supports all major LLMs and vector databases you want to use, and it has built-in support for the Model Context Protocol (MCP), as well. It even has a **convenient Ollama integration** that makes it easy to use with open-weight models as well as custom fine-tunes.

With Vast, you can run Langflow with Ollama on powerful GPU instances at affordable rates, enabling cost-effective AI app development and prototyping. This guide will show you how to set up [Langflow \+ Ollama](https://docs.vast.ai/langflow-ollama) using our pre-built template in just minutes.

### Getting Started with Langflow \+ Ollama

Before you do anything else, set up your Vast account and add credit. If you need help with this step, review our [QuickStart guide](https://docs.vast.ai/quickstart) for a walkthrough of the setup process.

Now it's time to configure your template. Minimal changes are needed here, but you'll probably want to customize the template so that Ollama will download your preferred model automatically.

Follow our [step-by-step guide](https://docs.vast.ai/langflow-ollama) to get you started in our documentation.

### Conclusion

There is so much more to Langflow than it's possible to cover in this short introduction. The endlessly customizable framework lowers the barrier to building and testing AI applications, while Ollama keeps model choice open and flexible. Both beginners and seasoned developers can benefit from its ease of use and support for creating more complex projects.

Running Langflow \+ Ollama on **Vast.ai** gives you the cost-effective GPU infrastructure to go from prototype to production on your own terms.

Turn your ideas into working real-world solutions today\!


Run Langflow with Ollama on Vast.ai GPUs for fast AI prototyping. Step-by-step guide to build chatbots, agents, and custom AI apps with a visual editor.

Build and Test AI Applications with Langflow + Ollama on Vast.ai


[Video generation with AI](https://docs.vast.ai/video-generation) has come a long way. If you saw the infamous AI-generated video of Will Smith eating spaghetti a couple of years ago, you probably remember how unnatural and distorted the clip looked\! But that was just the beginning.

Things look very different today, with more advanced video generation tools producing much higher quality results. Today's models can create photorealistic visuals with smooth, consistent motion as well as stylized and cinematic effects if that's what's called for.

AI video generation has become a practical tool for anyone who needs an easy way to go from text to video, no matter the project. The right templates make that process even simpler.

With that in mind, let's take a look at a few of the best [AI video generation](https://docs.vast.ai/video-generation) templates available today.

### 1\. [ComfyUI \+ LTX Video](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%20%2B%20LTX%20Video)

[**LTX Video**](https://github.com/Lightricks/ComfyUI-LTXVideo) is a group of open-source AI video models built for high-quality diffusion-based video generation. It offers both speed and precision, generating high-resolution videos **up to 60 seconds long** (with some versions of the model limited to **10 seconds**), often in less time than it takes to watch the resulting video.

How is this possible? LTX Video uses a unique approach where it **compresses video data 192 times smaller than the original** – past the limits of what most models can achieve – while still maintaining crisp visuals and smooth, realistic motion. A bonus is that this drastically reduces the amount of memory and compute required. In fact, LTX Video is optimized specifically for consumer-grade GPUs.

While most diffusion models use a two-step pipeline where a second-stage model sharpens details, LTX Video does decoding and cleanup in just one step. With a Video-VAE (variational autoencoder) and a denoising transformer combined into a single integrated system, there's no need to patchify the video stream, and high-quality results are possible even under heavy compression.

This [template](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%20%2B%20LTX%20Video) pairs LTX Video with **ComfyUI**, a modular, node-based interface for Stable Diffusion workflows. Together, they serve as a seamless way to run efficient, high-compression video pipelines with a user-friendly interface.

### 2\. [Mochi](https://vast.ai/article/generating-videos-with-mochi-on-vast.ai)

Designed to excel in quality of motion as well as prompt adherence, [**Mochi**](https://github.com/genmoai/mochi) is a state-of-the-art open-source video generation model that performs competitively with leading closed models.

It demonstrates strong motion quality, generating smooth videos at **30 frames per second for durations of up to 5.4 seconds.** Fluid dynamics, fur and hair simulation, and human movement are all rendered with impressive realism.

Mochi's alignment with text prompts is also exceptional. Generated videos accurately depict the given input, allowing users to control characters, settings, and actions with a high degree of detail. Prompt adherence is benchmarked with a vision-language model and generated videos are evaluated using Gemini to ensure accurate, consistent results.

However, keep in mind that the initial release of Mochi only generates videos at **480p**, and the model is optimized for photorealistic styles, so animated content is not recommended.

### 3\. [Open-Sora](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Open-Sora)

[**Open-Sora**](https://github.com/hpcaitech/Open-Sora) is an open-source video generation model with a streamlined, user-friendly platform that lowers the barrier to efficient, high-quality video content creation. It can produce videos **up to 15 seconds long, at resolutions of up to 720p, with any aspect ratio.**

The model also supports a wide range of visual generation tasks: text-to-image, text-to-video, and image-to-video generation. This flexibility means users can run diverse workflows within a single framework.

Notably, Open-Sora introduces two innovations. The first is the Spatial-Temporal Diffusion Transformer (STDiT), a framework that decouples spatial and temporal attention. The second is a highly compressive 3D autoencoder that not only makes representations more compact but also accelerates training with its own ad hoc training strategy.

These features allow Open-Sora to generate high-quality results more efficiently – and with a ready-to-use template, it's easier than ever to start experimenting.

### Final Thoughts

AI video generation has matured rapidly in just a few short years. Instead of glitchy experiments, we now have reliable tools that deliver polished video content.

At **Vast.ai**, we believe in democratizing [access to these advanced AI tools](https://docs.vast.ai/video-generation), making them easy to use and available to everyone without the overhead of complex setup.

With our pre-built templates for [LTX Video](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%20%2B%20LTX%20Video), [Mochi](https://vast.ai/article/generating-videos-with-mochi-on-vast.ai), and [Open-Sora](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Open-Sora), you can launch cutting-edge video generation models in minutes – backed by affordable, on-demand cloud GPUs that save you from having to purchase and maintain your own expensive hardware. Use only the compute you need, when you need it.

Ready to experiment? Explore the [video generation templates](https://docs.vast.ai/video-generation) on Vast.ai and bring your ideas to life today\!


Generate AI videos with LTX Video, Mochi, and Open-Sora on Vast.ai. Step-by-step templates for high-quality, photorealistic text-to-video workflows with affordable GPU hosting.

Vast.ai: Best Video Generation Templates 2025


NVIDIA is [reportedly](https://www.reuters.com/world/china/nvidia-working-new-ai-chip-china-that-outperforms-h20-sources-say-2025-08-19/) working on a new China-specific AI chip that will be more powerful than the aging HGX H20 model the company is currently authorized to sell in China.

Recent reports [suggest](https://www.tomshardware.com/tech-industry/artificial-intelligence/nvidia-responds-to-reports-that-its-h20-gpu-for-china-is-ending-production-next-gen-b30a-green-light-up-to-the-united-states-government-according-to-ceo-jensen-huang) that the H20 – originally developed as a scaled-down alternative to the A100 and H100 that were banned from export to China – is nearing the end of its production run. Its successor: the tentatively named B30A chip.

Based on the latest Blackwell architecture, the B30A will feature a single-die design delivering roughly half the raw computing power of NVIDIA's flagship B300 accelerator, according to sources. The new chip will include high-bandwidth memory and NVLink support for fast data transfer between processors – both of which are also present in the Hopper-based H20.

NVIDIA aims to send test batches of the B30A chip to Chinese customers as early as sometime this month, sources said.

The move comes following remarks from President Donald Trump suggesting that more advanced NVIDIA chips could be cleared for sale in China. However, regulatory approval remains uncertain. There are ongoing concerns about granting China broader access to U.S. AI technology – which could weaken the United States' competitive edge in the sector.

At the same time, analysts warn that restricting China too heavily risks driving developers toward rivals like Huawei. NVIDIA argues that it's important to retain market share in China – which generated 13% of the company's revenue in the past year – so that its software ecosystem stays relevant in one of the world's largest AI markets.

NVIDIA is also preparing a second Blackwell-based China-specific chip called the RTX6000D, designed primarily for AI inference tasks, which is expected to start shipping this month. With weaker specs than the H20 and bandwidth just under U.S. export limits, it's set to be a compliant, lower-cost alternative within NVIDIA's lineup.

For those outside China, however, these region-specific models highlight the stark difference between scaled-down chips and the unrestricted performance available elsewhere.

### On-Demand Access to Powerful GPUs, Anytime

While NVIDIA adapts its offerings to meet specific market regulations, users elsewhere don't have to work around restricted versions.

On [**Vast.ai**](http://Vast.ai), you can access a wide range of powerful GPUs – from the [RTX 5090](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know) to the [A100](https://vast.ai/article/nvidia-rtx-4090-vs-a100-two-powerhouses-two-purposes), [H200](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights), [B200](https://vast.ai/article/nvidia-h200-vs-b200-comparing-datacenter-grade-accelerators), and more – on demand, at a fraction of the cost of traditional cloud providers. Our cloud GPU rental platform makes it easy. Just choose the hardware you need, spin up an instance in seconds, and pay only for the compute you use.

Get started with Vast today, and enjoy flexible, cost-effective GPU compute that scales with your needs\!


NVIDIA is developing the B30A AI chip for China, a Blackwell-based successor to the H20 with half the power of the B300. Learn what it means for AI exports, competition, and how you can still access unrestricted GPUs like the H200, B200, and A100 on Vast.ai.

Report: NVIDIA Developing New Blackwell-Based AI Chip for China


Remember when ChatGPT couldn't tell you the weather or gather top headlines? Are you still frustrated asking Gemini to create a JIRA ticket from a bug report email? Perhaps you're tired of Cursor hallucinating database column names because it's walled off from your live schema, so you're constantly copy-pasting the same context across your Claude and Cursor apps. Despite their eloquent email-writing powers, out-of-the-box Large Language Models (LLMs) are powerful predictors, not contextually-aware assistants. They operate in silos, forcing you to start with fresh context \-- or none at all \-- for every new task.

This constant context-juggling isn't just annoying, it's fragile. When a better model is released or your data governance rules change, do you have to re-implement your entire workflow? What if you want to easily swap out Gemini's new video generation model for one from Stable Diffusion?

With [Model Context Protocol](https://modelcontextprotocol.io/) (MCP), an open-source framework released by Anthropic for building truly helpful, environment-agnostic AI assistants, you can create a persistent, shareable context layer across models and tools. And with [Vast.ai](http://vast.ai)'s secure and affordable cloud GPU instances, implementing your own MCP has never been easier.

**Model Context Protocol Architecture**

MCP architecture is built on three key components:

-   Clients: The user-facing applications (like Cursor, a custom chatbot, or Claude) that coordinate between the user, the AI model, and the MCP server.
-   Servers: The adapter between your tools and AI models, serving tools, resources, and prompts to your LLM provider and other services.
-   Service Providers: The software tools and data sources (like Slack, Notion, GitHub, or your company's internal database) that have existing APIs.

With MCP, any developer can build a server to connect an API to any MCP-compatible client. You no longer have to wait for official integrations from LLM providers.

An MCP server provides three categories of capabilities to the model – tools, resources, and prompts. In the following guide, you'll focus on creating MCP tools and learn to create useful actions to spawn from your clients via a remote MCP server on [Vast.ai](http://Vast.ai)'s platform.

**Remote MCP Servers**

Initially, MCP architecture only supported locally running servers using stdio transport. Since summer 2025, it supports remote access via streamable-http, enabling scalable, AI-enabled workflows that integrate LLMs across your organization. Now teams can more easily integrate data services and build custom workflows within their workspaces.

​

[Vast.ai](http://vast.ai)'s affordable GPU instance rentals provide flexible and affordable hosting for as many MCP servers as you want to create – scaling up your team's AI workflows quickly and securely.

**Tutorial: Build a Stock Sentiment Analysis MCP**

Today, you'll build an MCP server for stock sentiment analysis correlation hosted on [Vast.ai](http://vast.ai)'s platform. This setup provides a foundation for team-based workflows. For example, a financial analyst company may use this MCP tool to probe further into market conditions all within the Claude interface, or spur discussions by connecting this output to a shared Slack channel, providing up to date and collective information to multiple teams. To begin, let's review what you'll need for this process.

[Vast.ai](http://vast.ai)'s GPU rentals offer space for a vast variety of MCP servers, easily accommodating 70B large language models or image diffusion models on the most up-to-date graphics card offerings. Their platform creates opportunities for creative, custom server integrations like creating a shared, product-specific image prompt for your product teams or pulling in proprietary climate energy data to your coding and marketing workflows.

By the end of this tutorial, you will understand how to build a similar bespoke MCP server for your organization's needs, and have a running blueprint for creating your collection of remote MCP servers on [Vast.ai](http://vast.ai). Let's get started.

**Prerequisites**

-   A [Vast.ai](http://vast.ai) account with an API key
-   A Hugging Face account with Mistral7B access token
-   Python 3.10+ environment

1. **Rent and Configure Your [Vast.ai](http://vast.ai) GPU Instance**  
   Start by selecting a GPU Instance from the Vast.ai console. For this project, you'll use a **Mistral 7B LLM** to summarize and analyze market sentiments, so we recommend using an **RTX 4090 GPU with 24GB of VRAM**. [Vast.ai](http://vast.ai) offers a variety of powerful graphics cards, including A100 and 80GB VRAM for larger LLM and diffusion tasks.  
   Begin by selecting a PyTorch (Vast) template. Modify this template to launch as an interactive shell server without a Jupyter notebook instance and with 24 GB of disk space. For more information on selecting templates, please refer to the [template guide.](https://docs.vast.ai/guides/instances/choosing/templates)<br />
   ![][image3]Now select an RTX 4090 offering at your required price point and select "Rent". Navigate to the "Instances" section in the sidebar. When the server is up and running, you should see "Running" next to "Status". Now you're ready to create your MCP server.  
   ![][image4]
2. **Create Your Environment**  
   In your terminal, install uv and set up your Python environment using the following command:

​​

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Now, restart your terminal to activate the \`uv\` command. Then, initialize a Python project.

```bash
uv init vast_mcp_server
cd vast_mcp_server
```

​​

Finally, create and activate your virtual environment to manage project dependencies.​

```bash
uv venv
source .venv/bin/activate
```

3. **Install Project Dependencies**  
   Create a requirements.txt file to outline project dependencies. You'll use this to write your server code locally and then install dependencies on your [Vast.ai](http://vast.ai) instance later.​

```bash
touch requirements.txt
```

Then add the following dependencies:​

```
fastmcp
pydantic
transformers
torch
requests
uvicorn
accelerate
```

Install the dependencies using the following command.

```bash
pip install -r requirements.txt
```

4. **Create Your MCP Server**  
   Now you'll create your first MCP server. This server will facilitate summarizing and analyzing news headlines using an LLM and correlating them with real-time stock data. This is a real-time task that out-of-the-box LLMs like Claude may struggle with. You'll create three MCP tools for stock analysis retrieval, sentiment analysis, and correlation – all available via your remote server.  
   This example uses [FastMCP](https://gofastmcp.com/getting-started/welcome) to facilitate simple protocol implementation. FastMCP uses simple decorators to handle the low-level implementation details, similar to the FastAPI approach.

Create a 'server.py' and 'test_server.py' file using the command:​

```bash
touch server.py test_server.py
```

First, you'll configure a Mistral 7B model for summarization and sentiment analysis for a given news article. You'll also set up a FastMCP server. Add the following code to the 'server.py' file.​

```python
import random
import logging
import asyncio
import os
import json
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelForCausalLM
from fastmcp import FastMCP
import torch


logging.basicConfig(level=logging.INFO)

# Initialize the MCP server
mcp = FastMCP("Financial Market Analyzer")

# Configure Mistral 7B LLM model
MODEL_NAME = "mistralai/Mistral-7B-Instruct-v0.2"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {DEVICE}")

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    device_map="auto"
)

class NewsItem(BaseModel):
    title: str
    description: str
```

Now, you'll create three tools. The first uses the Mistral 7B LLM to summarize and analyze a provided news article. In a real-world implementation, you may want to update this example to accept a URL as input. Add the following code:

```python
# Analyzes the sentiment and provides a summary of a news headline using a large language model.
@mcp.tool()
def analyze_sentiment(news_item: NewsItem):
    prompt = (
        f"You are a financial news analysis expert. "
        f"Analyze the following news headline and description. "
        f"Provide a brief summary and determine the sentiment as POSITIVE, NEGATIVE, or NEUTRAL. "
        f"Respond only with a JSON object containing two keys: 'summary' and 'sentiment'.\n\n"
        f"Headline: \"{news_item.title}\"\n"
        f"Description: \"{news_item.description}\"\n\n"
        f"JSON Response:"
    )

    inputs = tokenizer(prompt, return_tensors="pt").to(DEVICE)

    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)

    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

    try:
        json_start_index = generated_text.find("{")
        if json_start_index != -1:
            json_str = generated_text[json_start_index:]
            analysis_result = json.loads(json_str)
            summary = analysis_result.get("summary", "No summary found.")
            sentiment_label = analysis_result.get("sentiment", "NEUTRAL").upper()
        else:
            summary = "Could not generate a structured summary."
            sentiment_label = "NEUTRAL"
            logging.warning("LLM response did not contain a valid JSON object.")

    except json.JSONDecodeError as e:
        summary = "Could not parse LLM response."
        sentiment_label = "NEUTRAL"
        logging.error(f"Failed to parse JSON from LLM: {e}")

    logging.info(f"Analyzed '{news_item.title}': Sentiment={sentiment_label}, Summary='{summary}'")
    return {"title": news_item.title, "sentiment": sentiment_label, "summary": summary}
```

Now you'll add a stock retrieval tool. In this example, you'll mock the stock data. However, in an actual implementation, you might pull this data from a stock-related API. Add the following code:

```python
# Fetches mock real-time stock data. In a real-world implementation, you would connect to a secure financial database API.
# For this example, we are using a simple mock to simulate real-time data.
@mcp.tool()
def get_stock_data(symbol: str):
    price = round(random.uniform(100.0, 500.0), 2)
    volume = random.randint(100000, 5000000)
    logging.info(f"Fetched mock data for {symbol}: Price={price}, Volume={volume}")
    return {"symbol": symbol, "price": price, "volume": volume}
```

Finally, you'll add a tool that correlates the stock prices and news sentiment. Add the following code:

```python
@mcp.tool()
async def correlate_data(stock_data: dict, news_analysis: dict):
    sentiment = news_analysis['sentiment']
    price = stock_data['price']

    correlation_message = "No clear correlation found."
    if sentiment == "POSITIVE" and stock_data['volume'] > 2000000:
        correlation_message = f"POSITIVE news coincided with a high trading volume. Possible impact. News Summary: {news_analysis['summary']}"
    elif sentiment == "NEGATIVE" and price < 300:
        correlation_message = f"NEGATIVE news for a low-priced stock. Potential further decline. News Summary: {news_analysis['summary']}"

    logging.info(f"Correlated data: {correlation_message}")
    return {"stock_data": stock_data, "news_analysis": news_analysis, "correlation": correlation_message}
```

Now, you'll create the ability to run your MCP remote server via streamable-http. Add the following code:

```python
# Runs the MCP server remotely using streamable-http transport
if __name__ == "__main__":
    logging.info(f"MCP server started on port {os.getenv('PORT', 8080)}")
    asyncio.run(
        mcp.run_async(
            transport="streamable-http",
            host="0.0.0.0",
            port=os.getenv("PORT", 8080),
        )
    )
```

Great work! Now you have three MCP tools defined on your server. After deploying via [Vast.ai](http://vast.ai), you can access any of these tools remotely by integrating with Claude, Cursor, or other MCP supporting apps. Let's upload it to your [Vast.ai](http://vast.ai) GPU instance.

5. **Upload Your MCP Server to [Vast.ai](http://Vast.ai)**  
   First, retrieve your API key from the Vast.ai Console. Then, install the Vast.ai CLI in your Linux terminal using the following command.

```bash
pip install --upgrade vastai
```

Set your API key in your local Vast CLI configuration.

```bash
vastai set api-key #your-api-key-here
```

Now you'll be able to connect by SSH to your remote GPU instance. Navigate to your [Vast.ai](http://vast.ai) Console and select your GPU instance. Click on "Open Terminal Access". Here, you should see your SSH commands to connect to your GPU instance with the following structure:

```bash
ssh -p XXXXX root@XXX.XXX.XXX.XXX -L XXXX:localhost:XXXX
```

Navigate back to your terminal and use the command to connect to your instance. If your instance is running, you should see your terminal login to the remote [Vast.ai](http://vast.ai) instance with a welcome message.

Since you are using a Hugging Face model, you'll need to login to the Hugging Face CLI using the following:

```bash
huggingface-cli login
```

This command with prompt you for your access token from the Hugging Face website.

Then install the dependencies using:

```bash
pip install -r requirements.txt
```

Finally, you'll securely copy your [server.py](http://server.py) and requirement.txt files to the remote GPU. Navigate to the terminal you used to create your server environment and ensure the virtual environment is activated. Use the following command to move your files.

```bash
scp -P 29946 server.py requirements.txt root@172.81.127.37:/root/
```

Now you're ready to run and test your server!

6. **Run and Test Your Server**  
   In your open terminal connected to your [Vast.ai](http://vast.ai) GPU instance, simply run

```bash
uv run server.py
```

If all is well, you should see the models downloading and the server startup success message:

```
Starting MCP server 'Financial Market Analyzer' with transport 'streamable-http' on http://0.0.0.0:8080/mcp
INFO: Started server process [2135]
INFO: Waiting for application startup.
INFO:mcp.server.streamable_http_manager:StreamableHTTP session manager started
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080
```

As a final step, test the server connection to ensure your tools are available. In your Python environment, create a file called 'test_server.py' and paste the script below.

```python
import asyncio
import json

from fastmcp import Client

# Test the MCP server using streamable-http transport.
async def test_server():
    async with Client("http://localhost:8080/mcp") as client:

        # List available tools
        tools = await client.list_tools()
        for tool in tools:
            print(f"Tool found: {tool.name}")

        # Call the news analysis tool
        print("Calling stock tool for INVC")
        stock_result = await client.call_tool("get_stock_data", {"symbol": "INVC"})
        print(f"Result: {stock_result[0].text}")

        # Call the sentiment analysis and summarization tool
        print("2. Calling 'analyze_sentiment' tool...")
        news_item_payload = {
            "news_item": {
                "title": "Tech Giant Announces Record Profits",
                "description": "Shares surged after the company reported quarterly earnings that exceeded all analyst expectations, signaling strong growth."
            }
        }
        sentiment_result = await client.call_tool("analyze_sentiment", news_item_payload)
        print(f"Result: {sentiment_result[0].text}")

        # Call the correlation tool
        print("3. Calling 'correlate_data' tool...")
        correlation_payload = {
            "stock_data": json.loads(stock_result[0].text),
            "news_analysis": json.loads(sentiment_result[0].text)
        }
        correlation_result = await client.call_tool("correlate_data", correlation_payload)
        print(f"Result: {correlation_result[0].text}")

if __name__ == "__main__":
    asyncio.run(test_server())
```

```bash
uv run test_server.py
```

Navigate back to your Python environment terminal and run the script.

​If your server is working correctly, you should see output showing each of the tools and calling each tool in turn.

Congratulations! You have created your first MCP server on [Vast.ai](http://vast.ai). Now you can integrate this into Claude, Cursor, Windsurf, or any other MCP-supporting client, so you can customize, scale, and create more MCP servers for your organization and AI-enabled workflows.


Learn how to build and host a custom Model Context Protocol (MCP) server on Vast.ai's affordable GPU cloud. Step-by-step guide for creating AI-enabled workflows, integrating stock sentiment analysis, and scaling remote MCP servers with Claude, Cursor, and more.

Building Your First MCP Server on Vast.ai

Deploy OpenAI's GPT-OSS-20B on Vast.ai with Harmony SDK for transparent multi-channel AI reasoning. Step-by-step tutorial to build a weather assistant with function calls and reasoning visibility.

Build a Weather Assistant with OpenAI GPT-OSS and Harmony SDK on Vast.ai


## Introduction

Qwen-Image revolutionizes image generation with its exceptional ability to render complex text directly within images, supporting multiple languages including English and Chinese with perfect typographic detail. This powerful foundation model from Alibaba's Qwen series goes beyond simple image creation, offering advanced editing capabilities, style transfer, and precise control over visual elements.

What sets Qwen-Image apart is its nuanced understanding of text-image relationships. While other models struggle with text rendering, Qwen-Image seamlessly integrates readable text into visuals, making it ideal for creating signage, posters, infographics, and artistic compositions that require precise text placement and formatting.

Vast.ai provides the perfect platform for deploying Qwen-Image, offering access to high-performance GPUs at competitive prices with the flexibility to scale based on your needs.

In this guide, you'll deploy Qwen-Image on Vast.ai and explore its capabilities for text-integrated image generation.

## Setting Up the Environment

First, install the Vast CLI and configure your API key:

```bash
pip install --upgrade vastai
export VAST_API_KEY="<your-api-key>"
vastai set api-key $VAST_API_KEY
```

You can obtain your API key from the [Vast.ai Account Page](https://cloud.vast.ai/account/).

## Choosing the Right Hardware

Qwen-Image requires substantial GPU resources for optimal performance:

-   **GPU Memory**: Minimum 80GB VRAM for the full model
-   **Recommended Hardware**: NVIDIA A100 80GB or H100
-   **Storage**: 200GB disk space for model weights and dependencies

Search for suitable instances with these specifications:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 80 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 150 \
rentable = true"
```

This search filters for:

-   Modern GPUs with compute capability 7.5+
-   At least 80GB VRAM for the model
-   Static IP for stable connections
-   Verified providers for reliability
-   Adequate storage for model and outputs

## Deploying Qwen-Image

Deploy the instance using a pre-configured Docker image optimized for transformer models:

```bash
export INSTANCE_ID=<instance-id>
vastai create instance $INSTANCE_ID \
    --image huggingface/transformers-pytorch-gpu \
    --env '-p 8888:8888' \
    --disk 200 \
    --args bash -c "pip install jupyter && jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root --NotebookApp.token='' --NotebookApp.password=''"
```

**Key parameters explained:**

-   `--image`: HuggingFace's PyTorch GPU image with ML libraries pre-installed
-   `--env '-p 8888:8888'`: Maps Jupyter's default port for external access
-   `--disk 200`: Allocates 200GB for model weights and generated images
-   `--args`: Auto-installs Jupyter and launches a passwordless notebook server

## Connecting to Your Instance

After deployment:

1. Navigate to the [Instances tab](https://cloud.vast.ai/instances/)
2. Find your instance and wait for "Running" status
3. Look for the "Open Ports" section in instance details
4. Click the port mapping link (e.g., `123.123.123.123:12345 -> 8888/tcp`)
5. This opens Jupyter directly in your browser
6. Create a new notebook or upload the example notebook if available

## Installing Dependencies

Install the required packages in your notebook:

```bash
apt install -y git
pip install git+https://github.com/huggingface/diffusers
```

The latest diffusers library from GitHub includes Qwen-Image support.

## Loading the Model

Initialize Qwen-Image with optimal settings for GPU deployment:

```python
from diffusers import DiffusionPipeline
import torch

model_name = "Qwen/Qwen-Image"

# Load the pipeline
if torch.cuda.is_available():
    torch_dtype = torch.bfloat16
    device = "cuda"
else:
    torch_dtype = torch.float32
    device = "cpu"

pipe = DiffusionPipeline.from_pretrained(model_name, torch_dtype=torch_dtype)
pipe = pipe.to(device)
```

The model uses bfloat16 precision on GPU for optimal memory usage and performance.

## Using Qwen-Image

Now let's explore Qwen-Image's text rendering capabilities with examples:

### Example 1: Complex Text Integration with Multiple Languages

```python
import os

positive_magic = {
    "en": "Ultra HD, 4K, cinematic composition.", # for english prompt
}

# Create img folder if it doesn't exist
os.makedirs("img", exist_ok=True)

# Generate image
prompt = '''A coffee shop entrance features a chalkboard sign reading "Qwen Coffee 😊 $2 per cup," with a neon light beside it displaying "Welcome". Next to it hangs a poster showing a beautiful Chinese woman, and beneath the poster is written "π≈3.1415926-53589793-23846264-33832795-02384197". Ultra HD, 4K, cinematic composition'''
negative_prompt = " " # using an empty string if you do not have specific concept to remove

# Set image dimensions (16:9 aspect ratio)
width, height = 1664, 928

image = pipe(
    prompt=prompt + positive_magic["en"],
    negative_prompt=negative_prompt,
    width=width,
    height=height,
    num_inference_steps=50,
    true_cfg_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(42)
).images[0]

# Save image to img folder
image.save("img/example_1.png")

# Display the image
display(image)
```

### Output

![Qwen Coffee Shop with Text](/uploads/example_1.webp)

This example demonstrates Qwen-Image's ability to:

-   Render multiple text elements (chalkboard, neon sign, poster)
-   Handle emojis and special characters
-   Integrate mathematical constants with precise formatting
-   Maintain readability across different text styles

### Example 2: Creative Fantasy Scene

```python
positive_magic = {
    "en": "Ultra HD, 4K, cinematic composition.", # for english prompt
}

# Create img folder if it doesn't exist
os.makedirs("img", exist_ok=True)

# Generate image
prompt = '''Rare photo: UFO hovering over a yeti riding the Loch Ness monster.'''
negative_prompt = " " # using an empty string if you do not have specific concept to remove

# Set image dimensions (16:9 aspect ratio)
width, height = 1664, 928

image = pipe(
    prompt=prompt + positive_magic["en"],
    negative_prompt=negative_prompt,
    width=width,
    height=height,
    num_inference_steps=50,
    true_cfg_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(42)
).images[0]

# Save image to img folder
image.save("img/example_2.png")

# Display the image
display(image)
```

### Output

![Fantasy Scene with UFO and Mythical Creatures](/uploads/example_2.webp)

This showcases Qwen-Image's creative capabilities in composing imaginative scenarios.

### Example 3: Futuristic Space Scene

```python
positive_magic = {
    "en": "Ultra HD, 4K, cinematic composition.", # for english prompt
}

# Create img folder if it doesn't exist
os.makedirs("img", exist_ok=True)

# Generate image
prompt = '''Now I want you to show a spaceships approaching Andromeda Galaxy!, it should be futuristic spaceships out on exploring universe'''
negative_prompt = " " # using an empty string if you do not have specific concept to remove

# Set image dimensions (16:9 aspect ratio)
width, height = 1664, 928

image = pipe(
    prompt=prompt + positive_magic["en"],
    negative_prompt=negative_prompt,
    width=width,
    height=height,
    num_inference_steps=50,
    true_cfg_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(42)
).images[0]

# Save image to img folder
image.save("img/example_3.png")

# Display the image
display(image)
```

### Output

![Futuristic Spaceships Approaching Andromeda](/uploads/example_3.webp)

This demonstrates Qwen-Image's ability to create detailed sci-fi scenes.

## Advanced Features and Capabilities

### Flexible Aspect Ratios

Qwen-Image supports various aspect ratios for different use cases:

```python
aspect_ratios = {
    "1:1": (1328, 1328),    # Social media posts
    "16:9": (1664, 928),    # Widescreen displays
    "9:16": (928, 1664),    # Mobile/portrait content
    "4:3": (1472, 1140),    # Traditional displays
    "3:4": (1140, 1472),    # Portrait photos
    "3:2": (1584, 1056),    # Standard photos
    "2:3": (1056, 1584),    # Portrait standard
}
```

### Generation Parameters

Fine-tune your outputs with these key parameters:

-   `num_inference_steps`: Higher values (50-100) for better quality
-   `true_cfg_scale`: Controls prompt adherence (3.0-7.0 range)
-   `generator`: Set seed for reproducible results
-   `negative_prompt`: Specify elements to avoid

## Practical Applications

Qwen-Image's unique text rendering capabilities enable numerous applications:

-   **Marketing Materials**: Create posters, banners, and advertisements with perfectly rendered text
-   **Educational Content**: Generate infographics and diagrams with clear labels and annotations
-   **Social Media**: Design posts with integrated text overlays and captions
-   **Multilingual Content**: Produce visuals with text in multiple languages
-   **Creative Projects**: Combine artistic imagery with typographic elements
-   **UI/UX Mockups**: Generate interface designs with readable text elements

## Performance Optimization

To maximize performance on Vast.ai:

1. **Use bfloat16 precision**: Reduces memory usage while maintaining quality
2. **Batch generation**: Process multiple prompts together when possible
3. **Cache the model**: Keep it loaded between generations to avoid reload overhead
4. **Optimize resolution**: Start with lower resolutions for testing, then increase for final outputs

## Conclusion

Qwen-Image represents a significant advancement in image generation technology, particularly in its ability to seamlessly integrate readable text into visual content. Its support for multiple languages, mathematical notation, and various text styles makes it uniquely suited for projects requiring precise text-image integration.

By deploying on Vast.ai, you gain access to powerful GPUs capable of running this advanced model efficiently. Whether you're creating marketing materials, educational content, or artistic compositions, Qwen-Image provides the tools to generate professional-quality visuals with integrated text.

### Next Steps

Try these to explore further:

-   **Experiment with multilingual prompts**: Test the model's capabilities with Chinese, Japanese, or mixed-language text
-   **Create branded content**: Generate marketing materials with your company's text and logos
-   **Build an API service**: Wrap the model in a REST API for integration with your applications
-   **Explore style transfer**: Use the model's editing capabilities to transform existing images

[Get started on Vast.ai](https://cloud.vast.ai/) and experience Qwen-Image today!


Deploy Qwen-Image on Vast.ai to create stunning visuals with perfectly rendered text, multilingual support, and advanced editing features.

Deploying Qwen-Image for Advanced Text-Integrated Image Generation on Vast.ai


If you're working at the cutting edge of AI and high-performance computing (HPC), NVIDIA's H200 and B200 GPUs are likely already very much on your radar. They're each designed to handle the massive compute demands of datacenter-class workloads – such as training trillion-parameter models and running high-speed inference at scale.

While both GPUs deliver incredible power, it's crucial to understand the differences between the two in order to make the right choice for your needs.

Today, we'll explore how the **H200** and **B200** measure up against each other and how each fits into real-world deployments. Which one is right for you? Let's take a look.

### NVIDIA H200: Hopper-Based Heavyweight

Building on the foundation laid by the H100, the NVIDIA H200 is based on the same Hopper architecture while bringing substantial improvements in memory capacity and bandwidth. This means fewer bottlenecks, an advantage that becomes increasingly important as you move further into enterprise scale.

The H200 nearly doubles the memory of its predecessor, boasting **141 GB of HBM3e memory** and **4.8 TB/s of bandwidth** – delivering about **1.4X faster** data access. MLPerf benchmarks using Llama 2 70B demonstrated the H200's power: it reached over 31,000 tokens per second, about **45% faster** than the H100.

With the same **fourth-gen NVLink**, **PCIe Gen5**, **MIG support**, and full Tensor Core and Transformer Engine capabilities, the H200 is a solid next step for existing Hopper-based deployments. You don't have to rethink your entire infrastructure; you just get more horsepower.

To sum up, here are some of the most notable features of the H200:

-   **141 GB of HBM3e memory at 4.8 TB/s** – almost double the memory of the H100 with 1.4X the bandwidth.

-   **Up to 1.9X improved LLM inference performance**, with more tokens per second and higher throughput for gen AI workloads compared to its predecessor.

-   **Lower energy use and total cost of ownership** (TCO) over the H100 thanks to improved energy efficiency and thermal management.

-   **An upgrade path exists on qualified HGX/MGX platforms**, so that little to no reconfiguration is required.

In short, the H200 represents the very best of the Hopper-based lineup. When it was introduced in late 2024, it set a new standard for memory-intensive and high-throughput workloads in datacenter GPUs – and it's still one of the most powerful GPUs on the market today.

The B200, on the other hand, is an entire generational leap forward.

### NVIDIA B200: The New Generation

Built on NVIDIA's advanced Blackwell architecture, the B200 features **192 GB of ultra-fast HBM3e memory** with **8.0 TB/s of memory bandwidth.** Those are some staggering numbers, even compared to the H200's respectable 141 GB at 4.8 TB/s.

The Blackwell architecture also introduces **fifth-gen Tensor Cores** and **dual Transformer Engines**, which are particularly useful for AI workloads with long context windows and heavy token parallelism. The B200 scales much more efficiently than the H200 for the latest and most advanced model classes.

In fact, scaling is a major area where the B200 stands out. With **fifth-gen** **NVLink**, its inter-GPU bandwidth reaches **1.8 TB/s** within a node – compared to Hopper's 900 GB/s – with cross-node scaling supported via the NVLink Switch System.

Of course, this may not matter to you very much if you're only using a single card. At cluster scale, however, or in multi-GPU systems like the DGX B200 (which connects eight B200 GPUs together), the improvements in distributed training efficiency really make a difference.

The B200 truly sits at the frontier of performance. Its raw capacity and speed can support next-gen inference pipelines and AI workloads at hyperscale. According to NVIDIA, it delivers **3X the training performance** and **15X the inference performance** of the previous generation.

These gains do come at a cost in power draw. The B200 has a max thermal design power (TDP) of **1000 W** compared to the H200 at 700 W – which means it also requires far more robust cooling solutions. But the extra power consumption may just be worth it for the boost in performance.

To recap, a few advantages of the B200 include:

-   **192 GB of HBM3e memory at 8.0 TB/s** – ultra-high capacity that's well suited for trillion-parameter models and allows training of AI models _entirely in GPU memory_.

-   **Fifth-gen Tensor Cores and second-gen Transformer Engines** for higher efficiency in FP8, FP16, and mixed precision performance, especially in long-context and multi-modal workloads.

-   **NVLink 5 interconnect** with up to **1.8 TB/s** GPU-to-GPU bandwidth for multi-node scaling.

-   **Fine-grain micro-tensor scaling** that optimizes performance and accuracy, enabling 4-bit floating point (**FP4**) AI.

The B200 was built to shatter performance records. As you can imagine, it has an eyewatering price tag to match. Nonetheless, it can be helpful to compare the B200 and H200 based on specs alone.

### H200 vs. B200: Key Specs Compared

Here's a side-by-side look at some key features of the H200 and B200:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Feature
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                H200
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                B200
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2">Hopper</td>
            <td class="border border-gray-300 px-4 py-2">Blackwell</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory
            </td>
            <td class="border border-gray-300 px-4 py-2">141 GB HBM3e</td>
            <td class="border border-gray-300 px-4 py-2">192 GB HBM3e</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Bandwidth
            </td>
            <td class="border border-gray-300 px-4 py-2">4.8 TB/s</td>
            <td class="border border-gray-300 px-4 py-2">8 TB/s</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                CUDA Cores
            </td>
            <td class="border border-gray-300 px-4 py-2">16,896</td>
            <td class="border border-gray-300 px-4 py-2">16,896 x2</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Cores
            </td>
            <td class="border border-gray-300 px-4 py-2">528 (4th Gen)</td>
            <td class="border border-gray-300 px-4 py-2">528 x2 (5th Gen)</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Boost Clock
            </td>
            <td class="border border-gray-300 px-4 py-2">1.98 GHz</td>
            <td class="border border-gray-300 px-4 py-2">1.98 GHz</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Transformer Engine
            </td>
            <td class="border border-gray-300 px-4 py-2">Single</td>
            <td class="border border-gray-300 px-4 py-2">Single (2nd gen)</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Form Factor
            </td>
            <td class="border border-gray-300 px-4 py-2">SXM</td>
            <td class="border border-gray-300 px-4 py-2">SXM</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Interconnect
            </td>
            <td class="border border-gray-300 px-4 py-2">
                NVLink 4 (900 GB/s) per GPU
            </td>
            <td class="border border-gray-300 px-4 py-2">
                NVLink 5 (1.8 TB/s) per GPU
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Precision Formats
            </td>
            <td class="border border-gray-300 px-4 py-2">
                FP8, FP16/BF16, TF32, FP32, FP64, INT8
            </td>
            <td class="border border-gray-300 px-4 py-2">
                FP4, FP6, FP8, FP16/BF16, TF32, FP32, FP64, INT8
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Multi-Instance GPUs
            </td>
            <td class="border border-gray-300 px-4 py-2">Up to 7</td>
            <td class="border border-gray-300 px-4 py-2">Up to 7</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Max Thermal Design Power (TDP)
            </td>
            <td class="border border-gray-300 px-4 py-2">700 W</td>
            <td class="border border-gray-300 px-4 py-2">1000 W</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Recommended Power Supply
            </td>
            <td class="border border-gray-300 px-4 py-2">1100 W</td>
            <td class="border border-gray-300 px-4 py-2">1400 W</td>
        </tr>
    </tbody>
</table>

Choosing the right GPU often comes down to more than just specs, however.

### Use Cases: Which GPU Should You Choose?

While both the H200 and B200 deliver game-changing performance capabilities, they each have distinct advantages depending on your needs.

#### You should consider the H200 if you prefer:

-   **A high-performance GPU** ideal for fine-tuning LLMs and high-throughput inference at the enterprise level without getting into exascale territory.

-   **A proven workhorse** that balances strong performance with lower TDP and cost.

-   **Being able to seamlessly upgrade** from existing Hopper deployments without overhauling infrastructure.

#### Or the B200 might be the right choice if you need:

-   **To push model boundaries** and handle long-context and/or multi-modal foundation model training at scale.

-   **The absolute maximum throughput and efficiency** for multi-GPU or multi-node clusters.

-   **A future-proof investment** for infrastructure built to support the next generation of cutting-edge AI and AGI.

### Conclusion

At **Vast.ai**, we know that the choice between GPUs like the H200 and B200 can depend on a lot more than just performance. Access and cost are important factors, as well.

The H200 and B200 represent the very best of NVIDIA's current lineup, and they're among the most powerful and expensive GPUs in the world today. Buying them outright is undoubtedly out of reach for most teams.

That's what we're here for. With Vast, you can rent high-end GPUs like the [A100](https://vast.ai/article/H100-vs-A100-Comparing-two-Powerhouse-GPUs), [H100](https://vast.ai/article/nvidia-h100-vs-l40s-power-meets-versatility), [H200](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights), and soon the B200, with on-demand pricing and spot instances – and pay only for what you use.

On average, **Vast users save 5–6X on GPU compute** compared to traditional cloud providers. You'll find a variety of options on our platform to fit your needs, so you can focus on what really matters: your work, not your budget.

Explore Vast today and enjoy enterprise-grade compute without enterprise-level costs.


The H200 and B200 GPUs represent the top end of NVIDIA's current lineup for AI and high-performance computing (HPC). Both are designed for serious, large-scale workloads – from training foundation models to running complex scientific simulations.

NVIDIA H200 vs. B200: Comparing Datacenter-Grade Accelerators

Deploy DeepSeek R1 on Vast.ai with vLLM and integrate Crew AI for scalable multi-agent workflows. Create cost-effective, OpenAI-compatible AI endpoints.

Running DeepSeek R1 + Crew AI on Vast.ai


## Introduction

OpenAI has just released the GPT-OSS family of models, marking their return to open-weight model releases. The family includes GPT-OSS-120B (120 billion parameters) and GPT-OSS-20B (20 billion parameters), offering developers access to state-of-the-art language capabilities previously locked behind proprietary APIs.

GPT-OSS offers configurable reasoning capabilities with reasoning levels - from "low" for quick responses to "high" for complex problem-solving. The model uses OpenAI's harmony encoding system, which structures conversations and reasoning in a sophisticated format that enables fine-grained control over the model's thinking process.

This breakthrough means you can now:

-   Deploy OpenAI-quality models on your own infrastructure
-   Scale inference according to your exact needs

Vast.ai provides the perfect platform for running GPT-OSS models. With its marketplace of GPUs, you can choose the right hardware for your needs - whether running the efficient 20B model or the powerful 120B variant.

In this guide, we'll show you how to deploy both GPT-OSS models on Vast.ai using vLLM for optimized inference, with a focus on the 120B model. You'll learn how to interact with these models using the harmony encoding system for different reasoning levels.

## Setting Up the Environment

Before we can deploy our model, we need to set up our Vast.ai environment. First, install the Vast SDK:

```bash
# Install required packages
pip install --upgrade vastai
pip install --upgrade openai
pip install --upgrade openai-harmony
```

Set up your Vast API key (available from your [Account Page](https://cloud.vast.ai/account/)):

```bash
# Set your Vast.ai API key
export VAST_API_KEY="" # Your key here
vastai set api-key $VAST_API_KEY
```

## Choosing the Right Hardware

The GPT-OSS models have different hardware requirements:

-   **GPT-OSS-120B**: Requires an H100 GPU with 80GB VRAM
-   **GPT-OSS-20B**: Requires 16GB+ VRAM

For this guide, we'll demonstrate with the 120B model on H100. Let's search for suitable instances:

```bash
# Search for suitable GPU instances
vastai search offers " \
gpu_name = H100_SXM \
geolocation=US \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 120 \
rentable = true"
```

## Deploying GPT-OSS with vLLM

Now let's deploy our GPT-OSS instance using vLLM's optimized inference server. We'll use the latest vLLM image with special GPT-OSS support. The pip install commands are from the model's Hugging Face pages ([120B](https://huggingface.co/openai/gpt-oss-120b), [20B](https://huggingface.co/openai/gpt-oss-20b)).

### Deploying GPT-OSS-120B (H100 recommended)

```bash
# Deploy vLLM instance for 120B model
export INSTANCE_ID= # Insert instance ID from search results

vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 160  --onstart-cmd 'uv pip install --system --upgrade transformers kernels torch openai; uv pip install --system --pre vllm==0.10.1+gptoss --extra-index-url https://wheels.vllm.ai/gpt-oss/ --extra-index-url https://download.pytorch.org/whl/nightly/cu128; vllm serve openai/gpt-oss-120b --max-model-len 80000'
```

### Alternative: Deploying GPT-OSS-20B on A100

Here's how to deploy the 20B model on an A100 GPU:

```bash
# Deploy vLLM instance for 20B model
export INSTANCE_ID= # Insert instance ID from search results

vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 80  --onstart-cmd 'uv pip install --system --upgrade transformers kernels torch openai; uv pip install --system --pre vllm==0.10.1+gptoss --extra-index-url https://wheels.vllm.ai/gpt-oss/ --extra-index-url https://download.pytorch.org/whl/nightly/cu128; vllm serve openai/gpt-oss-20b'
```

After deployment, wait for the model to download and start serving. You can monitor the logs to see when it's ready. Once running, find your instance's IP address and port from the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console.

## Interacting with GPT-OSS Using Harmony Encoding

GPT-OSS uses OpenAI's harmony encoding system, which provides structured conversation formatting and enables different reasoning levels. Let's set up our client:

```python
from openai import OpenAI
from openai_harmony import (
    load_harmony_encoding,
    HarmonyEncodingName,
    Role,
    Message,
    Conversation,
    SystemContent,
    DeveloperContent,
)

# Your server details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize client
client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

# Load harmony encoding
enc = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)

## Simple Chat Function
def chat_gpt_oss(prompt, reasoning="medium"):
    """
    Simple chat with GPT-OSS
    reasoning: "low", "medium", or "high"
    """
    # Create conversation with harmony format
    conversation = Conversation.from_messages([
        Message.from_role_and_content(Role.SYSTEM, SystemContent.new()),
        Message.from_role_and_content(
            Role.DEVELOPER,
            DeveloperContent.new().with_instructions(f"Reasoning: {reasoning}")
        ),
        Message.from_role_and_content(Role.USER, prompt),
    ])

    # Get formatted prompt
    formatted_prompt = enc.render_conversation_for_completion(conversation, Role.ASSISTANT)

    # Make API call
    response = client.completions.create(
        model="openai/gpt-oss-120b",
        prompt=formatted_prompt,
        max_tokens=2000,
        temperature=0.7,
    )

    return response.choices[0].text


prompt = "Explain quantum computing to a 4th grader. Keep in concise."
```

## Comparing Reasoning Levels

One of GPT-OSS's most interesting features is its configurable reasoning levels. Let's see how the model responds differently to the same prompt with low and high reasoning:

### Low Reasoning Response

```python
result = chat_gpt_oss(prompt, "low")
print("----LOW REASONING----")
print(result)
```

Output:

```
----LOW REASONING----
analysisWe need to explain quantum computing to a 4th grader, concise. Use simple analogies, like bits as light switches, qubits as coins that can be heads, tails, or both. Keep short.assistantfinal**Quantum Computing in a Nutshell (for a 4th‑grader)**

1. **Regular computers = light switches**
   - A normal computer works with bits that are either **0** (off) or **1** (on), just like a light switch that's either down or up.

2. **Quantum computers = magic coins**
   - A quantum computer uses **qubits**.
   - Think of a qubit like a coin that can be **heads (0)**, **tails (1)**, **or both at the same time** while it's spinning in the air. That "both at once" is called **superposition**.

3. **Working together**
   - When you have many regular bits, you have to try every possible combination one after another.
   - With many qubits, they can be in many combinations **all at once** because they can be linked together in a special way called **entanglement**. It's like a group of magic coins that always know what the others are doing, even if they're far apart.

4. **Why it matters**
   - Because of superposition and entanglement, a quantum computer can solve some puzzles (like cracking codes or simulating molecules) **much faster** than a regular computer.

**Bottom line:** A regular computer is like a row of on/off switches; a quantum computer is like a bunch of spinning magic coins that can be many things at once, letting it try many solutions at the same time.
```

### High Reasoning Response

```python
result = chat_gpt_oss(prompt, "high")
print("----HIGH REASONING----")
print(result)
```

Output:

```
----HIGH REASONING----
analysisWe need to explain quantum computing to a 4th grader, concise. Use simple analogies, avoid jargon, but keep accurate. Should be short. Provide analogies like coins, dice, superposition, entanglement. Keep concise.assistantfinal**What is a quantum computer?**

Imagine you have a super‑smart robot that can solve puzzles much faster than a regular computer. A quantum computer is that robot, but instead of using normal "on/off" switches (bits) like a regular computer, it uses tiny, magical switches called **qubits**.

### Bits vs. Qubits
- **Bit (regular computer):** Like a tiny light that can be **off (0)** or **on (1)**.
- **Qubit (quantum computer):** Like a tiny spinning top that can be **0**, **1**, **or both at the same time** (this is called *superposition*).

### Why is that cool?
If you have 10 bits, they can show only one of 2¹⁰ = 1,024 possible patterns at a time.
If you have 10 qubits, they can be in all 2¹⁰ = 1,024 patterns **at once**! That lets a quantum computer try many solutions together.

### A Simple Analogy
Think of a maze puzzle:

- **Regular computer:** Sends a single mouse down the maze, step by step, until it finds the exit.
- **Quantum computer:** Sends a whole swarm of mice that can explore every possible path at the same time and tell you the exit almost instantly.

### Entanglement (Teamwork)
Sometimes two qubits become "best friends" and link together, no matter how far apart they are. Changing one instantly changes the other. This is called **entanglement**, and it lets quantum computers coordinate their work in a super‑efficient way.

### Bottom Line
A quantum computer uses qubits that can be 0, 1, or both at once, and they can be entangled with each other. This lets the machine look at many possibilities at the same time, solving certain problems way faster than ordinary computers.

So, it's like a super‑fast, super‑clever puzzle‑solver that can try lots of answers all at once!
```

## Comparing Low vs High Reasoning Outputs

The quantum computing explanations demonstrate the distinct capabilities of GPT-OSS-120B's reasoning levels. With low reasoning, the model produced a concise, structured explanation using simple analogies like "magic coins" and "light switches" - delivering clear concepts in a compact format suitable for quick understanding.

The high reasoning response revealed significantly more sophisticated thinking. The model provided detailed analogies like the maze puzzle with mice, explained mathematical concepts (2¹⁰ patterns), and structured the explanation with clear sections and formatting. Most notably, it offered multiple perspectives on the same concepts - from the spinning top analogy to the teamwork explanation of entanglement.

This difference in reasoning depth represents a significant advancement in model control. Rather than simply adjusting temperature or other parameters, GPT-OSS allows direct specification of how thoroughly the model should think through problems. The low reasoning approach prioritized efficiency while maintaining accuracy, while high reasoning prioritized comprehensive understanding with rich examples and detailed explanations.

## Conclusion

OpenAI's first open-source model thus introduces not just accessibility to advanced language capabilities, but a new paradigm for controlling model reasoning depth - enabling applications to dynamically adjust between quick responses and thorough analysis based on context and requirements.


Learn how to deploy and run OpenAI's GPT-OSS models on Vast.ai using vLLM, with configurable reasoning levels and harmony encoding system.

Running OpenAI's GPT-OSS on Vast.ai


At **Vast.ai**, we believe privacy and security are inextricably woven into our mission to equip customers worldwide with accessible, scalable GPUs. In honor of this commitment, we are pleased to announce that we have achieved **SOC 2 Type II certification** – an independent assurance that with us, your data is secure, reliably available, and continuously protected.

Vast.ai’s Enduring Security Commitment

Earlier this year, our [SOC 2 Type I audit](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) confirmed we have the right internal controls in place to meet trust services criteria for data security, integrity, and privacy. The Type II audit put those controls to the test, evaluating their operational effectiveness over an extended period.

Achieving **SOC 2 Type II certification** is independent validation that the security infrastructure we've built over 6+ years of trusted service continues to meet rigorous [compliance standards](https://vast.ai/article/security-and-compliance-at-vast-ai) in day-to-day execution. This achievement strengthens our ability to support the evolving security needs of enterprises, research institutions, and organizations building on our infrastructure.

Here's a quick look at SOC 2 Type II and what it entails.

### What Is SOC 2 Type II?

System and Organization Controls 2 (SOC 2\) audits – developed by the American Institute of Certified Public Accountants (AICPA) – protect customer data by rigorously evaluating an organization’s internal controls across security, confidentiality, privacy, and processing integrity benchmarks.

- **Type I** reports provide a control design snapshot at a single point in time.

- **Type II** reports go further, assessing how those controls perform over months of continuous operation – confirming they're both well-designed and consistently implemented.

Achieving both Type I and Type II certifications means your sensitive data is secure, available, and privately stored across all our systems. This establishes a foundation of enduring excellence aligned with rigorous industry standards for all customers, which we can tailor to your unique data security and regulatory compliance needs to scale infrastructure and further reduce risks.

After completing our initial 3-month Type II audit, we’ve launched a standard 12-month cycle to ensure we have no gaps in coverage. This two-phase strategy lets us move fast on improvements, then settle into an annual rhythm of verification. We are now in the 12-month cycle with continuous coverage year round.

### Vast.ai's Secure Cloud Offering

For customers with stricter security and compliance requirements, our [**Secure Cloud**](https://cloud.vast.ai/?secure_cloud=true) offering provides high-security GPU access through our certified datacenter partners. This environment, already backed by stringent operational and security controls, now benefits from the added assurance of a SOC 2 Type II audit across our platform infrastructure.

Our **[Trust Center](https://trust.vast.ai/?utm_source=blog)** offers a deeper look into our controls, compliance stance, FAQs and Subprocessors, and our other compliance certifications provide additional safeguards.

Why This Matters

Earning our **SOC 2 Type II certification** is another step forward in strengthening our foundation of trust and security that our customers rely on every day. It reflects both our adherence to proven industry practices and our ongoing commitment to transparency.

This achievement isn't the end of our compliance journey – it's part of an ongoing initiative. Vast.ai will undergo SOC 2 Type II audits every 12 months to ensure continual coverage.

Additionally, we've requested an [**SOC 3 report**](https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-3), which will provide a publicly available summary of our security posture for those who want to review our controls without accessing the full SOC 2 report.

### Our Ongoing Commitment

Achieving the **SOC 2 Type II certification** is a crucial step in our compliance roadmap as we continue to build trustworthy and dependable GPU infrastructure for our users. Alongside other security and [compliance measures](https://vast.ai/compliance) across cloud and AI services, such as our [SOC 2 Type I certification](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) and [Vulnerability Bounty Program](https://vast.ai/article/announcing-the-vast-ai-vulnerability-bounty-program), we are weaving security and trust throughout our platform for every member of our community.

Through this integrated approach, we’re dedicated to protecting the privacy and integrity of the data entrusted to us by developers, researchers, and enterprise teams around the world. We take this responsibility seriously as a distributed peer-to-peer platform powering global compute, and we look forward to sharing future updates as we continue to advance our security and compliance roadmap.

For more information on Vast.ai's comprehensive security and compliance stance, or to request a copy of our SOC 2 reports, please contact us at [compliance@vast.ai](mailto:compliance@vast.ai).

Thank you to our team at Vast.ai and our audit partners for helping us achieve this milestone.

We’re committed to protecting your data and upholding the trust you place in us, and we’ll keep you updated on our ongoing progress in our mission to provide powerful, dependable GPUs.


Vast.ai has achieved SOC 2 Type II certification, demonstrating our commitment to enterprise-grade security, data protection, and compliance. Learn about our comprehensive security controls, Secure Cloud offering, and ongoing compliance roadmap that ensures your data remains secure and protected.

Vast.ai Achieves SOC 2 Type II Certification


The landscape of AI inference is rapidly evolving, with organizations seeking cost-effective solutions that don't compromise on performance or control. While cloud providers offer convenient managed services, they often come with premium pricing and limited customization options. On the other hand, on-premises deployments provide control but require significant hardware investments.

A hybrid approach offers the best of both worlds: maintaining local control over your inference pipeline while leveraging cost-effective remote GPU resources. This blog post demonstrates how to implement such a system using LiteLLM as a local proxy and Vast.ai for remote GPU hosting.

## The Hybrid Architecture Advantage

This architecture combines local control with cost-effective cloud GPU access through two key components:

**LiteLLM** is a Python SDK and proxy server that provides a unified OpenAI-compatible interface for 100+ LLM APIs from different providers. It includes features like cost tracking, rate limiting, and logging capabilities when used as a proxy server.

**Vast.ai** is a GPU marketplace that offers access to over 20,000 GPUs from secure datacenters. According to their website, users can save up to 80% compared to traditional cloud services with their pay-as-you-go pricing model.

## What You'll Learn

In this blog post, we'll deploy a complete hybrid inference pipeline:

1. Deploy a vLLM server with the DeepSeek-R1 model on Vast.ai
2. Configure LiteLLM locally to proxy requests to the remote server
3. Test the complete pipeline using OpenAI client libraries
4. Demonstrate advanced reasoning capabilities

The result is a flexible, cost-effective inference setup that you control locally while leveraging remote GPU resources.

## Deploy LiteLLM + vLLM Pipeline on Vast.ai

Let's start by installing and configuring the Vast AI API client.

### Install Vast AI Client

Get your API key from the [Account Page](https://cloud.vast.ai/account/) in the Vast Console.

```bash
pip install --upgrade vastai
```

```bash
# Here we will set our api key
export VAST_API_KEY="" #Your key here
vastai set api-key $VAST_API_KEY
```

### Hardware Requirements

For our LiteLLM + vLLM pipeline, we need a GPU with 24GB+ RAM for the DeepSeek-R1 model weights and KV cache, plus a static IP address with direct ports for stable connections.

Let's search for suitable instances:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 24 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

## Deploy vLLM on Vast.ai

Choose an instance ID from the search results above and deploy using the Docker command below.

This deployment uses the vllm/vllm-openai:latest image to serve the deepseek-ai/DeepSeek-R1-0528-Qwen3-8B model with reasoning capabilities on port 8000:

```bash
export INSTANCE_ID="" #insert instance ID

vastai create instance $INSTANCE_ID \
  --image vllm/vllm-openai:latest \
  --env '-p 8000:8000' \
  --disk 60 \
  --args \
    --model deepseek-ai/DeepSeek-R1-0528-Qwen3-8B \
    --served-model-name deepseek \
    --max-model-len 4096 \
    --reasoning-parser qwen3 \
```

## Get Vast IP and Port

After deployment, go to the [Instances Tab](https://cloud.vast.ai/instances/) in the Vast AI Console and find your instance.

Click the IP address button at the top of the instance. A panel will show the IP address and forwarded ports:

```
Open Ports
XX.XX.XXX.XX:YYYY -> 8000/tcp
```

Note down VAST_IP (XX.XX.XXX.XX) and VAST_PORT (YYYY) for the next steps.

## Installing LiteLLM and Dependencies

Now let's install LiteLLM locally to proxy requests to our Vast.ai server:

```bash
pip install litellm
pip install 'litellm[proxy]'
```

### Configure LiteLLM

Set your Vast.ai instance details from the previous step:

```python
# Configure your Vast.ai instance details
VAST_IP = ""
VAST_PORT = ""
MODEL_NAME = "deepseek"

# Configure LiteLLM settings
LITELLM_PORT = 4000  # Change this if port 4000 is in use

# Create the API base URL
API_BASE = f"http://{VAST_IP}:{VAST_PORT}/v1"
print(f"Vast.ai API Base URL: {API_BASE}")
print(f"LiteLLM will run on port: {LITELLM_PORT}")
```

```python
# Write config file with variables
config_content = f"""model_list:
  - model_name: {MODEL_NAME}
    litellm_params:
      model: openai/{MODEL_NAME}
      api_base: {API_BASE}
      api_key: fake-key
general_settings:
  master_key: fake-key
"""

with open('litellm_config.yaml', 'w') as f:
    f.write(config_content)
    
print("Config file created with:")
print(f"- Model: {MODEL_NAME}")
print(f"- API Base: {API_BASE}")
```

```python
import subprocess
import time

# Kill any existing LiteLLM processes
subprocess.run("pkill -f litellm || true", shell=True)
time.sleep(1)

# Start LiteLLM
cmd = f"nohup litellm --config litellm_config.yaml --port {LITELLM_PORT} --host 0.0.0.0 > litellm.log 2>&1 &"
subprocess.run(cmd, shell=True)
time.sleep(5)

print(f"✅ LiteLLM running on port: {LITELLM_PORT}")
print(f"URL: http://localhost:{LITELLM_PORT}/v1")

# Save port for other cells
with open('.litellm_port', 'w') as f:
    f.write(str(LITELLM_PORT))
```

Output:
```
✅ LiteLLM running on port: 4000
URL: http://localhost:4000/v1
```

## Install OpenAI Python Client

Now install the OpenAI client to test our setup:

```bash
pip install openai
```

## Test the Pipeline

Let's test our complete LiteLLM + Vast.ai setup with a basic API call:

```python
from openai import OpenAI

client = OpenAI(
    api_key="fake-key",
    base_url=f"http://localhost:{LITELLM_PORT}/v1"
)

# Basic test
try:
    response = client.chat.completions.create(
        model="deepseek",
        messages=[
            {"role": "user", "content": "Hello! How are you?"}
        ],
        max_tokens=1000
    )
    print("Response:", response.choices[0].message.content)
    
except Exception as e:
    print(f"❌ Error: {e}")
```

Output:
```
Response: 
Hello! I'm just a friendly AI here, so I don't get tired or have feelings—but I'm fully ready and happy to help you! How are you doing today? Let me know if there's anything you'd like to chat about or need assistance with.
```

Perfect! Now let's test the reasoning capabilities of the DeepSeek model:

```python
try:
    response_1 = client.chat.completions.create(
        model="deepseek",
        messages=[
            {"role": "user", "content": "what is 8 * 7?, Give me your answer in step by step reasoning."}
        ],
        max_tokens=3000
    )
    print("Response:", response_1.choices[0].message.content)
    
except Exception as e:
    print(f"❌ Error: {e}")
```

Output:
```
Response: 
### Step-by-Step Reasoning for 8 * 7

Multiplication is a basic arithmetic operation that can be thought of as repeated addition. To find the product of 8 and 7, you can add the number 8 to itself 7 times (or equivalently, add the number 7 to itself 8 times). I'll use the repeated addition method with the number 8 to make it clear.

Start with the first 8:
- Add the second 8: 8 + 8 = 16
- Add the third 8: 16 + 8 = 24
- Add the fourth 8: 24 + 8 = 32
- Add the fifth 8: 32 + 8 = 40
- Add the sixth 8: 40 + 8 = 48
- Add the seventh 8: 48 + 8 = 56

As shown, after adding 8 seven times, the result is 56.

Alternatively, you can verify this with another method, such as multiplying 7 eight times or subtracting 8 from 8 * 8 (since 8 * 7 = (8 * 8) - 8 = 64 - 8 = 56). Both methods confirm the result.

**Final Answer:** 56
```

The pipeline is working correctly. You now have a complete LiteLLM + Vast.ai setup with local proxy control and remote GPU inference.

**Note**: For production deployments, consider using Docker containers for LiteLLM rather than the command-line approach shown here.

## Conclusion

This hybrid inference architecture demonstrates how modern AI deployment can balance cost, control, and performance. By combining LiteLLM's local proxy capabilities with Vast.ai's GPU marketplace, we've created a system that offers several key advantages:

**Cost Considerations**: Vast.ai's marketplace model and pay-as-you-go pricing can provide cost savings compared to traditional cloud providers, while running LiteLLM locally eliminates additional hosting costs for the proxy layer.

**Local Control**: Running LiteLLM locally provides control over request routing, logging, and configuration. The proxy server includes built-in features for cost tracking, rate limiting, and observability integrations.

**API Compatibility**: LiteLLM's OpenAI-compatible interface allows existing applications to integrate with minimal changes, while supporting multiple LLM providers through a unified API.

**Flexible Architecture**: This setup allows you to experiment with different models and providers while maintaining a consistent interface for your applications.

Whether you're building AI-powered applications, conducting research, or optimizing existing ML pipelines, this hybrid approach provides a practical path to cost-effective, controllable AI inference that doesn't sacrifice capability for economics.

Deploy hybrid AI inference using LiteLLM + Vast.ai. Run local OpenAI-compatible proxies with remote GPU models for scalable, cost-effective LLM performance.

Hybrid AI Inference: Local LiteLLM Proxy with Remote Vast.ai GPU

Deploy Llama 4, Qwen3, and Kimi K2 on Vast.ai in minutes. Use optimized templates like Oobabooga, vLLM, HuggingFace TGI, and Open WebUI—no setup hassle.

Modular MAX vs vLLM Performance Comparison on Vast.ai


Today, we’re announcing the launch of our Vast.ai Vulnerability Bounty Program. This is a new initiative we’ve created to help us improve, expand, and innovate our platform while making it more secure. Vast.ai’s mission is to provide the best GPUs for AI compute at accessible and affordable prices for everyone who needs it. With this bounty program, we’re inviting AI developers, researchers, and enthusiasts to collaborate with us. The launch of this program is in direct response to our community’s feedback, and we appreciate all the collaboration happening already.

### **Responsible Vulnerability Disclosure**

### We take security seriously and encourage responsible disclosure of any [vulnerabilities](https://vast.ai/vulnerability-disclosure-policy).

**What is the [Vast.ai](http://Vast.ai) Vulnerability Bounty program?**

The program provides a structured way to report bugs, suggest new features, or responsibly uncover vulnerabilities. Whether you’re a security researcher or a community builder, there’s an opportunity for you to do the right thing and be rewarded.

### **What Is In Scope?**

-   Vast.ai web console, REST API & billing flows
-   Provider Daemon code (host agent)
-   Match‑making & pricing engine
-   Default Docker & KVM isolation on reference images
-   GPU memory‑isolation / tenant breakout flaws

What is out of Scope?

-   User workloads & third‑party container images
-   Social‑engineering, physical security, or denial‑of‑service (DoS) tests
-   Brute‑force attacks against customer passwords or MFA
-   Any activity that violates applicable law, exports regulations, or provider Terms of Service

### If you discover a security issue, we ask that you:

-   Report it directly to our team at [security@vast.ai](mailto:security@vast.ai?subject=Vulnerability%20Bounty%20Program%20Inquiry).

-   Allow us a reasonable time to address the issue before any public disclosure.

### In return, we commit to:

-   Acknowledging your report promptly and keeping you informed of progress.

-   Recognizing your efforts with a form of bounty rewards, platform credit, or joining our hall of fame.

### **How to Get Started?**

1. Join the [Vast.ai Discord](https://discord.gg/NCgfu5BrdQ) to connect with our team and community.

2. Submit your bounty contributions and start hunting.

Please review [the policy](https://vast.ai/vulnerability-disclosure-policy) for the Rules of Engagement and specifics on scope, which will be updated periodically.


Help secure Vast.ai and earn rewards. Join our Vulnerability Bounty Program to report bugs, suggest features, and collaborate with the AI infrastructure community.

Announcing the Vast.ai Vulnerability Bounty Program


In Part 1 of this series, we created a quantized version of the Qwen3-8B model (Qwen3-8B-W8A8) using LLM-Compressor with 8-bit weight and activation quantization. In this notebook, we'll demonstrate how to deploy and compare this quantized model against its full precision counterpart (Qwen3-8B) using Vast.ai. We'll examine both deployment processes and compare the outputs to assess any quality differences between the models while highlighting the efficiency gains from quantization.

## Key benefits of quantized models:

- Reduced memory footprint (approximately 1.7x smaller in our case)
- Lower inference latency
- Decreased computational requirements
- More affordable deployment options

Let's start by exploring how to deploy our quantized model on Vast.ai.

## Deploying our Model on Vast.ai

First, we will install the Vast.ai SDK and input our API key. This allows us to interact with the Vast.ai platform programmatically and manage our GPU instances.

```bash
#In an environment of your choice
pip install --upgrade vastai
```

```bash
# Here we will set our api key
export VAST_API_KEY="VAST_API_KEY" #Your key here
vastai set api-key $VAST_API_KEY
```

Next, we'll search for an appropriate GPU instance to serve our quantized model. Since this is a W8A8 quantized version of Qwen3-8B, we can use a machine with less VRAM than would be required for the full model. We're looking for machines with at least 24GB of VRAM, which is sufficient for our 8-bit quantized model and its context window:

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 24 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 20 \
rentable = true"
```

Once we've identified a suitable instance from the search results, we'll deploy our quantized model to that instance. We're using the vLLM Docker image with an OpenAI-compatible API, which makes it easy to serve and interact with our model.

Note that we're deploying the W8A8 quantized version of Qwen3-8B that was created and uploaded to Hugging Face in Part 1. Please replace `your-hf-username` with your actual Hugging Face username to use your compressed model:

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 20 --args --model your-hf-username/Qwen3-8B-W8A8
```

## Calling Our Model Using the OpenAI SDK

Now that our model is deployed and running on Vast.ai, we'll set up a client to interact with it. The vLLM server exposes an OpenAI-compatible API, allowing us to use the OpenAI SDK to send requests to our model.

First, we need to install the OpenAI SDK if you haven't already:

```bash
pip install openai
```

Then, we use the OpenAI SDK and the `VAST_IP_ADDRESS` and `VAST_PORT` from our instance to call our model.

Also add your hugginface username `HF_USERNAME` to call the model we saved to hugginface.

```python
from openai import OpenAI

VAST_IP_ADDRESS = "VAST_IP_ADDRESS"
VAST_PORT = "VAST_PORT"

HF_USERNAME="HF_USERNAME"

# Create a client instance pointing to the vLLM server
client = OpenAI(
    api_key="dummy-key",  # vLLM doesn't require a real API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"  # Point to your vLLM server
)

# Call the model
response = client.chat.completions.create(
    model=f"{HF_USERNAME}/Qwen3-8B-W8A8",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What are the major applications of deep learning?"}
    ],
    temperature=0.7,
    max_tokens=1000
)

# Print the response
print(response.choices[0].message.content)
```

Output:

```
<think>
Okay, the user is asking about the major applications of deep learning. I need to provide a comprehensive yet clear answer. Let me start by recalling the different areas where deep learning is commonly used.

First, computer vision comes to mind. That includes image recognition, object detection, and maybe even image generation. Then there's natural language processing, like chatbots, translation, and sentiment analysis.

I should also mention speech recognition, which is used in virtual assistants. Autonomous vehicles rely on deep learning for perception tasks. Healthcare is another big area, such as medical imaging analysis and drug discovery.

Finance uses deep learning for fraud detection and algorithmic trading. Recommender systems in e-commerce and streaming services are also important. Maybe touch on generative models like GANs and diffusion models for creating art or text.

Wait, I should categorize these applications to make it organized. Let me list them under different headings. Also, I need to ensure I don't miss any major areas. Oh, maybe mention robotics and control systems too.

I should check if there are any recent advancements or emerging fields. For example, deep learning in gaming, like reinforcement learning for game AI. Also, maybe something about anomaly detection in various industries.

Need to keep the explanation straightforward without too much jargon. Make sure each application is explained briefly but clearly. Avoid going into too much technical detail unless necessary.

Let me structure the answer with clear sections for each major application. Start with an introduction, then list each area with a brief description. Conclude by summarizing the versatility of deep learning.

Wait, the user might be looking for both current and potential future applications. Should I include that? Maybe a short note at the end about ongoing research and future possibilities.

Also, check for any overlaps or if some applications are better categorized under specific fields. For example, NLP and speech recognition are both related to language processing.

I think that's a solid structure. Now, I'll draft the answer accordingly, making sure it's informative and covers all key areas without being too verbose.
</think>

Deep learning, a subset of machine learning, has revolutionized numerous fields due to its ability to model complex patterns in data. Here are its major applications across various domains:

---

### **1. Computer Vision**
- **Image Recognition**: Identifying objects, scenes, and patterns in images (e.g., facial recognition, medical imaging analysis).
- **Object Detection**: Locating and classifying multiple objects in a single image (e.g., autonomous vehicles, surveillance systems).
- **Image Generation**: Creating new images using Generative Adversarial Networks (GANs) or diffusion models (e.g., AI art tools).
- **Video Analysis**: Action recognition, video summarization, and scene understanding.

---

### **2. Natural Language Processing (NLP)**
- **Text Generation**: Writing articles, stories, or code using models like GPT or BERT.
- **Translation**: Real-time language translation (e.g., Google Translate).
- **Sentiment Analysis**: Determining emotions in text (e.g., social media monitoring).
- **Chatbots & Virtual Assistants**: Interactive AI assistants (e.g., Siri, Alexa, customer service bots).

---

### **3. Speech Recognition & Synthesis**
- **Voice Assistants**: Converting speech to text (e.g., Alexa, Google Assistant).
- **Speech-to-Text**: Transcribing audio for accessibility or transcription services.
- **Text-to-Speech**: Synthesizing natural-sounding speech (e.g., audiobooks, virtual characters).

---

### **4. Autonomous Systems**
- **Self-Driving Cars**: Perceiving environments, navigating, and making real-time decisions.
- **Robotics**: Controlling robotic arms, navigation, and object manipulation.
- **Industrial Automation**: Predictive maintenance and quality control in manufacturing.

---

### **5. Healthcare**
- **Medical Imaging**: Diagnosing diseases (e.g., tumors in X-rays, MRIs) using CNNs.
- **Drug Discovery**: Accelerating the development of new medications through molecular modeling.
- **Personalized Medicine**: Predicting patient outcomes based on genetic data and medical history.

---

### **6. Finance**
- **Fraud Detection**: Identifying unusual transaction patterns.
- **Algorithmic Trading**: Predicting market trends and executing trades.
- **Risk Management**: Assessing credit risk and financial stability.

---

### **7. Recommender Systems**
- **E-commerce**: Personalized product recommendations (e.g., Amazon, Netflix).
- **Streaming Services**: Content suggestions based on user preferences.
- **Social Media**: Curating news feeds and trending topics.

---

### **8. Gaming & Entertainment**
- **Game AI**: Creating intelligent opponents or NPCs (e.g., AlphaGo, Dota 2 bots).
- **Content Creation**: Generating music, art, or video using deep learning models.

---
```

## Compare with full `Qwen/Qwen3-8B` Model

Now that we've seen the output from our quantized model, let's deploy and test the full precision version of the same model for comparison. This will help us evaluate any potential quality differences and understand the tradeoffs of quantization.

First, we need to find a more powerful machine with significantly more GPU memory. The full precision model requires approximately 3-4x more VRAM than its quantized counterpart. Let's search for instances with at least 80GB of VRAM:

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 80 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 40 \
rentable = true"
```

Now we'll deploy the full precision model. Notice the differences in our configuration compared to the quantized model:

1. We're requiring more disk space (40GB vs 20GB)
2. We're using a much more powerful GPU with significantly more VRAM
3. We're deploying the original model from Qwen directly (Qwen/Qwen3-8B)

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 40 --args --model  Qwen/Qwen3-8B
```

With the full precision model deployed, let's query it with the same prompt we used for the quantized model to perform a direct comparison of their outputs:

```python

VAST_IP_ADDRESS = "VAST_IP_ADDRESS"
VAST_PORT = "VAST_PORT"

# Create a client instance pointing to the vLLM server
client = OpenAI(
    api_key="dummy-key",  # vLLM doesn't require a real API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"  # Point to your vLLM server
)

# Call the model
response = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What are the major applications of deep learning?"}
    ],
    temperature=0.7,
    max_tokens=1000
)

# Print the response
print(response.choices[0].message.content)
```

```
<think>
Okay, the user is asking about the major applications of deep learning. Let me start by recalling what I know. Deep learning is a subset of machine learning that uses neural networks with many layers. It's been really popular in various fields. I need to list the main areas where it's applied.

First, computer vision comes to mind. Image recognition, object detection, maybe even things like facial recognition. Then there's natural language processing, like chatbots, translation services, text generation. Speech recognition is another big one, like virtual assistants.

What else? Autonomous vehicles use deep learning for perception tasks, like identifying objects on the road. Healthcare is another area, maybe medical imaging analysis, predicting diseases. Then there's recommendation systems used by companies like Netflix or Amazon.

Wait, what about robotics? Deep reinforcement learning is used in robotics for control systems. Also, maybe gaming, like in AlphaGo. Financial services for fraud detection or algorithmic trading.

I should also mention some more specialized areas like generative models for creating images or music, and maybe something about NLP like sentiment analysis. Oh, and maybe mention that these applications are driven by the ability of deep learning to handle large amounts of data and complex patterns.

Wait, the user might be looking for a comprehensive list, so I should structure it into categories. Let me check if I missed any major areas. Maybe cybersecurity? Or maybe something like anomaly detection in various industries. Also, maybe mention the difference between deep learning and traditional machine learning in terms of application.

I need to make sure each application is clearly explained with examples. Let me organize them into sections. Computer Vision, NLP, Speech Recognition, Autonomous Vehicles, Healthcare, Recommendation Systems, Robotics, Gaming, Financial Services, Cybersecurity, Generative Models, and maybe others.

I should also note that deep learning is used in both supervised and unsupervised tasks. Oh, and maybe mention some specific models like CNNs for images, RNNs or Transformers for NLP. But the user might not need the technical details, just the applications.

Wait, the user might be a student or someone new to the field, so keeping it clear and concise is important. Avoid jargon where possible. Also, make sure to highlight the impact of each application.

Let me make sure I cover all major areas. Maybe add something about drug discovery in healthcare. Or maybe mention that deep learning is used in weather prediction or climate modeling.

Hmm, I think that's a good list. Let me structure it with bullet points and explanations for each application. Also, maybe conclude with a note on the versatility and impact of deep learning.
</think>

Deep learning, a subset of machine learning, has revolutionized numerous fields due to its ability to model complex patterns and process large datasets. Here are the **major applications** of deep learning:

---

### **1. Computer Vision**
- **Image Recognition**: Identifying objects, people, and scenes in images (e.g., Google Photos, medical imaging).
- **Object Detection**: Locating and classifying objects in real-time (e.g., autonomous vehicles, security cameras).
- **Facial Recognition**: Used in biometric authentication (e.g., smartphones, border control).
- **Image Generation**: Creating realistic images (e.g., GANs for art, style transfer).

---

### **2. Natural Language Processing (NLP)**
- **Text Generation**: Writing articles, stories, or code (e.g., chatbots, AI assistants like Siri).
- **Machine Translation**: Languages like Google Translate use neural networks for real-time translation.
- **Sentiment Analysis**: Analyzing emotions in text (e.g., social media monitoring, customer feedback).
- **Speech Recognition**: Converting speech to text (e.g., voice assistants, transcription services).

---

### **3. Autonomous Systems**
- **Self-Driving Cars**: Detecting pedestrians, traffic signs, and obstacles (e.g., Tesla Autopilot, Waymo).
- **Robotics**: Controlling robotic arms and navigation systems (e.g., industrial robots, drones).
- **Reinforcement Learning**: Training agents for tasks like game playing (e.g., AlphaGo, Dota 2).

---

### **4. Healthcare**
- **Medical Imaging**: Diagnosing diseases from X-rays, MRIs, or CT scans (e.g., cancer detection).
- **Drug Discovery**: Accelerating the development of new drugs by predicting molecular interactions.
- **Personalized Medicine**: Tailoring treatments based on patient data (e.g., genomics analysis).

---

### **5. Recommendation Systems**
- **E-commerce**: Product recommendations (e.g., Amazon, Netflix).
- **Streaming Services**: Content suggestions (e.g., Spotify, YouTube).
- **Social Media**: Curating news feeds (e.g., Facebook, Instagram).

---

### **6. Finance**
- **Fraud Detection**: Identifying unusual
```

## Conclusion: Comparing Model Outputs and Size Tradeoffs

After deploying and testing both the quantized model (W8A8) and the full precision model, we can make several key observations:

### Model Size and Resource Requirements

- **Full Precision Model (Qwen3-8B)**:

  - Required ~80GB+ VRAM GPU
  - Needed 40GB disk space
  - Model size: ~16GB (FP16)

- **Quantized Model (Qwen3-8B-W8A8)**:
  - Required only 24GB VRAM GPU (~3.3x less memory)
  - Needed only 20GB disk space
  - Model size: ~9.5GB (8-bit weights and activations)

### Output Quality Comparison

Looking at the outputs from both models, we can observe:

1. **Structural Similarity**: Both models organized their responses with similar categories (Computer Vision, NLP, Healthcare, etc.) and maintained good formatting with markdown.

2. **Content Quality**: Both models produced high-quality, relevant content with appropriate examples for each application domain. The specific details varied slightly, but the core information was consistent.

3. **Reasoning Process**: Both models showed similar thinking processes in their `<think>` sections, demonstrating that the quantized model retained the structured reasoning capabilities of the original model.

4. **Minimal Quality Degradation**: The quantized model showed negligible degradation in output quality compared to the full precision model, despite being significantly smaller.

### Key Takeaways

1. **Deployment Accessibility**: The W8A8 quantized model makes deployment much more accessible, requiring far less expensive hardware while maintaining output quality.

2. **Cost Efficiency**: Using the quantized model can significantly reduce cloud GPU costs by allowing deployment on less expensive hardware.

3. **Production Readiness**: 8-bit quantization appears to be production-ready for this type of model, offering an excellent balance between efficiency and performance.

4. **Use Case Fit**: For most general text generation and understanding tasks, the quantized model appears to be a suitable replacement for the full precision model.

This notebook demonstrates that weight and activation quantization (W8A8) using LLM Compressor is an effective approach for deploying large language models in resource-constrained environments without significant quality compromises.

Vast.ai provides an excellent platform for this workflow, allowing us to use high-end GPUs for model compression and more cost-effective GPUs for efficient inference.


Compare quantized vs. full Qwen3-8B models deployed on Vast.ai. See quality vs. efficiency tradeoffs using W8A8 compression with LLM-Compressor and vLLM.

Using LLM-Compressor to Quantize Qwen3-8B on Vast.ai (Part 2)


As AI models grow increasingly powerful, they're also becoming increasingly expensive to deploy. The latest language models require massive amounts of GPU memory and computational resources, putting them out of reach for many teams and use cases. Model compression has emerged as a critical technique for making these models more accessible while maintaining their performance.

Today, we'll explore how to compress large language models using LLM-Compressor and deploy them cost-effectively on Vast.ai. We'll walk through the complete process of taking a 16GB model and reducing it to approximately 9.5GB while preserving quality—making deployment significantly more affordable and accessible.

## The Tools: LLM-Compressor and Vast.ai

### LLM-Compressor: Making Models Leaner

The vLLM team has developed [LLM-Compressor](https://github.com/vllm-project/llm-compressor) as a comprehensive solution for reducing model size without sacrificing performance. Unlike other compression tools that focus on a single technique, LLM-Compressor offers a complete toolkit for optimization.

What makes it particularly powerful:

- **Comprehensive approach**: Combines quantization, pruning, and distillation techniques that can be mixed and matched
- **Production-ready**: Built specifically for deployment scenarios with seamless vLLM integration
- **Developer-friendly**: The one-shot API handles the complex compression pipeline automatically
- **Proven techniques**: Implements industry-standard methods like GPTQ, AWQ, and SmoothQuant

### Vast.ai: Affordable GPU Power

Traditional cloud providers can make GPU-intensive tasks prohibitively expensive. Vast.ai changes this equation by creating a marketplace where you can access high-end GPUs at competitive rates—often significantly cheaper than major cloud platforms.

For model compression workflows, Vast.ai offers several advantages:

- **Flexible pricing**: Pay-per-hour model means you only pay for compression time, not idle resources
- **Hardware variety**: Choose from different GPU types based on your specific memory and compute requirements
- **Quick deployment**: Get instances running in minutes without complex setup procedures
- **Community-driven**: Access to a wider range of hardware options through the marketplace model

## In this Tutorial

This tutorial demonstrates how to:

1. Compress the `Qwen3-8B` model using LLM-Compressor with 8-bit weight and activation quantization (W8A8)
2. Upload the compressed `Qwen3-8B-W8A8` model to Hugging Face for easy sharing and deployment

In the second part of this series, we'll show how to deploy the compressed `Qwen3-8B-W8A8` model on Vast.ai for efficient inference and compare its performance with the full-precision model.

## GPU Requirements

Model compression requires significant computational resources, especially for larger models. For compressing the Qwen3-8B model, we recommend the following specifications:

### Minimum Requirements:

- **GPU Memory**: 30GB+ VRAM (e.g., A5000, A6000, RTX 4090, V100-32GB)
- **GPU Compute**: CUDA-compatible GPU with compute capability 7.0+
- **System RAM**: 64GB recommended (minimum 32GB)
- **Storage**: 80GB free disk space for model files and datasets
- **CUDA Version**: 11.8 or newer

### Why these requirements?

- The full Qwen3-8B model requires approximately 16GB just to load in FP16 format
- During compression, we need memory for both the original model and compression overhead
- Calibration datasets and intermediate files require additional storage space
- Model compilers and quantization tools have their own memory overhead

For larger models (12B+), you would need even more VRAM (40GB+) or techniques like model sharding.

## Renting an Instance on Vast.ai

1. Ensure that you have a Vast.ai account. If not, sign up at [vast.ai](https://vast.ai)
2. Go to the Vast Templates in the Console https://cloud.vast.ai/templates/
3. Select the `PyTorch (CuDNN Runtime)` Template - this provides a pre-configured environment with PyTorch and CUDA
4. Filter for an instance with the following specifications for optimal performance:
   - 1 GPU with high VRAM (A5000, A6000, RTX 4090, or similar)
   - 30GB+ of VRAM (required for full model loading plus quantization overhead)
   - 80GB+ of storage (for model weights, checkpoints, and datasets)
   - 32GB+ of system RAM (64GB recommended for smoother operation)
5. Select an instance and click rent - hourly pricing will vary based on the selected hardware
6. Install the Vast TLS certificate in your browser to access the notebook server securely https://docs.vast.ai/instances/jupyter#1SmCz
7. Go to your Instances https://cloud.vast.ai/instances/ and click "Open" to access the Jupyter server on your instance
8. Upload this notebook to the server through the Jupyter interface

## Setup and Installation

First we will install and import our required packages. The cell below installs:

- Transformers: For loading and saving the model in Hugging Face format
- PyTorch: The deep learning framework that powers our model
- Hugging Face Hub: For uploading our compressed model
- LLM-Compressor: The toolkit we'll use for quantization

```bash
# Install required packages
# Using specific versions to ensure compatibility
pip install transformers==4.51.3  # Recent Transformers version with good Qwen3 support
pip install torch==2.7.0          # PyTorch version compatible with current CUDA drivers
pip install huggingface_hub       # Core Hugging Face Hub library
pip install huggingface_hub[hf_xet]  # Extension for handling large file transfers
pip install huggingface_hub[cli]     # Command-line interface for HF Hub
pip install llmcompressor            # The model compression toolkit
```

Next, import the necessary libraries:

```python
# Import necessary libraries
import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from llmcompressor.modifiers.quantization import GPTQModifier  # For weight quantization
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier  # For activation quantization preprocessing
from llmcompressor import oneshot  # Simplified API for end-to-end compression
```

## Define Model and Output Path

Next, we'll define our model variables and prepare for downloading the `Qwen/Qwen3-8B` model.

- `MODEL_ID`: The Hugging Face model ID for the original model
- `OUTPUT_MODEL_ID`: A name for our compressed model version
- `OUTPUT_DIR`: Local directory where the compressed model will be saved

```python
# Define model path and output directory
MODEL_ID = "Qwen/Qwen3-8B"  # Original model on Hugging Face
OUTPUT_MODEL_ID = "Qwen3-8B-W8A8"  # Our compressed model name
OUTPUT_DIR = "./Qwen3-8B-W8A8"  # Local directory for saving
os.makedirs(OUTPUT_DIR, exist_ok=True)  # Create output directory if it doesn't exist
```

Download the model files using the snapshot_download function:

```python
from huggingface_hub import snapshot_download

# Download the model files to local directory
# This is a more reliable way to download large models compared to the standard AutoModel approach
# This will download all model files (~16GB for Qwen3-8B) and may take some time depending on connection speed
snapshot_download(
    repo_id="Qwen/Qwen3-8B",
    local_dir_use_symlinks=False  # Set to True to use symlinks instead of copying files (saves space but may cause issues)
)
```

## Configure Quantization

Now we'll define our compression recipe, which consists of two steps:

1. **SmoothQuant**: A technique that redistributes the quantization difficulty between weights and activations, making activations easier to quantize with minimal accuracy loss

2. **GPTQ**: A weight quantization method specifically designed for large language models that uses a reconstruction approach to minimize the impact of quantization on model quality

We're using W8A8 quantization, which means:

- W8: Weights stored in 8-bit precision (instead of FP16/FP32)
- A8: Activations computed in 8-bit precision at inference time

This will reduce the model size by approximately 1.7x and potentially increase inference speed.

```python
# Create quantization recipe
# A recipe is a sequence of compression techniques applied in order
recipe = [
    # First apply SmoothQuant to make activations easier to quantize
    # The smoothing_strength parameter controls how aggressively to shift quantization difficulty
    # from activations to weights (higher = more shifting, 0.8 is a good balance)
    SmoothQuantModifier(smoothing_strength=0.8),

    # Then apply GPTQ for weight and activation quantization
    GPTQModifier(
        targets="Linear",  # Apply to all linear layers in the model
        scheme="W8A8",    # 8-bit weights and activations (other options: W4A16, W8A16, etc.)
        ignore=["lm_head"] # Don't quantize the language modeling head (important for output quality)
    )
]
```

## Apply Quantization with oneshot API

Now we'll apply our compression recipe to the model using the oneshot API.

The oneshot API handles the entire compression workflow:

1. Loading the model from Hugging Face or local path
2. Creating a calibration dataset (or using an existing one)
3. Running calibration to determine optimal quantization parameters
4. Applying the compression recipe
5. Saving the compressed model

We're using the `open_platypus` dataset for calibration, which consists of diverse open-source technical questions and answers. This helps ensure the model maintains accuracy across a range of technical topics.

**Note**: This cell will take a significant amount of time (30-60 minutes) and GPU memory to run.

```python
print(f"Starting compression of {MODEL_ID}...")

# Apply quantization using oneshot API
oneshot(
    model=MODEL_ID,               # HF model ID or local path to load the model from
    dataset="open_platypus",      # Calibration dataset - preset dataset with technical Q&A
    recipe=recipe,                # Our compression recipe defined above
    output_dir=OUTPUT_DIR,        # Where to save the compressed model
    max_seq_length=2048,          # Maximum sequence length for calibration samples
    num_calibration_samples=512,  # Number of samples to use for calibration (more is better but slower)
)
```

## Test the Compressed Model

After compression, it's important to verify that the model still performs reasonably well. This cell loads the compressed model and runs a simple inference test.

We'll evaluate the model on a prompt about AI technology to see how well it generates coherent and relevant text. This helps us verify that the quantization didn't significantly degrade model quality.

Note that a proper evaluation would involve more systematic testing across multiple prompts and metrics.

```python
# Load the compressed model from our output directory
tokenizer = AutoTokenizer.from_pretrained(OUTPUT_DIR)
model = AutoModelForCausalLM.from_pretrained(
    OUTPUT_DIR,
    device_map="auto",      # Automatically place model on available devices (CPU/GPU)
    torch_dtype="auto"      # Use the model's native precision (will be INT8 for quantized weights)
)

# Simple test to verify the model still works
prompt = "AI technology is transforming industries by"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate text with some randomness for creativity
with torch.no_grad():  # Disable gradient calculation for inference
    outputs = model.generate(
        inputs["input_ids"],
        max_new_tokens=100,     # Generate up to 100 new tokens
        do_sample=True,         # Use sampling instead of greedy decoding
        temperature=0.7         # Control randomness (lower = more deterministic)
    )

# Decode the generated text and print it
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Prompt: {prompt}")
print(f"Generated text: {generated_text}")
```

## Upload the Model to Hugging Face

Now that we have successfully compressed the model, we'll upload it to Hugging Face Hub for easy sharing and deployment.

First, we'll set up the necessary credentials and repository information:

```python
from huggingface_hub import HfApi, login
import os

# Define variables for Hugging Face upload
# You'll need to replace these with your actual Hugging Face information
HF_USERNAME = "HF_USERNAME" # Add your Hugging Face username here
HF_TOKEN = "HF_TOKEN"       # Replace with your actual Hugging Face token (from settings page)
HF_MODEL_ID = f"{HF_USERNAME}/{OUTPUT_MODEL_ID}"  # Full repository path on Hugging Face
```

Next, we'll create a proper model card (README.md) with information about our compressed model. This helps others understand what the model is, how it was compressed, and how to use it:

```python
# Create a detailed model card describing the compressed model
model_card_content = f"""---
language:
- en
- zh
library_name: transformers
license: other
datasets:
- open_platypus
---

# Qwen3-8B-W8A8

This is a compressed version of [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) using llm-compressor with the following optimizations:
- 8-bit weight quantization using GPTQ
- 8-bit activation quantization
- SmoothQuant pre-processing

## Model Details

- **Original Model**: Qwen/Qwen3-8B
- **Quantization Method**: GPTQ + SmoothQuant (W8A8)
- **Compression Libraries**: [llm-compressor](https://github.com/vllm-project/llm-compressor)
- **Calibration Dataset**: open_platypus (512 samples)
- **Optimized For**: Inference with vLLM
"""

# Write README.md to model directory
readme_path = os.path.join(OUTPUT_DIR, "README.md")
with open(readme_path, "w") as f:
    f.write(model_card_content)
```

Finally, we'll upload the compressed model to Hugging Face Hub. This process may take some time depending on your internet connection speed, as the model files are large (approximately 9.5GB even after compression):

```python
# Login to Hugging Face with your token
login(token=HF_TOKEN)

# Initialize the Hugging Face API
api = HfApi()

# Create the repository (if it doesn't exist)
api.create_repo(
    repo_id=HF_MODEL_ID,
    exist_ok=True,        # Don't error if repo already exists
    private=False,        # Set to True if you want a private repository
)

# Upload the model files
print(f"Starting upload of {OUTPUT_DIR} to {HF_MODEL_ID}...")
api.upload_folder(
    folder_path=OUTPUT_DIR,
    repo_id=HF_MODEL_ID,
    repo_type="model",
    ignore_patterns=["*.tmp", "*.log", "__pycache__"],  # Files to ignore during upload
    commit_message="Upload compressed Qwen3-8B with W8A8 quantization",
)

print(f"Model successfully uploaded to: https://huggingface.co/{HF_MODEL_ID}")
```

## Wrapping Up: From 16GB to 9.5GB Without Breaking the Bank

We've just walked through a transformation that would have seemed impossible a few years ago—taking a sophisticated language model and cutting its resource requirements by approximately 40% while preserving its capabilities.

### What We Accomplished

Our compression workflow delivered results across multiple dimensions:

**Technical achievements:**

- Reduced model footprint from 16GB to 9.5GB using W8A8 quantization
- Maintained output quality through careful calibration with technical datasets
- Created a production-ready model compatible with modern inference engines

**Economic impact:**

- Enabled deployment on more affordable GPUs with less VRAM requirements
- Reduced cloud inference costs through lower resource requirements
- Made advanced language models accessible to teams with limited budgets

**Operational benefits:**

- Faster model loading and reduced memory pressure during inference
- Better resource utilization allowing for higher throughput or multi-model serving
- Simplified deployment pipeline through automated compression workflows

### The Bigger Picture

Model compression offers an alternative approach to AI deployment. Rather than accepting ever-increasing resource requirements, we can actively optimize models for real-world constraints. The combination of LLM-Compressor's sophisticated algorithms and Vast.ai's accessible infrastructure makes this optimization practical for any team.

### Coming Up Next

In Part 2, we'll put our compressed model to the test. We'll deploy both our optimized Qwen3-8B-W8A8 model and the original full-precision version on Vast.ai, then conduct detailed comparisons of their performance, cost, and output quality. You'll see exactly how much you can save without compromising on results—and learn when compression makes the most sense for your specific use case.


Compress LLMs like Qwen3-8B with LLM-Compressor and deploy on Vast.ai. Cut model size ~40% while preserving quality—save cost, speed up inference, and scale smartly.

Model Compression with LLM-Compressor and Deployment on Vast.ai (Part 1)


You wouldn't buy a car without taking it for a test drive, so why deploy a large language model (LLM) without benchmarking it first?

When you're running LLMs at scale, you need more than just theoretical performance data. You're going to want to know how they actually perform under real conditions – throughput, latency, and hardware efficiency all matter. Skipping this step can result in higher costs and wasted resources.

Benchmarking early helps you avoid inefficiencies down the line, and it shouldn't take hours to get started. That's where **vLLM** comes in.

### A Smarter Way to Serve and Benchmark LLMs

vLLM is an open-source library that optimizes the inference and serving of LLMs, and it [simplifies](https://blog.vllm.ai/2023/06/20/vllm.html) the benchmarking process with an efficient architecture built around the attention algorithm PagedAttention. This mechanism reduces memory waste and improves throughput by storing attention data non-contiguously.

When fast and efficient LLM inference is crucial, understanding your model's performance starts with benchmarking. With vLLM, you can spin up a high-performance inference server and start collecting those benchmarks in minutes.

To make things even easier, you don't need to spin up your own infrastructure from scratch. Vast.ai provides an ideal platform to leverage vLLM, offering high-performance GPUs at a fraction of the cost of traditional cloud providers. With Vast, it's easy to find the right hardware for your model, and with simple Docker integration, you can get up and running with vLLM quickly.

This guide demonstrates how to benchmark any LLM using vLLM on Vast.ai, in just a few minutes.

### Getting Started: Benchmarking LLMs with vLLM

This guide walks you through how to **serve a large language model using vLLM** and run a benchmark test to evaluate performance. It’s using a **H100** **GPU server from [Vast.ai](https://vast.ai/)** with the **"PyTorch" template**. In this example, we’ll use the model `meta-llama/Llama-3.1-8B-Instruct`, but you can use **any Hugging Face-supported model** compatible with vLLM.

**1\. Install required Python packages**

First, install the **vLLM** library along with **Pandas** and Hugging Face’s **datasets** library. For example, you can use pip in your Python environment:

```shell
pip install vllm pandas datasets
```

This installs vLLM and the other libraries for data handling.

**2\. Log in to Hugging Face**

To access the Meta Llama 3.1 model, you need a Hugging Face API token. Then run:

```shell
huggingface-cli login
```

This will prompt you to paste your Hugging Face **access token** (from your Hugging Face account settings). After entering the token, the CLI saves it so vLLM and other tools can download the model.

## **3\. Start the vLLM server**

Now start the vLLM server hosting the Llama 3.1 8B Instruct model. vLLM’s OpenAI-compatible server listens on localhost port 8000 by default. Use the `vllm serve` command with **nohup** to run it in the background and redirect output to a log file. For example:

```shell
nohup vllm serve meta-llama/Llama-3.1-8B-Instruct > vllm.log 2>&1 &
```

This command launches the server with the specified model and writes all output (including any Uvicorn logs) to `vllm.log`. The server will run in the background. (The vLLM docs give a similar example: `vllm serve <model-name>`) By default, this serves the model on [`http://localhost:8000`](http://localhost:8000).

## **4\. Test the server with a curl request**

Once the server is running, you can test it by sending a chat completion request to the API. For example, use `curl` to call the chat endpoint (`/v1/chat/completions`) with a simple prompt:

```shell
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
          "model": "meta-llama/Llama-3.1-8B-Instruct",
          "messages": [
              {"role": "system", "content": "You are a helpful assistant."},
              {"role": "user", "content": "What is the capital of France?"}
          ]
        }'
```

This sends a JSON payload to the server. The `model` field names our loaded model, and `messages` contains a chat-style conversation. The server will respond with a JSON completion. (The official vLLM docs show a similar curl example for chat completions)

You should see a JSON response in the terminal with the model’s answer.

## **5\. Clone the vLLM repository**

To access the benchmark script, clone the vLLM source repository:

```shell
git clone https://github.com/vllm-project/vllm.git
cd vllm/benchmarks
```

This pulls the latest vLLM code. Now you have the `benchmark_serving.py` script and related files in the `benchmarks` folder.

## **6\. Run the benchmark script**

Finally, run the built-in benchmark script to measure serving performance. The `benchmark_serving.py` tool sends many requests to the running server and reports throughput/latency. For example, you might run:

```shell
  python3 vllm/benchmarks/benchmark_serving.py \
    --backend vllm \
    --base-url http://127.0.0.1:8000 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --tokenizer meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name random \
    --random-input-len 128 \
    --random-output-len 128 \
    --num-prompts 80 \
    --max-concurrency 4 \
    --temperature 0.7

```

-   `--backend vllm`: Use the vLLM backend to send benchmark requests.

-   `--base-url http://127.0.0.1:8000`: Address of your running vLLM server.

-   `--model meta-llama/Llama-3.1-8B-Instruct`: Name of the model being benchmarked.

-   `--tokenizer meta-llama/Llama-3.1-8B-Instruct`: Tokenizer used for encoding/decoding; usually the same as the model.

-   `--dataset-name random`: Use synthetic/randomly generated prompts instead of a real dataset.

-   `--random-input-len 128`: Length (in tokens) of each generated input prompt.

-   `--random-output-len 128`: Desired length (in tokens) of the model's output.

-   `--num-prompts 80`: Total number of prompts to send during the test.

-   `--max-concurrency 4`: Number of prompts sent in parallel (simulates batch size).

-   `--temperature 0.7`: Controls randomness in generation; higher \= more creative.

This example command (modeled on published vLLM benchmarks) will run 80 chat requests against your server with concurrency 4\. After it finishes, the script prints stats like throughput and latency.

By following these steps and citing vLLM’s documentation and examples, you can serve `Llama-3.1-8B-Instruct` with vLLM and evaluate its performance. Adjust parameters (batch size, number of tokens, etc.) in the commands above as needed for your environment.

### Conclusion

And that's it\! A fast, easy-to-use LLM benchmarking workflow powered by vLLM. In just ten minutes, you can gather meaningful performance data that will help you make smarter decisions about which models (and hardware) are right for your specific project and workload.

Need powerful GPUs to run your tests? Vast.ai offers on-demand access to H100s, RTX 5090s, and other high-performance options at a fraction of the typical price – so you can save 5-6X on GPU compute. Get started today and benchmark on your terms\!


Benchmark LLMs in minutes using vLLM on Vast.ai. Serve models like Llama 3.1-8B, test performance, and optimize cost with fast, efficient, real-world insights.

How to Benchmark An LLM with vLLM in 10 Minutes




Are you looking to run powerful open-source LLMs like **Llama 4**, **Kimi K2**, or **Qwen3** without the hassle of managing complex infrastructure? Vast.ai makes it easy to train and deploy these models with a curated set of templates built for speed, flexibility, and scale. 

Our [templates](https://docs.vast.ai/templates) cover everything from browser-based interfaces like Oobabooga and Open WebUI to optimized backends like HuggingFace TGI and vLLM. 

Below, we break down the top recommended templates on **Vast.ai** and how you can use them to get your LLM project off the ground quickly and easily. 

### 1\. [Oobabooga (LLM WebUI)](https://docs.vast.ai/oobabooga-llm-webui)

[Oobabooga](https://github.com/oobabooga/text-generation-webui) provides a user-friendly web interface for interacting with open-source LLMs. Its UI  resembles the original ChatGPT style – ideal for those who prefer a graphical interface over command-line operations. 

With support for **multiple text generation backends in one UI/API**, you can switch between different models easily without restarting and maintain fine control over settings. Models like **Falcon**, **Llama**, and **Vicuna** can be loaded and explored with just a few clicks.

Our [Oobabooga template](https://docs.vast.ai/oobabooga-llm-webui) facilitates quick deployment, making it suitable for both beginners and experienced users.

### 2\. [HuggingFace TGI with Llama 3](https://docs.vast.ai/huggingface-tgi-with-llama3)

Ideal for serving **Llama 3**, this template is optimized for **high-performance text generation tasks** using HuggingFace's [Text Generation Inference (TGI) server](https://github.com/huggingface/text-generation-inference). It supports other popular open-source LLMs from HuggingFace, including **Falcon**, **StarCoder**, **BLOOM**, and **GPT-NeoX**, and it's particularly beneficial for applications requiring low-latency responses and scalability, such as chatbots or content generation tools.

For this [template](https://docs.vast.ai/huggingface-tgi-with-llama3), you'll need your own HuggingFace access token – and you'll also need to apply for permission to use Llama 3 on HuggingFace, since the model is hosted in a gated repository.

### 3\. [Open WebUI (Ollama)](https://docs.vast.ai/ollama-webui)

[Open WebUI](https://github.com/open-webui/open-webui) is an extensible, user-friendly platform for self-hosted AI deployments. It operates entirely offline and supports a range of open-source LLM backends like **Llama 4**, **Kimi K2**, **DeepSeek R1**, and **OpenAI-compatible APIs**. 

With **built-in support for retrieval-augmented generation** (RAG), Open WebUI seamlessly integrates document interactions and web search into the chat experience. You can even incorporate image generation capabilities as well as engage with multiple LLMs simultaneously in parallel.

Our [Ollama \+ WebUI template](https://docs.vast.ai/ollama-webui) will automatically set up Open WebUI as a web-based interface and expose a port for the Ollama API, making it easy to run and interact with LLMs directly from your instance.

### 4\. [vLLM](https://docs.vast.ai/vllm-llm-inference-and-serving)

Optimized for [serving open-source LLMs](https://vast.ai/article/serving-online-inference-with-vllm-api-on-vast) with high-throughput inference, [vLLM](https://github.com/vllm-project/vllm) uses a novel architecture that drastically reduces memory overhead, making it easier to serve large models like **Llama**, **Mistral**, and **Falcon** efficiently. Notably, it supports continuous batching for faster and more efficient multi-user inference at scale.

vLLM provides an OpenAI-compatible API server for easy integration into existing workflows, and it's particularly well suited for developers building **commercial or research applications that require fast and stable model responses.** The vLLM framework seamlessly supports most open-source models on HuggingFace, including the ones named above as well as **Mixtral**, **DeepSeek**, **LLaVA**, **BLOOM**, **GPT-NeoX**, **Qwen**, and more.

Our [vLLM template](https://docs.vast.ai/vllm-llm-inference-and-serving) contains everything needed for you to get started – all you have to do is specify the model you want to serve and the corresponding vLLM configuration.

### From Setup to Serving in Minutes

With Vast.ai's market-based cloud GPU rental platform, you can avoid the usual infrastructure and budget roadblocks as you spin up high-performance instances tailored to your workloads. 

Our [library](https://cloud.vast.ai/templates) of **ready-to-use templates** – covering everything from web-based interfaces to optimized inference engines – make it easy to train and deploy LLMs at scale. We take care of the heavy lifting, so you can spend less time configuring and more time building.

Why wait? **Get started with open-source LLMs on Vast.ai today\!**

Run open-source LLMs like Llama 4, Qwen3, and Kimi K2 with ease. Vast.ai templates let you deploy fast using Oobabooga, vLLM, HuggingFace TGI, or Open WebUI.

Open-Source LLMs You Can Train and Deploy on Vast.ai Right Now


The DeepSeek R1-0528 release introduces a significant improvement to the reasoning capabilities of language models by making it easier to access "thinking" mode without requiring complex prompt engineering or pre-pending thinking tokens. This enhanced model provides transparent, step-by-step reasoning that's particularly valuable for educational applications, complex problem solving, and scenarios where transparency in AI decision-making is crucial.

In this post, we'll explore how to deploy the **DeepSeek-R1-0528-Qwen3-8B** model using **vLLM** on Vast.ai's cloud GPU platform, leveraging the new **qwen3 reasoning parser** that simplifies access to the model's internal thinking process.

---

## The Open Source AI Revolution

DeepSeek R1-0528 represents more than just a technical advancement—it's a significant step toward democratizing advanced AI capabilities. As an **open source model**, it makes these reasoning and advanced AI capabilities more accessible.

Unlike closed-source models that require vendor dependence, usage costs, and data privacy concerns, DeepSeek R1-0528 enables **complete control** over your AI infrastructure. Organizations can now deploy advanced reasoning capabilities locally, maintaining data sovereignty while avoiding variable per-token pricing.

This democratization means organizations can access cutting-edge AI without massive infrastructure investments. Advanced reasoning capabilities are no longer exclusive to large tech companies—anyone with GPU access can deploy their own reasoning-capable AI system.

---

## What Makes DeepSeek R1-0528 Special?

The DeepSeek R1-0528 release addresses a key limitation in previous DeepSeek models: the complexity of accessing the model's internal reasoning process. Previously, users needed to manually prepend "thinking" tokens to model outputs or use complex prompt engineering to see step-by-step reasoning.

---

## Deploying DeepSeek R1-0528 on Vast.ai

### Step 1: Set Up Vast.ai Environment

First, install the Vast CLI and configure your API key:

```bash
pip install --upgrade vastai

export VAST_API_KEY="your_vast_api_key"
vastai set api-key $VAST_API_KEY
```

### Step 2: Find Suitable Hardware

The DeepSeek-R1-0528-Qwen3-8B model requires specific hardware capabilities:

-   **Minimum 24GB GPU RAM** for model weights and KV cache
-   **Single GPU configuration** (sufficient for the 8B parameter model)
-   **Static IP address** for stable API endpoint hosting
-   **Direct port access** for the vLLM OpenAI-compatible server
-   **60GB+ disk space** for model storage and dependencies

Search for suitable instances:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 24 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 60 \
rentable = true"
```

### Step 3: Deploy the Model

Deploy using the vLLM OpenAI-compatible server with the qwen3 reasoning parser:

```bash
export INSTANCE_ID="your_instance_id"

vastai create instance $INSTANCE_ID \
--image vllm/vllm-openai:latest \
--env '-p 8000:8000' \
--disk 60 \
--args --model deepseek-ai/DeepSeek-R1-0528-Qwen3-8B \
--max-model-len 4096 \
--reasoning-parser qwen3
```

**Key deployment parameters:**

-   `--reasoning-parser qwen3`: Enables the simplified reasoning parser
-   `--max-model-len 4096`: Accommodates longer reasoning sequences
-   `--tensor-parallel-size 1`: Single GPU configuration
-   `--gpu-memory-utilization 0.90`: Optimized memory usage

### Step 4: Get Instance Details and Test Connection

Navigate to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console and locate your instance. Click the IP address button to view the forwarded ports.

Test the connection with a simple curl command:

```bash
export VAST_IP_ADDRESS="your_ip_address"
export VAST_PORT="your_port"

curl -X POST http://$VAST_IP_ADDRESS:$VAST_PORT/v1/completions \
-H "Content-Type: application/json" \
-d '{"model": "deepseek-ai/DeepSeek-R1-0528-Qwen3-8B", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

---

## Testing the Model's Reasoning Capabilities

### Setup Python Client

```python
import time
from openai import OpenAI

# Replace with your actual Vast.ai instance details
VAST_IP_ADDRESS = "your_ip_address"
VAST_PORT = "your_port"

client = OpenAI(
    api_key="DUMMY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)
```

### Basic Functionality Test

```python
start = time.time()
test_resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{"role": "user", "content": "Hello"}]
)
elapsed = time.time() - start

print("Response:", test_resp.choices[0].message.content)
print(f"Elapsed time: {elapsed:.4f} s")
```

### Reasoning Capability Test

This is where DeepSeek R1-0528 truly shines. The model naturally provides step-by-step reasoning without complex prompt engineering:

```python
reasoning_prompt = """
Write the History of Paris in a paragraph, show your thinking
step by step first before paragraph and then write the history
in a paragraph
"""

start = time.time()
resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{"role": "user", "content": reasoning_prompt}]
)
elapsed = time.time() - start

print("Model output:\n", resp.choices[0].message.content)
print(f"Elapsed time: {elapsed:.3f} s")
```

**Expected output structure:**

1. **Thinking process**: The model shows its step-by-step reasoning
2. **Final answer**: A well-structured paragraph about Paris's history

### Mathematical Reasoning Test

```python
math_prompt = "If 8 × 7 = ?, show your work step by step."

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{"role": "user", "content": math_prompt}]
)

print("Mathematical reasoning:\n", resp.choices[0].message.content)
```

The model will naturally break down the problem, show intermediate steps, and arrive at the solution with clear reasoning traces.

---

## Key Benefits of DeepSeek R1-0528

### 1. **Simplified Reasoning Access**

No need to prepend thinking tokens or use complex prompt engineering. The `qwen3` reasoning parser automatically handles the extraction and presentation of reasoning steps.

### 2. **Educational Value**

Perfect for educational applications where students need to understand the problem-solving process, not just the final answer.

### 3. **Transparency and Trust**

Users can see exactly how the model arrives at its conclusions, building trust in AI-powered decision-making systems.

### 4. **Cost-Effective Deployment**

Running on Vast.ai provides significant cost savings compared to major cloud providers while maintaining high performance.

### 5. **Easy Integration**

OpenAI-compatible API ensures seamless integration with existing applications and workflows.

---

## Use Cases

### Educational Applications

-   Step-by-step problem solving for mathematics, science, and logic
-   Teaching critical thinking and reasoning skills
-   Creating interactive learning experiences

### Business Intelligence

-   Transparent decision-making in automated systems
-   Audit trails for AI-powered recommendations
-   Complex analysis with clear reasoning paths

### Research and Development

-   Understanding model behavior and capabilities
-   Testing reasoning patterns across different domains
-   Developing more interpretable AI systems

---

## Conclusion

The DeepSeek R1-0528 release represents a significant step forward in making AI reasoning more accessible and transparent. By combining this powerful model with Vast.ai's cost-effective GPU infrastructure and the simplified qwen3 reasoning parser, developers can now easily deploy transparent, reasoning-capable AI systems.

Whether you're building educational tools, business intelligence systems, or research applications, DeepSeek R1-0528 on Vast.ai provides an excellent foundation for applications that require clear, step-by-step reasoning capabilities.

The elimination of complex prompt engineering for accessing reasoning modes makes this model particularly valuable for production applications where transparency and interpretability are key requirements.

---

## Useful Resources

-   [DeepSeek R1-0528 Model on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B)
-   [vLLM Documentation](https://docs.vllm.ai/)
-   [Vast.ai GPU Marketplace](https://vast.ai)
-   [OpenAI Python SDK](https://github.com/openai/openai-python)

---

_Ready to deploy transparent, reasoning-capable AI? Get started with DeepSeek R1-0528 on Vast.ai today! 🚀_


Deploy DeepSeek R1-0528 on Vast.ai to access transparent, step-by-step AI reasoning without prompt tricks—ideal for education, research, and business use.

DeepSeek R1-0528: Enhanced Reasoning with Simplified Thinking Mode on Vast.ai


We're back with a new update from the team at **Vast.ai**\! This month, we've rolled out fresh hardware options, new video-generation tools, and continued improvements behind the scenes.

The big news is that our GPU lineup now includes the **NVIDIA DGX B200** as well as the **RTX PRO 6000 WS**\! These powerful machines open the door to even more demanding workloads, bringing you the compute power required for everything from multi-modal inference to precision-driven simulations.

We also recently reached an exciting milestone: we now have **over one thousand RTX 5090 GPUs available on Vast** – 1,180 and counting.

Here are the details about what's new this month\!

### NVIDIA Cloud GPU Updates

Whether you're training foundation models, rendering high-resolution video, or running inference, our latest hardware additions give you extra flexibility:

-   **NVIDIA DGX B200** – A universal AI supercomputer built on the Blackwell architecture, equipped with **eight GPUs** and **1.4 TB** of GPU memory. With massive bandwidth and compute capacity, it's well suited to handle virtually any enterprise AI workload.

-   **RTX PRO 6000 WS** – A high-end Blackwell-based workstation GPU featuring **96 GB** of GDDR7 memory and up to **4000 TOPS** of AI compute. It can easily handle massive datasets and multi-billion-parameter models while supporting seamless multi-app workflows.

### New Guide and Templates

We've focused on video generation this month, adding a couple of new templates and a guide to help you go from idea to clip as seamlessly as possible:

-   [**Guide to Video Generation**](https://docs.vast.ai/video-generation) – A walkthrough for setting up and using ComfyUI on Vast.ai, so you can create advanced stable diffusion pipelines (ideal for video generation workflows) with this powerful node-based interface.

-   **LTX Video (ComfyUI)** – A ready-to-use template that pairs the LTX Video model with ComfyUI, optimized for consumer-grade GPUs.

-   **Open-Sora** – A template for this open-source video generation model with a streamlined, user-friendly platform that lowers the barrier to efficient, high-quality video content creation.

And that's a wrap for June 2025\!

### Our Commitment

Whether you're looking to launch a B200 instance, experiment with video generation using our latest templates, or scale up with our expanding lineup of RTX 5090 GPUs, we're here to support your work at every stage. Our goal at **Vast.ai** is to make powerful compute accessible, flexible, and easy to use – so you can focus on building, training, and creating.

Need help? Reach out to our customer support team anytime at [support@vast.ai](mailto:support@vast.ai), or connect with the community on our Discord server for tips, troubleshooting, and more.

### Change Log

#### New Templates & Guides:

-   [Guide to Video Generation](https://docs.vast.ai/video-generation)
-   LTX Video (Comfy UI) – Video-Gen model optimized for consumer-grade GPUs.
-   Open-Sora – Democratizing efficient video production for everyone.

#### New GPUs:

-   DGX B200
-   RTX PRO 6000 WS

Plus we now have over one thousand 5090 GPUs available.


New DGX B200 and RTX PRO 6000 WS GPUs, over 1000 RTX 5090s available, and new video generation templates.

June 2025 Product Update


Ten years ago today, our CEO and founder, Jacob Cannell, published ["The Brain as a Universal Learning Machine"](https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine) on LessWrong.com.

Drawing from neuroscience, machine learning, and systems theory, he proposed that **the human brain is not a patchwork of specialized, genetically programmed modules, but a general-purpose learning system** – or, as he put it, "a biological implementation of a _Universal Learning Machine_."

In hindsight, the timing was notable. Cannell published the piece just as deep learning was moving from niche research to broader relevance in artificial intelligence (AI). In the decade since then, rapid progress in scalable, trainable, data-driven models has validated the idea that learning, not pre-engineered structure, is the key to building intelligent systems.

So on the tenth anniversary of its publication, we'd like to revisit the key ideas of the article, reflect on what's aged well, and look ahead at what's still unfolding.

We’d encourage you to read the full article on [Less Wrong](https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine).


A look at the brain as a universal learning machine, and how it can be used to learn about the world.

A Decade Later: Revisiting the Brain as a Universal Learning Machine


Serving multiple machine learning models simultaneously is a key challenge in production AI systems. The increasing size of foundational models and constraints on GPU memory often mean that deploying separate models for each task results in high RAM usage, underutilized hardware, and latency problems caused by repeatedly swapping entire models in and out of GPU memory. This not only increases cost but also complicates scaling.

However, by leveraging Lorax, a framework for dynamically loading LoRA adapters—and Vast.ai's flexible GPU marketplace, developers can efficiently serve several specialized models on a single base model deployment. This results in significantly better hardware utilization, reduced infrastructure costs, and low-latency inference for diverse AI workloads, making this a game-changer for enterprises deploying their own AI infra.

In this blog post, we'll explore how to deploy and run multiple LoRA adapters on a shared base model using Lorax on Vast.ai's cloud platform. This setup enables hosting thousands of fine-tuned models simultaneously, loading task-specific adapters on demand, and seamlessly switching between tasks such as math problem solving and customer support classification with minimal overhead.

---

## What Are LoRA Adapters and Lorax?

- **LoRAs (Low-Rank Adaptation):** Lightweight, task-specific parameter adjustments that adapt a large base model to new tasks with far fewer parameters than full fine-tuning. LoRAs can be swapped in and out without needing to reload the entire large model, saving significant RAM and time.

- **Lorax (LoRA eXchange):** An efficient serving framework that enables running thousands of LoRA adapters on a single GPU by dynamically loading these adapters into a base model at inference time. Lorax maintains high throughput and low latency while drastically reducing the deployment cost of hosting multiple fine-tuned variants.

---

## Deploying Multiple Models Using Lorax and Vast.ai

The following example uses the `mistralai/Mistral-7B-v0.1` base model hosted on Vast.ai, along with two LoRA adapters:

- `predibase/gsm8k`: Designed for solving math problems like the GSM8K benchmark.
- `predibase/customer_support`: Specializes in customer service query classification.

We deploy the base model once and dynamically switch between these specialized LoRA adapters, enabling efficient multi-model serving on the same underlying GPU instance.

---

### Step 1: Set Up Vast.ai Environment

1. **Install Vast CLI and configure API key:**

```bash
pip install vastai==0.2.6

export VAST_API_KEY="your_vast_api_key"
vastai set api-key $VAST_API_KEY
```

2. **Search for a suitable GPU instance:**

The `mistralai/Mistral-7B-v0.1` model requires at least 16GB VRAM, but it's safer to select a larger instance (e.g., 32GB VRAM) for LoRA adapter loading.

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 32 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

3. **Deploy the instance:**

Make sure you have accepted the usage terms for `Mistral-7B-v0.1` on Huggingface before proceeding.

```bash
export INSTANCE_ID=your_instance_id

vastai create instance $INSTANCE_ID --image ghcr.io/predibase/lorax:main \
--env '-p 8080:80 --shm-size 1g -e HUGGING_FACE_HUB_TOKEN="your_hf_token"' \
--disk 80 --args --model-id mistralai/Mistral-7B-v0.1
```

This command runs the Lorax server that hosts the base model and dynamically loads adapters on demand.

4. **Get Instance IP Address and Port**

Now, we need to get our IP address and port to call our model. We first need to wait for our machine to download the image and the model and start serving. This will take a few minutes. The logs will show you when it's done.

Next, go to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console and find the instance you just created.

At the top of the instance, there is a button with an IP address in it. Click this and a panel will show up of the ip address and the forwarded ports.
You should see something like:

```
Open Ports
XX.XX.XXX.XX:YYYY -> 8000/tcp
```

You will need the IP address (XX.XX.XXX.XX) and the port (YYYY) for the next step.

---

### Step 2: Query LoRA Adapters via HTTP Requests

Lorax supports on-demand loading of LoRAs from the Huggingface Hub, minimizing memory overhead versus deploying multiple large models simultaneously.

First, configure environment variables to connect to your Vast instance and set your Huggingface token:

```python
HF_TOKEN = "your_hf_token"
VAST_IP_ADDRESS = "your_instance_ip"
VAST_PORT = "your_vast_port"
```

#### Example: Math Problem Solving with `predibase/gsm8k`

Prepare a prompt:

```python
question = "James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week?"
math_prompt = f"Please answer the following question: {question}\nAnswer"
```

Make a POST request to the `/generate` endpoint specifying the LoRA adapter ID:

```python
import requests

headers = {
    "Content-Type": "application/json",
    "Authorization": f"Bearer {HF_TOKEN}"
}
data = {
    "inputs": math_prompt,
    "parameters": {
        "max_new_tokens": 100,
        "adapter_id": "predibase/gsm8k",
        "adapter_source": "hub"
    }
}

url = f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/generate"
response = requests.post(url, headers=headers, json=data)
print("Response:", response.json()["generated_text"])
```

**Response:**

```
He runs 3*3=<<3*3=9>>9 sprints a week
So he runs 9*60=<<9*60=540>>540 meters a week
#### 540
```

#### Example: Customer Support Classification with `predibase/customer_support`

Prepare a customer support transcript prompt:

```python
transcript = "Hi I am having trouble with my account. It says I need to reset my password to log in but I already reset my password."

customer_support_prompt = f"""Consider the case of a customer contacting the support center.
The term task type refers to the reason for why the customer contacted support.

### The possible task types are: account issue, billing issue, product issue, none of the above

Summarize the issue/question/reason that drove the customer to contact support:

Transcript: {transcript}

Task Type:
"""
```

Make the API call:

```python
data = {
    "inputs": customer_support_prompt,
    "parameters": {
        "max_new_tokens": 100,
        "adapter_id": "predibase/customer_support",
        "adapter_source": "hub"
    }
}

response = requests.post(url, headers=headers, json=data)
print("Response:", response.json()["generated_text"])
```

**Response:**

```
account issue
```

---

### Step 3: Access Models Using the OpenAI SDK

Lorax provides compatibility with the OpenAI API, allowing easy integration into existing apps with minimal code changes.

1. **Install OpenAI SDK**

```bash
pip install --upgrade openai
```

2. **Call the math adapter:**

```python
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1",
    default_headers={"Authorization": f"Bearer {HF_TOKEN}"}
)

resp = client.completions.create(
    model="predibase/gsm8k",
    prompt=math_prompt,
    max_tokens=100,
    temperature=0.7
)

print("Response:", resp.choices[0].text)
```

Output:

```
James runs 3 x 3 = <<3*3=9>>9 sprints in a week.
He runs 9 x 60 = <<9*60=540>>540 meters in a week.
#### 540
```

3. **Call the customer support adapter:**

```python
resp = client.completions.create(
    model="predibase/customer_support",
    prompt=customer_support_prompt,
    max_tokens=100,
    temperature=0.7
)
print("Response:", resp.choices[0].text)
```

Output:

```
account issue
```

---

## Why This Approach Matters

Serving multiple specialized AI models traditionally involves deploying each fine-tuned model independently, consuming lots of GPU memory and resulting in higher cost and slower inference when switching tasks.

By contrast, Lorax’s approach uses a single shared base model and dynamically loads lightweight LoRA adapters tailored for different tasks. This yields several benefits:

- **Reduced RAM and GPU usage:** Only the small adapter layers are loaded on demand, not the entire large model.
- **Lower infrastructure cost:** A single GPU instance can serve thousands of specialized models.
- **Faster context switching:** Switching between tasks is near-instant as you don’t reload the full model.
- **Flexibility:** New adapters can be added or updated independently.
- **Ease of integration:** OpenAI-compatible APIs simplify adoption.

Combined with Vast.ai’s cost-effective, flexible GPU rental options, this method provides a scalable, production-ready solution for deploying multi-model AI services with a minimal footprint.

---

## Conclusion

Efficiently serving multiple machine learning models is vital for scaling AI-powered products. The combination of Lorax’s dynamic LoRA adapter serving and Vast.ai’s flexible GPU infrastructure offers a powerful, cost-effective, and scalable way to support many specialized models on a single deployment.

Whether solving math problems, classifying customer inquiries, or other tasks, this approach reduces overhead and unlocks practical multi-model serving for modern AI applications.

Try setting up your own multi-LoRA serving environment today to experience the benefits firsthand!

---

## Useful Resources

- [Lorax GitHub Repository](https://github.com/predibase/lorax)
- [Vast.ai GPU Marketplace](https://vast.ai)
- [Mistral-7B-v0.1 on Huggingface](https://huggingface.co/mistralai/Mistral-7B-v0.1)
- [OpenAI Python SDK](https://github.com/openai/openai-python)

---

_Happy modeling! 🚀_


Serve multiple ML models efficiently using Lorax and LoRA adapters on Vast.ai—cut costs, reduce latency, and maximize GPU use with dynamic adapter loading.

Efficiently Serving Multiple Machine Learning Models with Lorax and vLLM on Vast.ai


Nearly every startup faces the same two pressures: to move fast and make every dollar count.

Time and budget may be limited while demands remain high – especially for teams working with AI, machine learning, or other compute-heavy workloads. For startups, the pressure to deliver quickly and prove technical viability makes access to reliable infrastructure even more critical.

Traditional cloud providers often aren't built with startups in mind. Pricing may be steep or overly complex, resources can be constrained, and it's easy to end up locked into systems that don't scale with your needs. On the other hand, buying your own hardware adds upfront costs and delays, which can be difficult to justify for early-stage teams.

Vast.ai offers another path. **Our market-based cloud GPU rental platform gives startups the flexibility to run powerful workloads without the usual friction.**

Here's a closer look at why startups choose to build with Vast.ai.

### Why Vast.ai Makes Sense for Startups

Startups need infrastructure that keeps up with their pace, budget, and technical demands. The following are some of the reasons Vast.ai hits the mark.

#### 1\. Instant Access to GPUs

With Vast.ai, there are no waitlists or delays. Startups can instantly access powerful GPUs like **H100s**, **A100s**, and **RTX 5090s**, backed by a global network of providers that ensures better availability and built-in redundancy.

Plus, with machines available around the world, startups can deploy GPUs in regions closer to their major user concentrations, reducing network latency for performance-sensitive applications.

#### 2\. Cost-Effective Compute Power

Vast.ai offers significantly cheaper GPU rental compared to traditional cloud providers – **on average 5-6X lower in cost** – making high-performance compute far more accessible.

For bootstrapped startups and solo founders, Vast.ai levels the playing field, enabling smaller teams on a budget to compete in compute-intensive spaces, as well as handle temporary spikes in workload without having to invest in expensive infrastructure.

#### 3\. Flexible Startup-Friendly Billing

Vast.ai's **pay-as-you-go pricing** gives startups the freedom to scale compute resources up or down as needed, with no lock-in contracts. This way, **you only pay for what you use**, and you can quickly adapt to new demands as they arise.

On-demand and interruptible (bid auction) [rental options](https://docs.vast.ai/instances/rental-types) offer even more control – ideal for variable usage patterns and budget-conscious teams. On top of that, real-time cost tracking and detailed resource utilization dashboards help you monitor and optimize performance.

#### 4\. Enterprise-Grade Security

With Vast.ai, **startups can choose the level of trust and security that best fits their needs.** Providers [range](https://vast.ai/compliance) from verified hosts, suitable for general computing needs, to our vetted datacenter partners who maintain third-party compliance certifications (at minimum ISO 27001 certified) and sign comprehensive Data Processing Agreements (DPAs) with Vast.ai.

For sensitive or production workloads, our Secure Cloud tier ensures you’re connecting with partners in certified locations that are current with ISO 27001 and/or Tier 3/4 standards. (You can even filter for it. Look for the blue datacenter label\!)

We've also [completed](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) our SOC 2 Type I certification – part of our ongoing commitment to the highest standards of regulatory compliance.

#### 5\. Developer Control

Vast.ai provides **transparent instance specs** (GPU type, RAM, storage, and more), so startups can choose exactly what fits their workload and budget. Developers can bring their own Docker images, run their own stack, and configure environments to match their specific requirements.

With a low barrier to entry, even early-stage teams without dedicated DevOps support can get up and running quickly. Our selection of pre-configured [templates](https://docs.vast.ai/templates) are like recipes that help users easily set up instances; templates can define the Docker image, environment variables, and launch mode for each new instance.

#### 6\. Ideal for AI Workloads

Vast.ai is well suited for AI startups working on large-scale training, fine-tuning, and inference. Our customizable, on-demand [CPU/GPU clusters](https://vast.ai/clusters) ensure reliable throughput and flexible resource allocations for continuous or large-scale workloads.

With support for large language models (LLMs) and diffusion models, our platform is compatible with all major machine learning (ML) frameworks and offers the flexibility to run custom containers and tooling.

For teams that want to get started quickly, Vast.ai provides **ready-to-use GPU instances pre-configured for frameworks like TensorFlow and PyTorch**, reducing setup time and simplifying deployment.

#### 7\. Built to Experiment

With Vast.ai, **startups can prototype, test, and explore without any long-term commitments** or expensive upfront investment.

For instance, teams can run quick tests and train models on demand, access and evaluate cutting-edge GPUs like the H200 that may otherwise be out of reach, and even spin up high-performance environments on short notice for hackathons and demos.

### The Bottom Line

**Great ideas can come from anywhere, and they deserve the resources to grow.** By making GPU compute power more open and accessible, Vast.ai gives startups and early-stage teams a real shot at achieving their goals.

Built with transparency, accessibility, and security in mind – and aligned with the open-source ethos that many founders value – our platform supports the kind of flexibility and control that your team needs from day one.

Ready to build, test, and scale on your terms? Discover what's possible with Vast.ai\!


Vast.ai offers startups instant, low-cost access to powerful GPUs—perfect for AI, ML, and compute-heavy work—without lock-ins or upfront infrastructure costs.

7 Reasons Why Startups Use Vast.ai


The GeForce RTX 4090 and the A100 both sit at the high end of NVIDIA's GPU lineup, but they're built for very different worlds.

The **RTX 4090** is a powerful consumer-grade GPU, designed for ultra high-performance gaming, professional creative work, and even some entry-level AI workloads. It's surprisingly capable across a variety of tasks, especially for its price point.

The **A100**, on the other hand, isn't aimed at consumers at all. It's an enterprise-grade GPU built for data centers, research labs, and AI teams running large-scale model training, inference, and simulation workloads – designed to move serious data at serious speed.

In this post, we'll look at how these two machines compare with each other, and why you might choose one over the other based on your specific needs.

### NVIDIA GeForce RTX 4090: A Workhorse with Range

When it launched in September 2022 as the top-tier consumer GPU of its generation, the RTX 4090 quickly became the go-to choice for 4K gaming, 3D rendering, and AI-enhanced creative workflows.

Built on NVIDIA's **Ada Lovelace** architecture and powered by the **AD102 chip**, the RTX 4090 introduced significant upgrades in ray tracing and Tensor Core acceleration, and delivered notable performance gains with **DLSS 3** in a variety of games.

While it's no longer the flagship GPU of NVIDIA's GeForce line (that title now goes to the RTX 5090, which you can read about [here](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know)), the RTX 4090 definitely still holds its own. It remains one of the most capable and well-rounded GPUs available at its price point for consumers and technical teams alike.

#### Some of the RTX 4090's key specs and features include:

-   **24 GB of GDDR6X memory** with **1,008 GB/s bandwidth** on a **384-bit** interface.  

-   **16,384 CUDA cores** for high-parallel throughput.  

-   **128 third-gen RT cores** and **512 fourth-gen Tensor cores** for advanced ray tracing and AI workloads.  

-   **DLSS 3** support for real-time upscaling and performance boosts in many games.  

-   **450 watts** of thermal design power (TDP), requiring an **850-watt power supply** due to its high processing capabilities.

If you're looking for serious performance in a desktop form factor, the RTX 4090 continues to deliver. It may not be the newest card on the market, but it remains a workhorse that gets the job done across a wide range of applications.

As capable as the RTX 4090 is, it's not built for everything. If your workloads go beyond what a high-end consumer GPU can handle, the A100 just might bring the compute power you need.

### NVIDIA A100: A Powerhouse with Purpose

The A100 is built for scale. Based on NVIDIA's Ampere architecture and powered by the **GA100 chip**, it's designed to tackle massive workloads – like foundational AI model training – thanks to its raw throughput, high-bandwidth memory, and ability to scale across multi-GPU environments.

When the A100 launched in June 2021, it introduced **third-gen Tensor Cores**, support for **TF32** precision, and **Multi-Instance GPU** (MIG) capabilities – enabling flexible resource partitioning and efficient parallel processing across shared environments.

While some of its core specs might look modest compared to the RTX 4090, the A100 delivers superior performance in AI and data-intensive workloads because it's optimized where it counts: memory bandwidth, interconnects, and specialized compute.

For instance, the A100's memory clock is much lower than the RTX 4090's on paper (roughly 3 Gbps vs. 21 Gbps). However, the A100 uses **HBM2e memory** with a much wider **5,120-bit interface**. This design allows it to deliver around **2 TB/s** of bandwidth – double the RTX 4090 – despite the lower frequency. It's an approach that prioritizes efficiency and scale.

#### To recap, here are a few notable features of the A100:

-   **80 GB of HBM2e memory** with about **2 TB/s of bandwidth** (twice that of the RTX 4090).  

-   **5,120-bit memory bus**, an exceptionally wide interface enabling fast access to memory, reducing bottlenecks in AI and high-performance computing (HPC) workloads.  

-   **432 third-gen Tensor Cores**, with high-throughput **TF32** performance for AI and robust **FP64** support for HPC workloads.  

-   **Multi-Instance GPU (MIG)** support, so it can be partitioned into up to 7 isolated GPU instances – ideal for multi-tenant workloads and fine-grained resource allocation.  

-   **NVLink and NVSwitch** allow up to 16 A100 GPUs to be interconnected at up to **600 GB/s** of bi-directional bandwidth per GPU.

In short, the A100 delivers serious performance for large-scale AI and HPC workloads. But if you're comparing it head-to-head with the RTX 4090 on specs alone, here's how the two stack up.

### RTX 4090 vs. A100: Key Specs Compared

For reference, this table highlights some of the features and specs of the RTX 4090 and A100:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Specification
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 4090
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                A100
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Ada Lovelace
            </td>
            <td class="border border-gray-300 px-4 py-2">Ampere</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">VRAM</td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                24 GB GDDR6X
            </td>
            <td class="border border-gray-300 px-4 py-2">80 GB HBM2e</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1313 MHz 21 Gbps
            </td>
            <td class="border border-gray-300 px-4 py-2">1512 MHz 3 Gbps</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Bus Width
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">384</td>
            <td class="border border-gray-300 px-4 py-2">5120</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Bandwidth
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1.01 TB/s
            </td>
            <td class="border border-gray-300 px-4 py-2">1.94 TB/s</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Streaming Multiprocessors (SMs)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">128</td>
            <td class="border border-gray-300 px-4 py-2">108</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">512</td>
            <td class="border border-gray-300 px-4 py-2">432</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                CUDA Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">16,384</td>
            <td class="border border-gray-300 px-4 py-2">6,912</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Ray Tracing Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">128</td>
            <td class="border border-gray-300 px-4 py-2">N/A</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Base Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2235 MHz
            </td>
            <td class="border border-gray-300 px-4 py-2">1065 MHz</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Boost Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2520 MHz
            </td>
            <td class="border border-gray-300 px-4 py-2">1410 MHz</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Multi-Instance GPUs
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                No MIG Support
            </td>
            <td class="border border-gray-300 px-4 py-2">
                Up to 7 MIGs @ 10GB each
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Thermal Design Power (TDP)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">450W</td>
            <td class="border border-gray-300 px-4 py-2">300W</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Recommended Power Supply
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">850W</td>
            <td class="border border-gray-300 px-4 py-2">700W</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Launch Date
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Sept. 20, 2022
            </td>
            <td class="border border-gray-300 px-4 py-2">June 28, 2021</td>
        </tr>
    </tbody>
</table>

As mentioned, specs are just one part of the equation. Ultimately, choosing the right GPU depends on your workload, environment, and priorities.

### Use Cases: Which GPU Should You Choose?

Here’s how the strengths of each GPU align with different types of use cases.

#### You should consider the RTX 4090 if you prefer:

-   **A powerful, desktop-friendly GPU** for high-end gaming, creative workflows, and light AI experimentation.
-   **Excellent performance-per-dollar in single-GPU setups**, particularly for developers and small teams.
-   **A widely available, less costly option** for rendering, video editing, or running small to medium AI models locally.

#### Whereas the A100 may be a better fit if you need:

-   **Scalable performance** for training large models, running high-throughput inference, or powering complex simulations.
-   **A data center-optimized GPU** with advanced memory architecture and precision modes tailored for AI and HPC workloads.
-   **Flexible deployment options**, including support for multi-instance workloads and high-speed interconnects in clustered environments.

\------------------------------

At **Vast.ai**, we understand there are other considerations, as well. Access, flexibility, and affordability are just as important, especially when you're experimenting, scaling, or moving fast.

That's why our market-based cloud GPU rental platform gives you affordable, on-demand access to RTX 4090s, A100s, and much more. With Vast.ai, you can instantly match the right hardware to your workload – without the upfront investment and overhead of managing your own infrastructure.

Try Vast today and discover what you can achieve with compute power that fits your needs and your budget\!


Compare NVIDIA’s RTX 4090 vs A100 GPUs to find out which is better for gaming, AI, HPC, or creative work—and how to access both on Vast.ai.

NVIDIA RTX 4090 vs. A100: Two Powerhouses, Two Purposes


Welcome back to another update from the team at **Vast.ai**\! Over the past two months, we've been hard at work refining our platform and rolling out features that make your experience on Vast faster, smoother, and more powerful. These updates are part of our ongoing effort to support your projects with scalable, flexible infrastructure you can rely on.

We're also pleased to have achieved [**SOC 2 Type I certification**](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) recently, a clear validation of the controls we've put in place to protect customer data and ensure the integrity of our infrastructure. The independent SOC 2 audit confirms that our security practices meet the highest standards of regulatory compliance – giving you added confidence as you build on our platform.

Another exciting update: [**H200 GPUs**](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights) are now available on Vast\! If you're pushing the limits of high-performance compute, these machines are ready to meet the challenge.

Let's take a look at what else is new\!

### NVIDIA Cloud GPU Updates

Whether it's expanded hardware options, new templates, or updated guides, our latest improvements give you a more powerful, customizable environment for every stage of your workflow.

#### Highlights of the latest enhancements:

* **New Templates Added**

  * [***Dia 1.6b TTS***](https://cloud.vast.ai/?ref_id=62897&template_id=6ba071acb5f590913c5eaf6c2dc3d0db) – Convert text to lifelike speech using the open-source text-to-speech model from Nari Labs (full guide [here](https://docs.vast.ai/tts-with-nari-labs-dia))..  
  * [***Tari Universe***](https://cloud.vast.ai/?ref_id=62897&template_id=338f9628e43e586a87d37d18c719291d) – Run crypto mining workloads on Vast via desktop application (note: cryptocurrency funding is required).  
  * [***Pinokio Desktop***](https://cloud.vast.ai/?ref_id=62897&template_id=06c4818322f002904b324165b9dba52f) – Deploy AI applications with verified scripts from trusted publishers, without manual setup.

* **Introducing Local Volumes (CLI-only)**

  * Local Volumes are **persistent Docker volumes** that keep your data intact across restarts, new instances, and failures – so your storage survives instance deletion.  
  * **Benefits:** Share one volume across multiple containers on the same machine, and spin up fresh instances without data-transfer overhead. 

* **Upgraded Instance Portal**

  * For Vast.ai instances using [recommended templates](https://cloud.vast.ai/templates/), instances now launch via **secure Cloudflare tunnels** by default.  
  * Create a user-friendly **custom tunnel URL** that points to a port on your instance.  
  * **Live logs** and **live stats** (CPU, Memory, and Disk) for your instance are now viewable.

* **Vast Teams Improvements**

  * **Smoother invites** (fast and easy team creation), **standalone teams** (no need to convert your individual account), and **context switching** (seamlessly switch between Individual and Team views).  
  * **Access Keys:** Share API Keys and view Team-wide Session Keys as owner.  
  * Various stability and usability improvements.

For more information and helpful links, check out the full change log for our April–May 2025 updates at the bottom of this page.

### Our Commitment

Whether you're trying out the H200, deploying AI applications with just a few clicks, launching a new template, or exploring our latest platform features, we're here to make your experience on **Vast.ai** as smooth and flexible as possible. However you use Vast, our goal remains the same: to deliver reliable, cost-effective compute whenever you need it.

Got questions? Help is always within reach\! Contact our customer support team at [support@vast.ai](mailto:support@vast.ai) anytime, or join our Discord server for additional resources and community support.

### Change Log

#### New Templates

* **Dia 1.6b TTS** – Transform text into ultra-realistic speech using Dia, the open-source text-to-speech model from Nari Labs.  
  * [Full Guide](https://docs.vast.ai/tts-with-nari-labs-dia)  
  * [Launch an Instance with the Dia Template](https://cloud.vast.ai/?ref_id=62897&template_id=6ba071acb5f590913c5eaf6c2dc3d0db)  
* **Tari Universe** – Desktop Crypto mining application you can run on Vast. IMPORTANT: You must fund your Vast account with cryptocurrency if you want to use the service for mining.  
  * [Launch an Instance with the Tari Universe Template](https://cloud.vast.ai/?ref_id=62897&template_id=338f9628e43e586a87d37d18c719291d)  
* **Pinokio Desktop** – Access an extensive library of verified scripts from trusted publishers and deploy AI applications effortlessly without manual setup.  
  * [Launch an Instance on Vast with the Pinokio Desktop Template](https://cloud.vast.ai/?ref_id=62897&template_id=06c4818322f002904b324165b9dba52f)

####  Introducing Local Volumes (CLI only – UI coming soon\!)

**What are Local Volumes?**  
Persistent Docker volumes to keep your data intact across restarts, new instances, and failures.

**Key client benefits:**  
• Persistent storage survives instance deletion.  
• Share one volume across multiple containers on the same machine.  
• Spin up fresh instances on the same machine without data-transfer overhead.

**Get started:**  
• Search for available Volumes via CLI:  vastai search volumes  
• General docs: [https://docs.vast.ai/volumes](https://docs.vast.ai/volumes) 

*Note: Currently, clients can only rent Local Volumes through CLI. UI support and Network Volumes are coming soon.*

#### Teams Improvements

We’ve improved the Vast Teams experience for user access management and collaboration:

* **Smoother Invites:** Get team members connected on Vast quickly.  
* **Standalone Teams:** Create Teams without converting your individual account.  
* **Context Switching:** Seamlessly switch between individual and Team views.  
* **Access Keys:** Share API Keys and view Team-wide Session Keys as owner.  
* **Team Basics:** Transfer credits to/from your team, transfer ownership, and other basics.  
* **Bug Fixes:** Various stability and usability improvements.

[Create a team](https://cloud.vast.ai/manage-members/) and let us know your feedback\!

#### Instance Portal upgrade

We've also rolled out a major upgrade to the **Instance Portal** that appears on Vast.ai instances using [recommended templates](https://cloud.vast.ai/templates/).

**What’s New:**

* **Secure Tunnel URLs:** Instances now launch via secure Cloudflare tunnels by default, eliminating the “not secure” screen.  
* **Custom Tunnel URLs:** Create a user-friendly URL that points to a port on your Instance. Optionally use a named tunnel to use a custom domain.  
* **Live Logs:** See a live stream of log updates on your Instance.  
* **Live Stats:** See live CPU, Memory, and Disk stats for your Instance.

Deploy a recommended template instance or check out the [updated Instance Portal guide](https://docs.vast.ai/instance-portal) to learn more\!

#### Opt in to the new Instances UI

We've rolled out an (optional) UI update as part of our ongoing UI refresh:

* Cleaner instance card design  
* Instance data fetching improvements  
* Various instance UI bug fixes

Opt-in: Go to [Settings \> Early Access](https://cloud.vast.ai/account/) and toggle it on. You can switch back anytime.

#### SOC2 Type I Certification complete.

#### H200s available now\!

Vast.ai adds H200 GPUs, SOC 2 Type I cert, new templates, persistent storage, and better teams & instance tools for faster, secure cloud compute.

April-May 2025 Product Updates


## Introduction

The recent advancements in AI for video generation have been very fast paced. Models now create much more realistic videos than before, with multiple different use cases. One such model is [HunyuanVideo](https://huggingface.co/tencent/HunyuanVideo), a model known for its impressive outputs—and tackle the practical challenges of running it efficiently on high-memory GPUs like the A100 or H100.

HunyuanVideo is Tencent's state-of-the-art text-to-video generation model that rivals or surpasses leading closed-source alternatives. As the largest open-source video generation model with over 13 billion parameters, HunyuanVideo represents a significant breakthrough in AI-powered video creation.

We also have a [notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/LznBGR/aec37016e507d00895435abda7f0a3bc6e29625a/files/hunyuan_video_vast.ipynb) to follow along once you deploy the Vast Instance.

### Key Innovations

- **Unified Architecture**: Employs a "Dual-stream to Single-stream" transformer design that effectively handles both image and video generation.
- **MLLM Text Encoder**: Utilizes a multimodal large language model for superior text-to-visual alignment compared to previous text-encoder architectures.
- **Efficient Compression**: Implements causal 3D VAE for efficient spatial-temporal compression to enable training at higher resolution.
- **Automatic Prompt Rewriting**: Features intelligent prompt enhancement to optimize user-given prompts tailored for the model's consumption.

### What You’ll Learn

In this guide, we will:

1. Set up a custom Docker template for HunyuanVideo on Vast.ai.
2. Launch the Docker template on a suitable GPU instance.
3. Download pretrained models and generate high-quality videos from text prompts.

Let’s get started with HunyuanVideo!

---

## Renting an Instance on Vast

### Create a Custom Template

Tencent maintains a custom Docker image for HunyuanVideo: `hunyuanvideo/hunyuanvideo:cuda_12`. To run the model on Vast, we'll create a custom template using this image.

Follow these steps:

1. Ensure you have a [Vast.ai](https://vast.ai/) account.
2. Navigate to the **Templates** page on the [Vast Console](https://cloud.vast.ai/templates/).
3. Find the existing template named **PyTorch (CuDNN Runtime)** and click **Edit**.
4. Rename the template to **HunyuanVideo**.
5. Replace the Docker image in the **Image Path:Tag** field with `hunyuanvideo/hunyuanvideo:cuda_12`.
6. Set the **On-start Script** to:
   ```
   git clone https://github.com/tencent/HunyuanVideo
   ```
   This ensures the HunyuanVideo repository is downloaded on instance startup.
7. Allocate sufficient disk space, for example, **80 GB**.
8. Click **Save & Use** to save your template and prepare to deploy it.

---

### Selecting an Instance

HunyuanVideo requires a GPU with **at least 80GB VRAM** to run smoothly. Available GPUs meeting this requirement include Nvidia’s A100 and H100 cards.

To select an instance:

1. Filter Vast instances based on:
   - **Instance Type**: A100 or H100
   - **Number of GPUs**: 1
   - **VRAM**: ≥ 80GB
2. Choose and rent a suitable instance.
3. Install the Vast TLS certificate in your browser, enabling secure access to your Jupyter notebook server: [Installing TLS Certificate Guide](https://docs.vast.ai/instances/jupyter#1SmCz).
4. Open the Jupyter server via https://cloud.vast.ai/instances/, clicking **Open** for your rented instance.
5. Upload this notebook into the directory `/workspace/HunyuanVideo/` on the server.

---

## Downloading the Model Weights

Before generating videos, we need to download pretrained model weights from Hugging Face. These include the video model weights and the text encoders.

Run the following commands inside your instance:

```bash
huggingface-cli download tencent/HunyuanVideo --local-dir ./ckpts
huggingface-cli download xtuner/llava-llama-3-8b-v1_1-transformers --local-dir ./ckpts/llava-llama-3-8b-v1_1-transformers
python hyvideo/utils/preprocess_text_encoder_tokenizer_utils.py --input_dir ckpts/llava-llama-3-8b-v1_1-transformers --output_dir ckpts/text_encoder
huggingface-cli download openai/clip-vit-large-patch14 --local-dir ./ckpts/text_encoder_2
```

For a more in-depth discussion of the checkpoints, refer to [Tencent’s checkpoint README](https://github.com/Tencent/HunyuanVideo/blob/main/ckpts/README.md).

---

## Generating Videos using HunyuanVideo

### Displaying Videos in Jupyter

To conveniently view generated videos within the Jupyter notebook, let’s define a helper function using `IPython.display.Video`:

```python
from IPython.display import Video

def show_video(video_path, width=640, height=360, embed=True):
    """
    Display a video in a Jupyter notebook.

    Parameters:
    -----------
    video_path : str
        Path to the video file (local file or URL)
    width : int, optional
        Width of the video player in pixels
    height : int, optional
        Height of the video player in pixels
    embed : bool, optional
        Whether to embed the video in the notebook (True)
        or just link to it (False)

    Returns:
    --------
    IPython.display.Video
        Video display object
    """
    return Video(video_path, width=width, height=height, embed=embed)
```

---

### Generating Your First Video: Cat on the Grass

Let’s generate a realistic video of a cat walking on the grass using the example prompt provided by Tencent.

Run the following command in your terminal:

```bash
python sample_video.py \
    --video-size 720 1280 \
    --video-length 129 \
    --infer-steps 50 \
    --prompt "A cat walks on the grass, realistic style." \
    --flow-reverse \
    --use-cpu-offload \
    --save-path ./results
```

This command will create a video approximately 129 frames long (roughly 4-5 seconds at standard frame rates) at 720x1280 resolution.

Once the generation finishes, locate the video file inside the `./results` directory.

To display it in your notebook, run:

```python
show_video("./results/[your_video_filename].mp4")
```

Replace `[your_video_filename].mp4` with the actual file name.

You should see a high-quality, realistically animated cat walking on grass, showcasing HunyuanVideo’s remarkable detail in animal motion and textures.

---

### Creating a Video of an Astronaut Walking on the Moon

Next, try generating a completely different scene:

```bash
python sample_video.py \
    --video-size 720 1280 \
    --video-length 129 \
    --infer-steps 50 \
    --prompt "An astronaut walks across the moon, realistic style." \
    --flow-reverse \
    --use-cpu-offload \
    --save-path ./results
```

After the video is generated, display it in your notebook the same way:

```python
show_video("./results/[your_astronaut_video].mp4")
```

The resulting video will demonstrate the model’s versatility at rendering vastly different environments and characters, from furry animals to astronauts in space suits.

---

## Conclusion and Next Steps

You've now successfully generated your first videos using HunyuanVideo on a Vast-powered cloud instance. This powerful model unlocks creative possibilities in AI-driven video generation with:

- **Flexible Resolution Options**: Try different `--video-size` values (e.g., 1280×720, 960×960) to optimize for your needs.
- **Adjustable Quality Settings**: Modify `--infer-steps` (default 50) to trade off video generation quality and speed.
- **Creative Control**: Experiment with `--embedded-cfg-scale` (default 6.0) to balance prompt fidelity and creative variance.
- **Deterministic Outputs**: Set `--seed` to reproduce favorite generated videos reliably.

Deploying large models like HunyuanVideo can be resource-intensive but is now accessible with cloud platforms like Vast, giving you access to top-tier GPUs without upfront hardware costs.

With these foundations, you’re ready to explore more sophisticated prompt engineering, fine-tune generation parameters, or even integrate HunyuanVideo into multimedia projects!

---

**Resources:**

- [HunyuanVideo GitHub Repository](https://github.com/tencent/HunyuanVideo)
- [Vast.ai Documentation](https://docs.vast.ai/)
- [HunyuanVideo Checkpoints README](https://github.com/Tencent/HunyuanVideo/blob/main/ckpts/README.md)

Learn how to run Tencent’s HunyuanVideo text-to-video model on Vast.ai using A100/H100 GPUs to generate realistic AI-powered videos from prompts.

Text to Video Generation Using HunyuanVideo on Vast


[RolmOCR](https://huggingface.co/reducto/RolmOCR) from Reducto is a powerful, open-source document OCR solution that delivers superior performance while requiring fewer resources than comparable models. Compared to its predecessor, [allenai/olmOCR-7B-0225-preview](https://huggingface.co/allenai/olmOCR-7B-0225-preview), this new model is built on a better base model, Qwen2.5-VL-7B instead of Qwen2.0-VL-7b. This model isn't trained to take in metadata information and with some 15% rotation of the documents, which enables it to be more flexible under different conditions. Overall, it excels at OCR on PDF's, documents, forms, and invoices. Because it is open source, companies can build on top of this model for their own proprietary pipelines while not sending data to other providers or model hosting services.

Vast.ai offers a GPU marketplace where you can rent compute power at lower costs than major cloud providers, with the flexibility to select specific hardware configurations optimized for specific models and GPU RAM needs, while keeping all of your company's data private.

In this guide, we'll demonstrate how to extract structured pricing data from invoice images using this new model. To follow along, we also have a notebook version you can find [here](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/nEbMzk/c6b76123dbef385aa3576dcda376eb2db876fc28/files/reducto_rolmocr_vast.ipynb)

## Deploying RolmOCR on Vast

### Install Vast

First, we'll install and set up the Vast API.

You can get your API key from the [Account Page](https://cloud.vast.ai/account/) in the Vast Console and set it below in `VAST_API_KEY`.

```bash
pip install vastai==0.2.6
```

```bash
export VAST_API_KEY="" # Your API key here
vastai set api-key $VAST_API_KEY
```

### Search for an Instance

Next, we'll search for an instance to host the model. While `reducto/RolmOCR` requires at least 16GB VRAM, selecting an instance with 60GB VRAM will accommodate larger documents and enable a wider context window.

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 60 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

### Deploy the Instance

Using the instance ID from the search, deploy the model to the instance.

Note: we set `VLLM_USE_V1=1` to use the v1 engine for vLLM, which `reducto/RolmOCR` requires.

```bash
export INSTANCE_ID= # insert instance ID here
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e VLLM_USE_V1=1' --disk 80 --args --model reducto/RolmOCR
```

## Using the OpenAI API to Call `reducto/RolmOCR`

### Download Dependencies

While waiting for the instance to finish setting up and starting to serve the model, we'll install the required Python packages locally.

```bash
pip install --upgrade openai datasets pydantic
```

### Download Sample Invoice Data

Now, we'll use the `datasets` library to load a subset of invoice data from the `katanaml-org/invoices-donut-data-v1` dataset on Hugging Face. This dataset contains 500 annotated invoice images with structured metadata for training document extraction models.

```python
from datasets import load_dataset

# Stream the dataset
streamed_dataset = load_dataset("katanaml-org/invoices-donut-data-v1", split="train", streaming=True)

# Take the first 3 samples
subset = list(streamed_dataset.take(3))
```

Next, we'll create an `encode_pil_image` function to convert images from the dataset into a base64 encoded string format suitable for the OpenAI API.

```python
import base64
import io
from PIL import Image

def encode_pil_image(pil_image):
    # Resize image while maintaining aspect ratio
    max_size = 1024  # Maximum dimension
    ratio = min(max_size / pil_image.width, max_size / pil_image.height)
    new_size = (int(pil_image.width * ratio), int(pil_image.height * ratio))
    resized_image = pil_image.resize(new_size, Image.Resampling.LANCZOS)

    # Convert to bytes with JPEG format and reduced quality for smaller size
    img_byte_arr = io.BytesIO()
    resized_image.save(img_byte_arr, format='JPEG', quality=85)
    img_byte_arr = img_byte_arr.getvalue()
    return base64.b64encode(img_byte_arr).decode("utf-8")
```

### Enforcing Structured Data Extraction

We'll create an `Invoice` Pydantic model for use to store the invoice number and the amount.

```python
from pydantic import BaseModel

class Invoice(BaseModel):
    invoice_number: str
    invoice_amount: str

json_schema = Invoice.model_json_schema()
```

## Configuring the API Client

Here we're creating a function to interface with the RolmOCR endpoint via the OpenAI API.

Set your instance's IP address and port from the [Instances tab](https://cloud.vast.ai/instances/) of the Vast AI Console.

You can see that we're leveraging the json schema fromt he above `Invoice` Model to pull have our model generate structured data.

```python
from openai import OpenAI

VAST_IP_ADDRESS = ""  # Insert your instance IP address
VAST_PORT = ""        # Insert your instance port
base_url = f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"

client = OpenAI(api_key="", base_url=base_url)

model = "reducto/RolmOCR"

def ocr_page_with_rolm(img_base64):
    response = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/png;base64,{img_base64}"},
                    },
                    {
                        "type": "text",
                        "text": "Return the invoice number and total amount for each invoice as a json: {invoice_number : str, invoice_amount: str}",
                    },
                ],
            }
        ],
        extra_body={"guided_json": json_schema},
        temperature=0.2,
        max_tokens=500
    )
    return response.choices[0].message.content
```

### Extract and Display Invoice Data

Iterate over the sample subset of invoices, display the image, extract the invoice number and amount with RolmOCR, and compare the extracted data to the ground truth.

```python
import matplotlib.pyplot as plt
import json

invoices = []
ground_truth = []
for sample in subset:
    # Display invoice image
    plt.figure(figsize=(10, 14))
    plt.imshow(sample['image'])
    plt.axis('off')
    plt.show()

    # Encode and process with OCR
    img_base64 = encode_pil_image(sample['image'])
    result = ocr_page_with_rolm(img_base64)
    result_dict = json.loads(result)
    invoices.append(result_dict)

    # Extract ground truth fields
    ground_truth_i = json.loads(sample["ground_truth"])
    ground_truth_dict = {
        "invoice_number": ground_truth_i["gt_parse"]["header"]["invoice_no"],
        "invoice_amount": ground_truth_i["gt_parse"]["summary"]["total_gross_worth"]
    }
    ground_truth.append(ground_truth_dict)

    print("Ground Truth")
    print(ground_truth_dict)

    print("Extracted Info")
    print(result_dict)
```

### Example Outputs

#### Invoice 1

![Invoice 1](/uploads/reducto_rolmocr_output_1.png)

```
Ground Truth
{'invoice_number': '40378170', 'invoice_amount': '$8,25'}
Extracted Info
{'invoice_number': '40378170', 'invoice_amount': '$8.25'}
```

#### Invoice 2

![Invoice 2](/uploads/reducto_rolmocr_output_2.png)

```
Ground Truth
{'invoice_number': '61356291', 'invoice_amount': '$ 212,09'}
Extracted Info
{'invoice_number': '61356291', 'invoice_amount': '$212.09'}
```

#### Invoice 3

![Invoice 3](/uploads/reducto_rolmocr_output_3.png)

```
Ground Truth
{'invoice_number': '49565075', 'invoice_amount': '$96,73'}
Extracted Info
{'invoice_number': '49565075', 'invoice_amount': '$96,73'}
```

As observed, the output from `reducto/RolmOCR` closely matches the expected ground truth, demonstrating its accuracy in extracting key invoice information. The differences between the extracted and ground truth in these cases is around commas versus decimal points, which can be easily solved for with simple downstream methods.

## Conclusion

In this walkthrough, we demonstrated how to deploy and use RolmOCR on Vast to extract structured data from invoice images efficiently.

RolmOCR delivers high accuracy for critical fields such as invoice numbers and amounts, while staying more robust with image-based proxessing and reduced RAM usage. When combined with Vast.ai's flexible and cost-effective GPU marketplace, this forms a powerful solution for scalable, privacy-focused document processing workflows.

This approach can be extended to extract a wide range of structured data from diverse document formats, unlocking advanced automation and insights for businesses of all sizes.


Learn how to deploy Reducto’s RolmOCR on Vast.ai to extract structured invoice data using affordable GPUs and accurate open-source document OCR.

Running RolmOCR on Vast


NVIDIA's GeForce RTX 5090D was supposed to be a clever workaround – a GPU specially created to comply with U.S. export regulations while still delivering advanced gaming performance to Chinese consumers.

But recent reports suggest that even this modified version may not make it to market.

### NVIDIA RTX 5090D: A China-Specific Solution

The RTX 5090D was designed specifically for the Chinese market in response to the U.S. government's increasingly strict export rules. These rules target high-performance chips like the NVIDIA H100 and H20, and AMD's MI300 series, banning them from being sold in China if they exceed certain memory and I/O bandwidth thresholds.

To get around these limits, NVIDIA modified the RTX 5090D to lower its AI throughput while keeping specs that matter to gamers intact – such as core count, memory size, and clock speeds. That may not have been sufficient, however.

### The Bandwidth Problem

Intended to limit China's ability to develop advanced AI systems, U.S. export regulations on chips impose strict caps on memory and I/O bandwidth: **1,400 GB/s and 1,100 GB/s, respectively – _or_ a combined total over 1,700 GB/s**.

If the RTX 5090D exceeds any of these thresholds, it could still be restricted, despite being designed with reduced AI capabilities and not typically suited for large-scale enterprise workloads. The problem is, it very well might fall into the restricted category based on total bandwidth.

NVIDIA appears to be [taking preemptive action](https://wccftech.com/nvidia-reportedly-prepares-for-a-ban-on-the-geforce-rtx-5090d-in-china/) – asking add-in board (AIB) partners to halt shipments and prep work, and even putting a pause on related orders by cooling module manufacturers.

So far, there's been no official statement from NVIDIA. But after getting burned when the RTX 4090 was unexpectedly swept up in a prior wave of export controls, NVIDIA is reportedly acting early this time with this "precautionary" move amid ongoing regulatory uncertainty between the U.S. and China.

### What This Means for the Market

Whether this sales halt is temporary or permanent remains to be seen, but the wider implications are potentially far-reaching. The RTX 5090D was the only flagship-class Blackwell GPU intended for Chinese gamers. If it's taken off the table, that leaves a gap in the market – and an opportunity for rivals like AMD, which is reportedly preparing a China-exclusive gaming GPU.

Furthermore, these ongoing developments reflect a growing reality: that geopolitical decisions are impacting GPU availability worldwide.

While Chinese consumers lose access to another high-end option, global suppliers are keeping a close eye on what happens next. Constraints in one region can ripple across the supply chain, affecting availability, pricing, and competitive positioning elsewhere, as well.

### High-End Compute On Demand

While geopolitical tensions disrupt supply in some regions, developers, researchers, and creators still need reliable access to powerful hardware. Vast.ai offers a flexible and cost-effective alternative.

Our cloud GPU rental platform delivers high-performance compute – H100s, H200s, RTX 5090s, and more – at a fraction of the typical cost. With on-demand pricing and spot instance options, **Vast users save 5–6X on GPU compute** compared to traditional cloud providers.

Explore your options and get started today\! With Vast, you don't have to wait to access the power you want.


NVIDIA's RTX 5090D, designed specifically for the Chinese market to bypass US export restrictions, now faces uncertainty as it may still exceed bandwidth thresholds. NVIDIA has reportedly asked partners to halt shipments amid regulatory concerns, potentially leaving a gap for competitors in China's high-end GPU market.

RTX 5090D Faces Uncertainty in China Amid Export Crackdown


NVIDIA has [confirmed](https://nvidianews.nvidia.com/news/nvidia-blackwell-rtx-pro-workstations-servers-agentic-ai) the launch of the RTX Pro Blackwell series – a revolutionary generation of workstation and server GPUs. Designed to meet the needs of professional designers, data scientists, developers, and content creators, the lineup features a cutting-edge **RTX Pro 6000 Blackwell** GPU for workstations, plus other desktop and laptop variants and a data center version.

After early signs of the card [appeared in shipping logs](https://vast.ai/article/nvidia-rtx-pro-6000-x-rumors-blackwell-gpu-with-96gb-vram-spotted-in-shipping-log) a couple of months ago, we now have full details straight from the source – and this GPU is no minor upgrade. Built on NVIDIA's Blackwell architecture and featuring an incredible **96GB of GDDR7 memory** on a **512-bit bus**, the RTX Pro 6000 is set to redefine what's possible on a single professional workstation.

Here's what we now know about the RTX Pro 6000 Blackwell GPU.

### NVIDIA RTX Pro 6000: Specs That Mean Business

At the heart of the RTX Pro 6000 is the **GB202 die**, aligning with previous high-end Blackwell GPUs, like the RTX 5090\. But while the RTX 5090 is geared toward gamers and other enthusiasts, the RTX Pro 6000 goes all in for professional use.

We've certainly seen professional-grade GPUs with massive VRAM configurations before – there's the RTX 6000 Ada with 48GB of memory, for instance. However, the RTX Pro 6000 offers **double that amount**, with the faster speeds and efficiency of GDDR7 ECC memory. This makes it one of the most powerful workstation GPUs ever released.

And that's not all. On top of **24,064 CUDA cores** and **1,792 GB/s of memory bandwidth**, here's what else you get with this beast of a machine:

- **Streaming multiprocessors** with new neural shaders, delivering up to **1.5X** faster throughput for AI-augmented graphics workflows.
- **Fourth-gen RT cores** offering up to **2X** faster ray tracing performance for photorealism and complex 3D environments, with optimizations for **RTX Mega Geometry**.
- **Fifth-gen Tensor cores** capable of **4,000 AI TOPS**, supporting **FP4** precision and **DLSS 4 Multi Frame Generation** for cutting-edge AI graphics and accelerated model prototyping.
- **Ninth-gen NVENC** **& sixth-gen NVDEC**, improving encoding and decoding speed and quality for pro video work.
- **PCIe Gen 5 interface**, doubling bandwidth for faster CPU-GPU data transfer and smoother performance on heavy data workloads.
- **DisplayPort 2.1** support for ultra-high refresh rates – up to **4K at 480Hz** and **8K at 165Hz** – enabling multi-monitor setups with high color accuracy.
- **Multi-Instance GPU (MIG)**, allowing secure partitioning into up to **four GPU instances** for better workload isolation and resource allocation.

Not surprisingly, the RTX Pro 6000 does require a substantial **600 watts** of power, albeit only slightly more than the 575 watts of the RTX 5090\. But with that power comes the performance needed for agentic AI, immersive XR, large-scale simulation, and autonomous systems development – workloads where speed, memory, and throughput are non-negotiable.

(For more details and specs, check out the table at the bottom of this page\!)

So the only question that remains is: when can you get your hands on one?

### Availability and Rollout Timeline

The RTX Pro 6000 Workstation Edition is [expected](https://nvidianews.nvidia.com/news/nvidia-blackwell-rtx-pro-workstations-servers-agentic-ai) to **start shipping in May 2025** from manufacturers like BOXX, Dell, HP, Lambda, and Lenovo.

The server version of the RTX Pro 6000 is also on the way, with systems from Cisco, Dell, HP, Lenovo, and Supermicro expected to launch "soon," according to NVIDIA. Laptop variants, featuring **24GB of VRAM**, are expected to ship later in the year, from Dell, HP, Lenovo, and Razer.

NVIDIA hasn't announced official pricing yet, but estimates place the RTX Pro 6000 around $8500 – quite the investment. Fortunately, for those who need top-tier professional-grade GPU performance without the upfront cost, there are more accessible options.

### Access GPU Power Without the Overhead

Here at Vast.ai, we're excited to expand our GPU rental offerings as the **RTX Pro 6000 series** rolls out this year. In the meantime, you can tap into powerful GPUs like the **H100**, **H200**, and other high-performance options, all available on-demand through our platform, today.

With Vast, you can enjoy flexible, cost-effective access to serious compute power – with no long-term commitments and no hardware overhead – that's ready when you are.

### NVIDIA RTX Pro 6000: Key Specs

| Specification                    | RTX Pro 6000                                         |
| :------------------------------- | :--------------------------------------------------- |
| **Architecture**                 | Blackwell                                            |
| **GPU**                          | GB202                                                |
| **GPU Memory**                   | 96GB GDDR7 ECC                                       |
| **Memory Bandwidth**             | 1792 GB/s                                            |
| **Memory Interface**             | 512-bit                                              |
| **CUDA Cores**                   | 24,064                                               |
| **5th-Gen Tensor Cores**         | 752                                                  |
| **4th-Gen RT Cores**             | 188                                                  |
| **L2 Cache**                     | 128 MB                                               |
| **AI TOPS**                      | 4000                                                 |
| **Single-Precision Performance** | 125 TFLOPS                                           |
| **RT Core Performance**          | 380 TFLOPS                                           |
| **System Interface**             | PCIe 5.0 x16                                         |
| **Display Connectors**           | 4x DisplayPort 2.1b                                  |
| **Video Engines**                | 4x NVENC (9th Gen); 4x NVDEC (6th Gen)               |
| **MIG Support**                  | Up to 4x 24GB; or 2x 48GB; or 1x 96GB                |
| **Graphics APIs**                | DirectX 12, Shader Model 6.6, OpenGL 4.6, Vulkan 1.3 |
| **Compute APIs**                 | CUDA 12.8, OpenCL 3.0, DirectCompute                 |
| **Thermal Solution**             | Double-flow-through                                  |
| **Form Factor**                  | 5.4” H x 12” L, dual slot, extended height           |
| **Total Board Power**            | 600 watts                                            |


NVIDIA has confirmed the launch of the RTX Pro Blackwell series – a revolutionary generation of workstation and server GPUs. Designed to meet the needs of professional designers, data scientists, developers, and content creators, the lineup features a cutting-edge RTX Pro 6000 Blackwell GPU for workstations, plus other desktop and laptop variants and a data center version.

NVIDIA RTX Pro 6000 Blackwell: Redefining the Workstation GPU


## Introduction

Meta's Llama 4 is a breakthrough AI model combining state-of-the-art multimodal capabilities with the computational efficiency of mixture-of-experts (MoE) architecture. These models can process both text and images through an early fusion design that seamlessly integrates different modalities. Perhaps most impressively, they support context windows of up to 10 million tokens - dramatically expanding what's possible with large language models.

The Llama 4 family includes:

- **Llama 4 Scout**: 17B active parameters with 16 experts - efficient enough to run on modest GPU setups.

- **Llama 4 Maverick**: 17B active parameters but with 128 experts - offering enhanced capabilities while maintaining efficiency.

- **Llama 4 Behemoth**: 288B active parameters with 16 experts - Meta's most powerful model (not yet publicly released).

What makes these models special:

- **Mixture-of-Experts (MoE)**: Unlike traditional models that activate all parameters for every token, MoE models selectively activate only a fraction - dramatically improving computational efficiency.
- **Early Fusion**: Vision and text processing are integrated directly into the model backbone, enabling true multimodal reasoning.
- **10M Token Context Window**: Process entire books, codebases, or document collections in a single prompt - enabling comprehensive analysis impossible with earlier models.

In this guide, we'll deploy Llama 4 models on Vast.ai using three practical hardware configurations:

1. Llama 4 Scout on 8× H100 GPUs
2. Llama 4 Scout on 4× H100 GPUs
3. Llama 4 Maverick on 8× H200 GPUs

You'll learn exactly how to set up each configuration, deploy the models with the vLLM server, and interact with them through an OpenAI-compatible API. By the end, you'll have hands-on experience running some of the most advanced AI models available today.

## Installing Vast

Let's start by installing the Vast.ai command-line tools and setting up our API key:

```bash
pip install --upgrade vastai
```

```bash
# Here we will set our api key
export VAST_API_KEY="" #Your key here
vastai set api-key $VAST_API_KEY
```

## Running Llama 4 Scout on 8× H100s

### Choosing the Right Hardware

First, we need to search for GPUs on Vast.ai to run the Llama 4 Scout model on 8× H100 GPUs with a 200k token context window. This model requires specific hardware capabilities to run efficiently with vLLM's optimizations. Here are our requirements:

1. 8× H100 GPUs (80GB of VRAM each) to accommodate:

   - Llama 4 Scout model weights (17B active parameters with 16 experts)
   - KV Cache for handling the 200k token context window

2. A static IP address for:

   - Stable API endpoint hosting
   - Consistent client connections

3. At least one direct port that we can forward for:

   - vLLM's OpenAI-compatible API server
   - External access to the model endpoint
   - Secure request routing

4. At least 600GB of disk space to hold the model and other dependencies

Let's search for machines that meet these requirements:

```bash
vastai search offers " \
gpu_name = H100_NVL \
num_gpus = 8 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 600 \
rentable = true"
```

### Deploying the Server via Vast

We'll choose a machine from our search results and copy and paste the id below to set `INSTANCE_ID`.

We use `vastai create instance` to create an instance that:

1. Uses `vllm/vllm-openai:latest` docker image. This gives us an OpenAI-compatible server.
2. Forwards port `8000` to the outside of the container, which is the default OpenAI server port.
3. Uses `--model meta-llama/Llama-4-Scout-17B-16E-Instruct` to serve the Llama 4 Scout model.
4. Uses Llama 4-specific parameters:
   - `--tensor-parallel-size 8` to distribute inference across multiple GPUs.
   - `--max-model-len 200000` to set a 200k token context window.
   - `--override-generation-config='{"attn_temperature_tuning": true}'` for optimal text generation quality.
5. Uses `--disk 600` to ensure that we have 600GB of disk space for model weights and dependencies.

Note: Ensure that you fill in your huggingface token `HUGGING_FACE_HUB_TOKEN` to access the model. You'll need to accept the Llama 4 license terms on the model's huggingface page: https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct

```bash
export INSTANCE_ID=19319856 #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGING_FACE_HUB_TOKEN' --disk 600 --args --model meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 8 --max-model-len 200000 --override-generation-config='{"attn_temperature_tuning": true}'
```

### Get Instance IP Address and Port

Now, we need to get our IP address and port to call our model. We first need to wait for our machine to download the image and the model and start serving. This will take a few minutes. The logs will show you when it's done.

Next, go to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console and find the instance you just created.

At the top of the instance, there is a button with an IP address in it. Click this and a panel will show up of the ip address and the forwarded ports.
You should see something like:

```
Open Ports
XX.XX.XXX.XX:YYYY -> 8000/tcp
```

You will need the IP address (XX.XX.XXX.XX) and the port (YYYY) for the next step.

### Call our Model

#### Install OpenAI

To call our model we will install the OpenAI SDK.

```bash
pip install --upgrade openai
```

#### Download our Data

To show the power of our large context window, we will download The Great Gatsby from Project Gutenberg and ask Llama 4 to summarize it.

Here we download the text for the book:

```python
import requests

# Download the text file to a variable
url = "https://www.gutenberg.org/ebooks/64317.txt.utf-8"
text = requests.get(url).text

# Print the first 100 characters to confirm it worked
print(text[:200])
```

### Call Our Model

We will then use the OpenAI SDK. To do this, we need to set our `VAST_IP_ADDRESS` and `VAST_PORT` that we found above.

```python
from openai import OpenAI

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client with your server URL
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an actual API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

prompt = "please summarize this book in two sentences:" + text

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    messages=[
        {"role": "user", "content": prompt}
    ],

)
print(response.choices[0].message.content)
```

### Model Response

We see that our model was able to ingest a large amount of text and summarize our book.

```
Here is a two-sentence summary of The Great Gatsby:

Set in the roaring twenties, The Great Gatsby is a classic novel by F. Scott Fitzgerald that revolves around the mysterious millionaire Jay Gatsby and his obsession with winning back his lost love, Daisy Buchanan. Through the eyes of narrator Nick Carraway, the novel explores themes of wealth, class, love, and the corrupting influence of materialism in the excesses of the Jazz Age
```

## Running Llama 4 Scout on 4× H100s

After successfully deploying Scout on 8× H100s, we'll now try a more cost-effective setup using 4× H100s. This will reduce our context window capacity but allow us to save on GPU costs.

We'll start by searching for an instance that meets our specifications:

```bash
vastai search offers " \
num_gpus = 4 \
gpu_name = H100_SXM \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 600 \
rentable = true"
```

We'll choose a machine from our search and create our instance with 4 GPUs and a 100k token context window using `--tensor-parallel-size 4` and `--max-model-len 100000`:

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGING_FACE_HUB_TOKEN' --disk 600 --args --model meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 4 --max-model-len 100000 --override-generation-config='{"attn_temperature_tuning": true}'
```

Once our instance is running, we'll set the new `VAST_IP_ADDRESS` and `VAST_PORT` and call our instance like we did earlier:

```python
from openai import OpenAI

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client with your server URL
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an actual API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

prompt = "please summarize this book in two sentences:" + text

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    messages=[
        {"role": "user", "content": prompt}
    ],

)
print(response.choices[0].message.content)
```

### Model Response

```
Here is a two-sentence summary of The Great Gatsby:

Set in the roaring twenties, The Great Gatsby is a classic novel by F. Scott Fitzgerald that revolves around the mysterious millionaire Jay Gatsby and his obsession with winning back his lost love, Daisy Buchanan. Through the eyes of narrator Nick Carraway, the novel explores themes of love, greed, class, and the corrupting influence of wealth, ultimately leading to a tragic confrontation that exposes the dark underbelly of the American Dream.
```

## Running Llama 4 Maverick on 8× H200s

Finally, we'll upgrade to Llama 4's larger Maverick model with a 100k token context window on 8× H200 GPUs - a more powerful configuration that delivers enhanced capabilities.

To start, we'll search for an H200 machine with 8 GPUs:

```bash
vastai search offers " \
num_gpus = 8 \
gpu_name = H200 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 1000 \
rentable = true"
```

We'll find a machine from our search and create an instance with Llama 4 Maverick using 8 GPUs and a 100k token context window by setting `--disk 1000`, `--model meta-llama/Llama-4-Maverick-17B-128E-Instruct`, `--tensor-parallel-size 8` and `--max-model-len 100000`:

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGING_FACE_HUB_TOKEN' --disk 1000 --args --model meta-llama/Llama-4-Maverick-17B-128E-Instruct --tensor-parallel-size 8 --max-model-len 100000 --override-generation-config='{"attn_temperature_tuning": true}'
```

Finally, we'll set the new `VAST_IP_ADDRESS` and `VAST_PORT`, change our model to `"meta-llama/Llama-4-Maverick-17B-128E-Instruct"`, and call our instance:

```python
from openai import OpenAI

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client with your server URL
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an actual API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

prompt = "please summarize this book in two sentences:" + text

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct",
    messages=[
        {"role": "user", "content": prompt}
    ],

)
print(response.choices[0].message.content)
```

### Model Response

```
Here is a two-sentence summary of The Great Gatsby:

The novel is narrated by Nick Carraway, who becomes fascinated with his wealthy neighbor Jay Gatsby and becomes entangled in Gatsby's quest to win back his lost love, Daisy Buchanan, through a complex web of relationships and tragic events. Ultimately, Gatsby's dream is destroyed, and he is murdered by George Wilson, the husband of Myrtle Wilson, who was having an affair with Tom Buchanan, Daisy's husband, highlighting the corrupting influence of wealth and the elusiveness of the American Dream.
```

## Conclusion

In this post, we've demonstrated how to deploy Meta's Llama 4 models on Vast.ai using three specific hardware configurations: Llama 4 Scout on 8× H100 GPUs with a 200k token context window, Scout on 4× H100 GPUs with a 100k context window, and Llama 4 Maverick on 8× H200 GPUs with a 100k context window. We've shown that with the right GPU setup, you can easily deploy and interact with these models through an OpenAI-compatible API.

Now that you have a basic deployment working, here are some next steps you might consider:

- **Expand the context window**: While we demonstrated context windows of 100k-200k tokens, Llama 4 supports up to 10M tokens. Try increasing `--max-model-len` to experiment with longer documents, books, or codebases. Note: You will also need to increase the number of GPUs.

- **Explore multimodal capabilities**: The tutorial focused on text processing, but Llama 4 is natively multimodal. Try adapting the deployment to accept image inputs using vLLM's multimodal capabilities and the early fusion architecture.

By leveraging Vast.ai's flexible GPU infrastructure, you can cost-effectively experiment with these cutting-edge models and build applications that take advantage of their sophisticated reasoning capabilities.


Learn how to deploy and interact with Meta's Llama 4 models on Vast.ai using three specific hardware configurations: Llama 4 Scout on 8× H100 GPUs with a 200k token context window, Scout on 4× H100 GPUs with a 100k context window, and Llama 4 Maverick on 8× H200 GPUs with a 100k context window.

Blog

H100 vs A100: Comparing Two Powerhouse GPUs

Serving Infinity

NextEpoch 2024: Bringing Machine Learning to the Next Generation of Scientists

Serving Online Inference with LMDeploy on Vast.ai

July 2024 Monthly Update: Billing Page Revamp, H100 NVL GPU Addition, and More!

Meta Launches Llama 3.1: A New Era in Open-Source AI