
<GpuPriceTicker
  variant="marquee"
  title="Live GPU market tape"
  gpus="RTX 5090,RTX 4090,H200,H100 SXM,RTX PRO 6000 S,RTX PRO 6000 WS,A100 SXM4,B200,B300,L40S,RTX 3090,Tesla V100"
/>

The answer: it depends.

Cloud GPU rental costs depend on the GPU model you're looking for, supply and demand across the marketplace, whether you're renting an on-demand or interruptible instance, and many other factors. Prices can vary from one day to the next.

To give you an idea of current marketplace rates on Vast.ai, below is a live pricing table that updates automatically with the rental costs of popular GPUs in real time.

In this guide, you'll also find a quick overview of the most in-demand GPUs on Vast.ai and tips for choosing the right hardware for your workload and budget.

## Live Cloud GPU Pricing on Vast.ai

<GpuPriceTicker
  variant="table"
  title="Live pricing for GPUs in this guide"
  gpus="RTX 5090,RTX 4090,H200,H100 SXM,RTX PRO 6000 S,RTX PRO 6000 WS,A100 SXM4,B200,B300,L40S,RTX 3090,Tesla V100"
/>

Updated automatically with current marketplace data, this card shows from and median hourly rental prices for the GPUs featured or referenced in this guide.

- **"From" hourly price** is the current 25th-percentile marketplace rate. It provides a lower-market reference without relying on one unusually cheap listing.
- **"Median" hourly price** represents the typical hourly rental rate across current offers.

The card is intentionally limited to GPUs covered in this guide and follows the same order as the guide.

### Live Vast.ai and Published Provider Price Comparison

For directional context, the Vast.ai row below updates automatically. Competitor rows use date-stamped rates from each provider's published pricing page.

<GpuProviderPriceComparison />

## The Most Popular Cloud GPUs Right Now

So, which GPUs are people actually renting? Here's a look at ten of the most in-demand GPU options on Vast.ai today.

<h3 className="gpu-model-heading">
  <span className="gpu-model-heading__index">1.</span>
  <a href="https://vast.ai/pricing/gpu/RTX-5090">RTX 5090</a>
  <GpuInlinePrice gpu="RTX 5090" />
</h3>

- 32GB GDDR7 memory
- 1.79 TB/s bandwidth

The NVIDIA GeForce RTX 5090 is one of the [most sought-after consumer GPUs](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know). It's well suited for fine-tuning smaller models, inference, and demanding creative workloads.

<h3 className="gpu-model-heading">
  <span className="gpu-model-heading__index">2.</span>
  <a href="https://vast.ai/pricing/gpu/RTX-4090">RTX 4090</a>
  <GpuInlinePrice gpu="RTX 4090" />
</h3>

- 24GB GDDR6X memory
- 1.01 TB/s bandwidth

Although now a legacy card, the RTX 4090 remains a popular option as a [dependable workhorse](https://vast.ai/article/nvidia-rtx-4090-vs-a100-two-powerhouses-two-purposes) for those who don't need cutting-edge specs. It continues to offer excellent performance for tasks like LLM inference and image generation.

<h3 className="gpu-model-heading">
  <span className="gpu-model-heading__index">3.</span>
  <a href="https://vast.ai/pricing/gpu/H200">H200</a>
  <GpuInlinePrice gpu="H200" />
</h3>

- 141GB HBM3e memory
- 4.8 TB/s bandwidth

One of the highest-demand cards on Vast.ai, the H200 is a [leading enterprise AI accelerator](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights) for serious model training and high-throughput inference.

<h3 className="gpu-model-heading">
  <span className="gpu-model-heading__index">4.</span>
  <a href="https://vast.ai/pricing/gpu/H100-SXM">H100 SXM</a>
  <GpuInlinePrice gpu="H100 SXM" />
</h3>

- 80GB HBM3 memory
- 3.35 TB/s bandwidth

The enterprise-grade [H100 SXM](https://vast.ai/article/nvidia-h100-vs-l40s-power-meets-versatility) is well suited for workloads involving massive datasets, large-scale simulations, and advanced AI training for foundation models.

<h3 className="gpu-model-heading">
  <span className="gpu-model-heading__index">5.</span>
  <a href="https://vast.ai/pricing/gpu/RTX-PRO-6000-S">RTX PRO 6000 S</a>
  <span>and</span>
  <a href="https://vast.ai/pricing/gpu/RTX-PRO-6000-WS">WS</a>
  <GpuInlinePrice gpu="RTX PRO 6000 S,RTX PRO 6000 WS" />
</h3>

- 96GB GDDR7 memory
- 1.79 TB/s bandwidth

Both the RTX PRO 6000 Server and Workstation Editions are [designed](https://vast.ai/article/which-nvidia-rtx-6000-is-right-for-you) for high-performance professional applications in areas like engineering, simulation, and media production. These machines balance AI acceleration with visualization and rendering capabilities.

<h3 className="gpu-model-heading">
  <span className="gpu-model-heading__index">6.</span>
  <a href="https://vast.ai/pricing/gpu/A100-SXM4">A100 SXM4</a>
  <GpuInlinePrice gpu="A100 SXM4" />
</h3>

- 80GB HBM2e memory
- 2.04 TB/s bandwidth

Built on the Ampere architecture, the A100 SXM4 has somewhat lower rental demand but continues to power production AI environments thanks to its mature software ecosystem and broad availability.

<h3 className="gpu-model-heading">
  <span className="gpu-model-heading__index">7.</span>
  <a href="https://vast.ai/pricing/gpu/B200">B200</a>
  <GpuInlinePrice gpu="B200" />
</h3>

- 192GB HBM3e memory
- 8.0 TB/s bandwidth

Running at high utilization on Vast.ai, the [B200](https://vast.ai/article/nvidia-h200-vs-b200-comparing-datacenter-grade-accelerators) is an incredibly in-demand GPU. It serves as a unified AI platform for develop-to-deploy pipelines for organizations of any size.

<h3 className="gpu-model-heading">
  <span className="gpu-model-heading__index">8.</span>
  <a href="https://vast.ai/pricing/gpu/L40S">L40S</a>
  <GpuInlinePrice gpu="L40S" />
</h3>

- 48GB GDDR6 memory
- 864 GB/s bandwidth

The L40S is best suited for inference and rendering-focused workloads rather than large-scale model training. With plenty of memory capacity for many production AI and graphics applications, it's a [popular choice](https://vast.ai/article/comparing-nvidia-l40-vs-l40s-and-more) for those looking to balance performance with cost.

<h3 className="gpu-model-heading">
  <span className="gpu-model-heading__index">9.</span>
  <a href="https://vast.ai/pricing/gpu/RTX-3090">RTX 3090</a>
  <GpuInlinePrice gpu="RTX 3090" />
</h3>

- 24GB GDDR6X memory
- 936 GB/s bandwidth

The RTX 3090 is a budget-friendly option for AI experimentation as well as gaming and professional tasks, delivering solid performance despite its age.

<h3 className="gpu-model-heading">
  <span className="gpu-model-heading__index">10.</span>
  <a href="https://vast.ai/pricing/gpu/TESLA-V100">Tesla V100</a>
  <GpuInlinePrice gpu="Tesla V100" />
</h3>

- 32GB HBM2 memory
- 900 GB/s bandwidth

Another older card, the Tesla V100 continues to offer a good fit for legacy AI workloads and research environments when accessing the latest hardware isn't a priority.

## How to Choose the Right GPU for Your Budget

To get the most bang for your buck, first identify your workload. The amount of VRAM you need may eliminate some GPUs from consideration entirely. From there, look for the hardware that delivers the performance you need without paying for unnecessary processing power.

Below are some recommended GPUs based on different types of workloads:

<div className="article-table-scroll gpu-comparison-table">
  <table>
    <thead>
      <tr>
        <th>Workload</th>
        <th>Recommended GPUs</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>LLM inference</td>
        <td data-label="Recommended GPUs">L40S, H100, H200</td>
      </tr>
      <tr>
        <td>Fine-tuning smaller models</td>
        <td data-label="Recommended GPUs">RTX 4090, RTX 5090, A100</td>
      </tr>
      <tr>
        <td>Training large language models</td>
        <td data-label="Recommended GPUs">H100, H200, B200, B300</td>
      </tr>
      <tr>
        <td>3D rendering</td>
        <td data-label="Recommended GPUs">RTX PRO 6000, L40S</td>
      </tr>
      <tr>
        <td>Scientific computing</td>
        <td data-label="Recommended GPUs">A100, H100</td>
      </tr>
    </tbody>
  </table>
</div>

**Tip:** Consider how quickly each GPU can complete your workload. A more powerful GPU may have a higher hourly rate, but if it finishes the job much sooner, your total compute cost may be lower.

If you're prioritizing value over maximum performance, compare the live from price, median, and availability in the pricing table at the top of this page.

## Start Building with the Right GPU on Vast.ai

Once you've found the hardware that fits your workload and budget, the next step is spinning up an instance. Vast.ai offers flexible ways to deploy.

Need a fully managed inference endpoint? [Vast.ai Serverless](https://vast.ai/products/serverless) automatically provisions GPUs and scales capacity up or down using predictive optimization, with no tiers or pricing surcharges.

Want to get up and running quickly? The Vast.ai [Model Library](https://vast.ai/model-library) provides ready-to-run templates for popular open-weight models and frameworks, so you can get started in minutes instead of configuring everything from scratch.

Browse [live GPU prices](https://vast.ai/pricing) and launch your instance on Vast.ai today.


Compare live cloud GPU rental prices on Vast.ai and explore popular GPUs for AI inference, fine-tuning, training, rendering, and scientific computing.

How Much Does It Cost to Rent a GPU in the Cloud? [Live Pricing Guide]


The NVIDIA DGX Spark is small enough to fit in your hand, yet it can run AI models with up to 200 billion parameters. With enterprise-grade capability in such a compact package, NVIDIA [describes it as the world's smallest AI supercomputer](https://nvidianews.nvidia.com/news/nvidia-dgx-spark-arrives-for-worlds-ai-developers).

But what are its actual specs, and what workloads is it best suited for? Here's everything you need to know about the DGX Spark.

## What Is the NVIDIA DGX Spark?

Billed as a personal AI supercomputer for the era of autonomous agents, the DGX Spark's compact footprint is one of its biggest selling points. It measures about 6 x 6 x 2 inches, weighs just 2.6 pounds, and occupies barely more desk space than a gaming console while delivering substantially more AI compute than a traditional desktop workstation.

Featuring the GB10 Grace Blackwell Superchip, Spark combines a 20-core Arm CPU with a Blackwell-architecture GPU. It also comes with a complete AI software stack, allowing developers to prototype, fine-tune, and run large language models locally.

### NVIDIA DGX Spark Specifications

Here's a quick summary of key DGX Spark specifications:

<div className="article-table-scroll gpu-comparison-table">
<table>
  <thead>
    <tr>
      <th scope="col">Specification</th>
      <th scope="col">NVIDIA DGX Spark</th>
    </tr>
  </thead>
  <tbody>
    <tr><td>Superchip</td><td data-label="NVIDIA DGX Spark">NVIDIA GB10 Grace Blackwell</td></tr>
    <tr><td>GPU</td><td data-label="NVIDIA DGX Spark">Blackwell architecture</td></tr>
    <tr><td>CPU</td><td data-label="NVIDIA DGX Spark">20-core Arm (10 Cortex-X925 + 10 Cortex-A725)</td></tr>
    <tr><td>Memory</td><td data-label="NVIDIA DGX Spark">128 GB LPDDR5X coherent unified memory</td></tr>
    <tr><td>Storage</td><td data-label="NVIDIA DGX Spark">Up to 4 TB PCIe Gen5 NVMe SSD</td></tr>
    <tr><td>AI performance</td><td data-label="NVIDIA DGX Spark">Up to 1 petaFLOP (FP4)</td></tr>
    <tr><td>Networking</td><td data-label="NVIDIA DGX Spark">NVIDIA ConnectX-7</td></tr>
    <tr><td>Operating system</td><td data-label="NVIDIA DGX Spark">NVIDIA DGX OS</td></tr>
    <tr><td>Form factor</td><td data-label="NVIDIA DGX Spark">150 x 150 x 50.5 mm</td></tr>
    <tr><td>Power draw</td><td data-label="NVIDIA DGX Spark">Approximately 140 W</td></tr>
    <tr><td>Maximum model size</td><td data-label="NVIDIA DGX Spark">Up to 200B parameters locally</td></tr>
    <tr><td>Dual-system support</td><td data-label="NVIDIA DGX Spark">Up to 405B-parameter models</td></tr>
  </tbody>
</table>
</div>

In addition to these impressive specifications, Spark's architecture is designed to maximize AI performance within a small desktop system.

## How the DGX Spark Punches Above Its Weight

One of the DGX Spark's most notable features is its 128 GB of coherent unified memory. Unlike most desktop systems that divide memory between the GPU and CPU, Spark allows both processors to access the same memory pool.

For AI development, this unified memory can let the DGX Spark run models up to 200 billion parameters locally while avoiding the quantization or model sharding that smaller GPUs often require to fit large models into limited VRAM.

You can also connect two Sparks over high-speed ConnectX networking. This roughly doubles capacity, allowing you to run models of around 405 billion parameters.

There is an important caveat. The DGX Spark's 273 GB/s memory bandwidth is modest compared with a data-center GPU. In practice, Spark has the memory capacity to fit large models that other desktop hardware cannot, but it is not built to deliver the highest possible inference throughput.

In short, the DGX Spark is optimized to make large models accessible on the desktop, not to replace a high-throughput inference server. That trade-off can be a good fit for prototyping and experimentation.

## Early Challenges and Ongoing Improvements

At launch, Spark received mixed reviews. Early adopters praised its AI capabilities, but reports of latency, throttling, and reboot issues led some shoppers to hesitate before using it for data-intensive work.

NVIDIA has since addressed some of these concerns through software updates. TensorRT-LLM optimizations and speculative decoding have improved performance, with NVIDIA reporting up to a 2.5x throughput increase across a range of benchmarks compared with launch performance.

![NVIDIA DGX Spark performance improvements from software and model optimizations](/uploads/dgx-spark-tensorrt-llm.png)

*Image credit: [NVIDIA](https://developer.nvidia.com/blog/new-software-and-model-optimizations-supercharge-nvidia-dgx-spark/).*

Another concern is its price. Originally announced with a $2,999 starting price, Spark launched in late 2025 at $3,999. In February 2026, NVIDIA raised the price to $4,699, citing global memory supply constraints.

Even at its current price, the DGX Spark remains one of the few desktop systems capable of running such large models locally. Whether it is worth the investment depends on the workload and how often you will use that capability.

## What the DGX Spark Is Built For

Spark is not designed to replace every AI workstation or server. For a mini AI supercomputer, it delivers a lot of power, but not enough for every use case.

The DGX Spark is a strong fit for:

- Prototyping new AI applications or fine-tuning models before deploying them at scale.
- Building agentic AI workflows that need ongoing testing and iteration or need to run continuously.
- Developing robotics, computer vision, and video analytics applications with NVIDIA's Isaac and Metropolis frameworks.

It may not be the best choice for:

- High-volume production inference, where the memory-bandwidth ceiling becomes a real constraint.
- Sustained, large-scale training workloads better suited to multi-GPU servers or cloud clusters.
- Gaming or general-purpose compute, where conventional desktop hardware can offer better performance and value.

## Final Thoughts

The NVIDIA DGX Spark packs a great deal into a compact form factor. With its GB10 Grace Blackwell Superchip, 128 GB of unified memory, and NVIDIA's AI software stack, it gives developers a capable local environment for experimenting with models that would otherwise be difficult to run on a desktop.

If you regularly prototype applications, fine-tune models, or build autonomous agents, Spark can be a capable workstation for local AI development. But not every workload justifies specialized hardware with a significant upfront cost.

On Vast.ai, you can rent high-performance GPUs on demand and pay only for the compute you use. Access hardware such as [RTX 5090s](https://vast.ai/pricing/gpu/RTX-5090), [H200s](https://vast.ai/pricing/gpu/H200), [B200s](https://vast.ai/pricing/gpu/B200), and [B300s](https://vast.ai/pricing/gpu/B300), then scale capacity up or down as your workloads change.

Get started in minutes. [Browse thousands of available GPU instances](https://vast.ai/pricing) on Vast.ai and find the compute you need today.


Explore NVIDIA DGX Spark specifications, real-world performance, limitations, and the AI development workloads it is best suited for.

NVIDIA DGX Spark: Everything You Need to Know


AI agents are significantly more expensive to operate than traditional AI chatbots because they perform multi-step reasoning, use external tools, maintain memory, and often execute several model inference calls for a single request. Understanding what drives these costs, and how to optimize your architecture, can dramatically reduce AI infrastructure spending.

A common scenario for organizations setting up AI agents for the first time is the unexpectedly high bill that arrives. It could even be ten times the anticipated cost. How does that happen?

According to Gartner, agentic models [use](https://www.gartner.com/en/newsroom/press-releases/2026-03-25-gartner-predicts-that-by-2030-performing-inference-on-an-llm-with-1-trillion-parameters-will-cost-genai-providers-over-90-percent-less-than-in-2025) 5-30x more tokens per task than a standard AI chatbot. Tokens are the currency of AI inference: small units of text data. Every time an agent reads data, makes a web search, or runs code, it's consuming tokens - and each token consumed translates directly into API and infrastructure costs.

## Breaking It Down: The Cost Drivers Behind AI Agents

Unlike a chatbot, an AI agent doesn't run only a single inference pass per query. It breaks tasks down into multiple steps, goes through a chain of decisions, interacts with external tools, splits into sub-tasks using specialized sub-agents, loops back to re-evaluate results, and may repeat the process before delivering an answer. Each of those actions has a cost.

### Reasoning Loops Multiply Inference

An AI agent typically operates in cycles of planning, acting, observing, and evaluating - and if it's not confident in its answer, it will often re-plan and try again. A single task can involve numerous model inference calls behind the scenes. Every loop adds more tokens and more compute, with costs racking up quickly - especially if there are no hard limits on retry cycles, resulting in runaway loops repeating over again.

### Context Windows Grow Over Time

Agents re-send context like tool outputs, memory, and previous reasoning to the model on every loop, causing the context window to massively grow in size. Across multi-step workflows, this results in context bloat. Poor prompt design also blows through tokens, as vague instructions force re-planning and re-attempts. At enterprise scale in particular, token consumption can compound drastically.

### Memory and Tool Calls Add Overhead

AI agents retrieve data from databases, make API calls, run web searches, use internal applications, and store long-term memory in vector databases. These operations add expenses beyond inference, as each tool call has its own latency and cost. Plus, if there's no shared memory across parallel agents, redundant calls will occur - another cost killer.

### Maintenance Is an Ongoing Expense

Keeping agents reliable requires ongoing testing, monitoring, optimization, and staying up to date with software and APIs. These operational expenses become even more critical as deployments scale.

Even though agentic workflows are inherently more expensive than chatbots, there are ways to minimize costs at the architecture layer.

## Architecting Around the Cost of AI Agents Without Killing Performance

The following are a few ways to improve agent efficiency:

- **Route by complexity.** Run certain steps in an AI agent's workflow on smaller, cheaper models - such as simple classification, extraction, and routing tasks - and send only the more advanced reasoning steps to a frontier model.
- **Cap your loops.** Set limits on iterations per task: if an agent can't resolve a request within the bounds you define, it should return a partial result rather than loop indefinitely.
- **Cache wherever possible.** Use prefix and prompt caching to store frequently reused context, and state checkpointing to allow agents to pick up from an earlier stage rather than restart an entire reasoning chain from scratch.
- **Trim context.** Employ summarization and selective retrieval techniques to keep context size under control.
- **Use rules before reasoning.** Reserve model inference for tasks that require reasoning, and use deterministic workflows and traditional code for those that don't, like checking thresholds and triggering predefined actions.
- **Fix your prompts.** Create clear, concise prompts to avoid clarification loops, re-planning, and failures that cost tokens unnecessarily.
- **Right-size your hardware.** Use less expensive GPUs for lightweight models, and scale up to more powerful hardware only when required.

But hardware flexibility only works if you have access to a range of machines to use. And right-sizing also depends on how and where you're running your models in the first place.

## GPU Infrastructure: Rethinking Where Inference Runs

When you run agentic workflows through a major model provider's API, you pay token-based pricing, which is generally a fixed amount per million tokens. However, self-hosting open-source and open-weight models on [rented GPU infrastructure](https://vast.ai/products/gpu-cloud) can significantly reduce inference costs: instead of paying per token, you pay for compute time - and you control how efficiently that compute gets used.

This is where Vast.ai fits in, providing agent-ready AI infrastructure across 40+ data centers and 20,000+ GPUs ranging from consumer cards to enterprise-grade hardware like H200s and B300s.

With Vast.ai, you pay only for the compute you use. Billing is metered to the second, with no minimum commitments, so bursty agent workloads remain cost-effective. You can also choose between on-demand instances - well suited for production workloads that need guaranteed uptime - or interruptible instances at a discount, which are a good fit for batch agent tasks or any workflows that can tolerate a restart.

Running your own inference offers other advantages, as well. For instance, you can reduce the effective cost per token by deploying quantized models, which use lower-precision numerical formats to reduce memory requirements and increase throughput. You can also employ KV-cache offloading, which moves an agent's attention-cache data outside GPU memory. This allows for longer context windows and more simultaneous agent sessions, improving utilization.

## The Bottom Line

Smart architecture helps control how many tokens your agents consume. Pairing it with Vast.ai's flexible infrastructure gives you greater control over cost per token, making agentic AI more affordable at scale.

Vast.ai also makes deployment easier. Our [Model Library](https://vast.ai/model-library) includes ready-to-use templates for agentic workflows and other AI applications, so you can spend less time on configuration and more time building.

Ready to get started? Run your [AI agents on Vast.ai](https://vast.ai) today.


AI agents cost more than chatbots because they perform multi-step reasoning, call tools, maintain memory, and often run several model inferences for one request.

Why AI Agents Cost More Than Chatbots (and How to Architect Around It)


NVIDIA has officially unveiled its next-generation AI platform, Rubin - and it's clearly aimed at shifting the economics of large-scale AI.

Bringing together six new chips to slash training time and inference token costs, Rubin "[takes a giant leap toward the next frontier of AI](https://www.nvidia.com/en-us/data-center/technologies/rubin/)," says NVIDIA CEO Jensen Huang.

Here's what we know about the platform - and what's still speculation.

## Rubin Platform: AI's Next Leap Forward

The NVIDIA Rubin platform takes a whole new approach to supercomputing.

The reality today is that always-on, next-generation AI factories have to continuously sustain enormous token throughput across distributed systems - providing the long context required for agentic reasoning, complex workflows, and multimodal pipelines - without slowing down under constraints on power and cost.

Scaling these workloads is all about eliminating bottlenecks. Rubin is purposefully designed for that task. Its **six-chip, rack-scale architecture** is built around the idea that the datacenter itself, rather than a single GPU server, is the unit of compute.

To that end, Rubin uses extreme co-design as a foundational principle. GPUs, CPUs, networking, security, software, power delivery, and cooling are all designed together as a single system instead of being optimized in isolation.

At the chip level, the platform brings together:

1. **NVIDIA Rubin GPU** - The primary AI accelerator, featuring a third-generation Transformer engine with hardware-level adaptive compression.
2. **NVIDIA Vera CPU** - An Arm-based processor with 88 NVIDIA Olympus cores and ultra-fast NVLink-C2C connectivity, built for high-bandwidth data movement and agentic reasoning across accelerated systems.
3. **Sixth-gen NVLink + NVLink Switch** - A high-speed GPU interconnect that unifies massive Mixture-of-Experts (MoE) models and enables faster, more efficient AI inference at scale.
4. **ConnectX-9 SuperNIC** - Advanced networking designed to move data efficiently between nodes and across clusters.
5. **BlueField-4 DPU** - Infrastructure offload and security processing that enables confidential computing and workload isolation at scale.
6. **Spectrum-6 Ethernet** - Next-gen AI networking built to sustain high-throughput, low-latency communication across large deployments.

It's an impressive combination. And in terms of performance, NVIDIA's published figures reflect just how powerful the platform is.

## NVIDIA Rubin GPU Specs and Performance Benchmarks

According to NVIDIA, the Rubin platform delivers up to [50 petaFLOPS](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/) of NVFP4 compute for AI inference. Its high-speed GPU interconnect delivers **3.6 TB/s** per GPU via sixth-gen NVLink - and scales up to **260 TB/s** per rack of 72 Rubin GPUs.

The platform's flagship innovation is the **Vera Rubin NVL72** rack-scale system, where the entire rack serves as one accelerator within a larger AI factory.

Notably, Vera Rubin NVL72 features NVIDIA Confidential Computing - the first rack-scale platform ever to do so. This maintains data security across CPU, GPU, and NVLink domains and protects the world's largest proprietary models, along with the training data and inference workloads that power them.

As context windows grow, so does the need to coordinate state across systems at gigascale. To address this issue, Rubin introduces a new inference context storage layer, with BlueField-4 enabling efficient sharing and reuse of key-value cache data across the infrastructure.

On top of that, the NVIDIA Rubin platform's second-generation RAS (Reliability, Availability, and Serviceability) engine features real-time health checks, fault tolerance, and proactive maintenance. Maximizing system productivity is the goal, right down to tray design. The rack is modular and cable-free, enabling up to **18x faster** assembly and servicing than the Blackwell generation.

## Beyond the Official Specs

Of course, there are always rumors in the run-up to launch. How far will NVIDIA push the envelope?

Unofficial reports suggest that Rubin's TDP may be an astonishing [**2.3 kW** per GPU](https://rcrtech.com/semiconductor-news/nvidia-bumps-vera-rubin-specs/) - up from the previously reported 1.8 kW. At the same time, memory bandwidth will jump to **22 TB/s** per GPU, potentially leveraging more aggressive HBM4 clocks.

That extra 500W of headroom could mean higher sustained boost clocks under continuous load and reduced throttling during long training runs. It could also result in more stable throughput for inference-heavy deployments, as well as higher rack-level performance density.

A power increase of this magnitude would ultimately enable more consistent performance under full stress. At hyperscale, that makes a huge difference.

## Conclusion

The NVIDIA Rubin platform shows what the next generation of AI factories will look like: rack-scale systems optimized for bandwidth, efficiency, and always-on sustained load.

But most teams don't operate their own AI factories. They need flexible and cost-effective access to high-performance GPUs today, across generations, without long-term commitments.

That's what Vast.ai was built for. Get instant access to [H200s](https://vast.ai/rent-h200), [B200s](https://vast.ai/rent-b200), and [B300s](https://vast.ai/pricing/gpu/B300) now, with pay-as-you-go pricing - and scale as your needs evolve.


From the Vera Rubin NVL72 to 22 TB/s memory bandwidth, here's everything we know so far about NVIDIA's next-generation AI supercomputing platform.

NVIDIA Rubin Platform: Everything We Know So Far


As AI systems scale, retrieval can become a major cost center. Agents need more context, retrieval-augmented generation (RAG) systems pull from larger document sets, and vector databases keep growing. As those indexes expand, embedding size starts to matter. Storage increases, memory pressure builds, and retrieval latency rises.

Matryoshka vector embeddings, also known as Matryoshka Representation Learning (MRL), give teams a practical way to control that tradeoff. MRL-trained embeddings can be shortened while preserving useful semantic meaning, so engineers can tune cost, speed, and quality without switching models.

For teams building on Vast.ai, this matters because embedding workloads often run next to other AI inference systems. Smaller vectors can reduce memory pressure, improve retrieval throughput, and make better use of available GPU compute across development, batch processing, and production systems.

## How Matryoshka Representation Learning Works

MRL trains models so each prefix of a vector is useful on its own. Like a Matryoshka doll, where each larger doll contains smaller dolls inside it, a full embedding contains smaller usable representations inside it.

That means an embedding can be generated once, then shortened later. The vector becomes smaller while keeping much of the information needed for semantic comparison.

For example:

- A 768-dimensional embedding can be reduced to 512, 256, 128, or 64 dimensions.
- The smaller vector can still preserve meaningful relationships between items.
- The same model can support multiple retrieval modes with different quality and latency targets.

Instead of treating embedding size as fixed, teams can adjust dimensionality based on what the system needs.

## Why Matryoshka Vector Embeddings Are Useful

In production systems, embedding size directly affects infrastructure cost and user experience. Smaller embeddings mean smaller indexes, less memory movement, and faster similarity calculations.

Key benefits include:

- **Reduced storage:** Smaller vectors create smaller indexes and lower storage overhead.
- **Lower memory usage:** More vectors can fit in RAM or GPU memory.
- **Faster retrieval:** Fewer dimensions reduce the cost of similarity search.
- **Better latency:** Less data moves through the system during search.
- **Flexible tradeoffs:** Teams can adjust quality versus performance without changing models.

The most important idea is control. Instead of choosing between a high-quality model and a cheaper model, engineers can often use one model and choose how much of its representation to use for a specific retrieval stage.

## Who Benefits from Matryoshka Embeddings?

Matryoshka embeddings are most useful for teams dealing with scale, latency, or infrastructure cost constraints.

### RAG and Search Engineers

- Large document collections
- Latency-sensitive retrieval pipelines
- Multi-stage retrieval and reranking systems

### Infrastructure and Platform Engineers

- Vector databases such as FAISS or pgvector
- Memory and compute optimization
- Distributed retrieval systems

### AI Product Teams

- Semantic search
- Recommendation systems
- Multi-environment deployments
- Growing retrieval costs as usage increases

### GPU Operators

- Embedding services running alongside LLM inference
- Throughput optimization per GPU
- VRAM usage reduction

If you are already serving embeddings or rerankers, this pattern fits naturally with existing workflows. For example, Vast.ai has guides for [serving online inference with Text Embeddings Inference](https://vast.ai/article/serving-online-inference-with-text-embeddings-inference-on-vastai) and [serving rerankers using vLLM](https://vast.ai/article/serving-rerankers-on-vastai-using-vllm), both of which are common building blocks in retrieval systems.

## How Matryoshka Vector Embeddings Are Used

The basic workflow is straightforward:

1. Choose a Matryoshka-capable embedding model.
2. Choose the dimensionality you want to use.
3. Encode your documents and queries.
4. Store or compare the truncated embeddings.
5. Optionally use larger embeddings or rerankers when accuracy matters more.

### Step 1: Choose a Matryoshka-Capable Model

Start with a model that supports useful shortened representations.

Common examples include:

- `nomic-embed-text-v1.5`
- `text-embedding-3-small`
- `text-embedding-3-large`

The exact model depends on your application, latency target, quality target, and deployment environment.

### Step 2: Choose Your Dimensionality

Dimensionality controls the size of each vector. Higher dimensions usually preserve more detail. Lower dimensions usually reduce storage, memory usage, and retrieval cost.

For `nomic-embed-text-v1.5`, a common starting point is:

- `768` dimensions for maximum quality
- `512` dimensions for high-quality production use
- `256` dimensions for balanced production use
- `128` or `64` dimensions for low-latency systems

The right choice should be measured against your own retrieval quality and latency requirements.

### Step 3: Encode with Truncation

Here is a minimal Python example using the SentenceTransformers library:

```python
#!/usr/bin/env python

"""
Install the dependency first:

pip install sentence-transformers
"""

from sentence_transformers import SentenceTransformer

documents = [
    "SQLite is a small embedded database stored in a single file.",
    "PostgreSQL is a client-server relational database.",
    "Embeddings map text into vectors for semantic search.",
]

queries = [
    "Which database runs inside my app process?",
    "How do I search by meaning instead of exact words?",
]

model = SentenceTransformer(
    "nomic-ai/nomic-embed-text-v1.5",
    trust_remote_code=True,
)

# Prefix documents with "search_document:" for retrieval tasks.
prefixed_docs = [f"search_document: {doc}" for doc in documents]
doc_vectors = model.encode(
    prefixed_docs,
    normalize_embeddings=True,
    truncate_dim=256,
)

# Prefix queries with "search_query:" for retrieval tasks.
prefixed_queries = [f"search_query: {query}" for query in queries]
query_vectors = model.encode(
    prefixed_queries,
    normalize_embeddings=True,
    truncate_dim=256,
)
```

Important: Matryoshka embeddings can be stored at full size and truncated later without running the model again. If you compare truncated vectors with cosine similarity or normalized dot product, renormalize the truncated vectors first. That is cheap vector math, not re-embedding.

Also note that `nomic-embed-text-v1.5` requires task instruction prefixes. Use `search_document:` for documents and `search_query:` for queries in RAG and retrieval pipelines. Omitting these prefixes can degrade retrieval quality.

### Step 4: Optimize the Retrieval Flow

One common pattern is:

- Retrieve with smaller embeddings for speed.
- Rerank with larger embeddings or a reranker model for accuracy.

This gives users fast first-stage retrieval while preserving higher-quality ranking where it matters.

## Where Matryoshka Embeddings Help Most

Matryoshka embeddings become especially useful once a system moves beyond local experimentation.

At that point, systems are often:

- Memory-bound
- Sensitive to throughput
- Running retrieval alongside other inference workloads
- Serving many users or agents at once

Smaller embeddings reduce the amount of data moving through memory and across the system. That can mean faster queries, more stable performance under load, and fewer memory pressure issues.

They also make experimentation easier. Engineers can test different vector sizes without changing models or rebuilding the entire retrieval pipeline. From a system design perspective, the model and pipeline stay mostly the same. What changes is how much of the representation is used at each stage.

## Glossary of Key Terms

**Embedding:** A way to convert text into numbers so a system can compare meaning.

**Vector dimensionality:** The number of dimensions in the embedding. Higher dimensionality usually means more detail, but also higher cost.

**Truncation:** Cutting down the size of a vector while retaining useful information.

**Vector database:** A system that stores embeddings and supports similarity search.

**Similarity search:** A search method that finds items close in meaning, not just exact keyword matches.

**RAG:** Retrieval-Augmented Generation. A system that retrieves relevant information before generating an answer.

**ANN:** Approximate Nearest Neighbor. A faster way to find similar vectors without comparing every vector exactly.

## Ready to Incorporate Vector Embeddings?

Matryoshka vector embeddings are useful because they make embedding size adjustable. That gives teams a practical way to balance retrieval quality, latency, memory usage, and cost.

If you are building retrieval systems, semantic search, RAG pipelines, or agentic AI workflows, Matryoshka embeddings can help improve performance without adding much system complexity.

If you are ready to incorporate vector embeddings into your AI systems, [get started on Vast today](https://cloud.vast.ai/signup) and deploy the compute power you need in minutes. For production inference workloads that need automated scaling, [Vast.ai Serverless](https://vast.ai/products/serverless) can help route capacity dynamically as demand changes.


Cut AI retrieval costs without sacrificing quality. Learn how Matryoshka vector embeddings let you shrink vector size while preserving meaning and get started on Vast.

Matryoshka Vector Embeddings: Flexible Embeddings for Cost-Efficient AI Systems


Inference is taking over AI infrastructure.

In 2026, inference workloads are [expected](https://www.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions/2026/compute-power-ai.html) to account for roughly two-thirds of all AI compute - up significantly from previous years - while the market for inference-optimized hardware continues to expand at an aggressive pace.

But this general shift from training to inference isn't playing out exactly how you might expect. We're also witnessing an explosion in computational complexity across multiple dimensions, which is reshaping how we think about AI infrastructure.

Let's look at some key trends.

## Inference Is the Newly Dominant AI Workload - with High Compute Demand

Training only has to happen once. Inference is continuous.

That's the simplest explanation for why inference workloads are dominating AI infrastructure today. The conventional wisdom was that this shift would cause total compute demand to stabilize, since inference requires far less compute than training runs.

But the opposite is happening. Total computational demand is actually [rising](https://www.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions/2026/compute-power-ai.html) at four to five times per year out to 2030, far outpacing the efficiency gains from newer AI chips. Three factors explain why.

First, there's the sheer [speed](https://hai.stanford.edu/ai-index/2026-ai-index-report) of AI adoption. It's reached 53% consumer adoption in three years - faster than PCs or the Internet. Enterprise adoption has accelerated even more quickly, hitting 88% this year.

Second, inference itself has become more sophisticated. Post-training techniques like reinforcement learning from human feedback, synthetic data augmentation, and test-time scaling, where models "think" through problems step by step, can [use](https://www.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions/2026/compute-power-ai.html) 30 to 100 times the compute of a simple inference query. In many cases, models now routinely do additional reasoning work during inference to improve accuracy and reduce hallucinations.

Third, modern inference workloads have advanced far beyond simple tasks like summarizing emails. Agentic workflows, multimodal generation, long-context reasoning, and retrieval-augmented generation (RAG) chain multiple operations together. This multiplies computational requirements.

As a result, while inference may be computationally lighter than training on a per-operation basis, the growing scale and complexity of production inference workloads are pushing compute demand to unprecedented levels.

## Inference Infrastructure Is Becoming More Specialized

Production inference environments have different priorities than training infrastructure. Training large foundation models prioritizes massive parallelism and GPU clusters optimized for processing enormous datasets.

Modern inference systems, on the other hand, must balance sometimes competing priorities: low latency, high concurrency, throughput efficiency, memory optimization, request routing, and workload orchestration. And the relative importance of each factor depends heavily on the application itself.

For example, an AI copilot serving millions of users at once needs to minimize response time. A batch inference pipeline doing async processing overnight can trade latency for throughput. Reasoning models running long chains of thought need sustained memory access and compute availability. Each workload has a different optimal hardware profile.

This specialization trend at the hardware level is extending to entire system architectures, as well. Some inference workloads can tolerate interruption and can be distributed across multiple locations. Some cannot, and instead require ultra-low latency in a single facility. Still others benefit from running in geographically dispersed data centers closer to end users.

So we're now seeing AI infrastructure becoming more heterogeneous - spanning hyperscale data centers, enterprise AI systems, edge deployments, and distributed GPU environments. The question now is less about which infrastructure to use and more about which infrastructure to use for which workload.

## Cost Efficiency and GPU Utilization Matter More Than Ever

Organizations can't simply throw more hardware at the problem. Raw compute capacity alone isn't a solution. Efficiency is equally important.

There's increasing pressure to reduce cost per inference, improve GPU utilization, and avoid overprovisioning.

Optimization techniques help. For instance, quantization reduces numerical precision with minimal performance impact; pruning removes non-essential parameters while preserving core model behavior; model distillation creates smaller, more efficient models; and methods like [TurboQuant](https://vast.ai/article/turboquant-explained-llm-memory-inference) enable large models to run on consumer GPUs through key-value (KV) cache compression.

But optimization alone isn't enough, because inference workloads are so variable. Some tasks can run efficiently on mid-range GPUs, whereas others need high-memory accelerators to get the job done even after models are optimized and compressed. Peak demand might require 10x the capacity of average load.

The priority now is flexibility. Organizations increasingly want the flexibility to match workloads to the most cost-effective machines possible. The ability to dynamically access the right compute resources at the right time - rather than pre-provision for worst-case scenarios - is becoming essential.

## The Future of AI Infrastructure Is Hybrid, Distributed, and Flexible

This need for flexibility is reshaping the future of AI infrastructure. There is no single "best" infrastructure model that applies to every situation.

Maintaining enough owned infrastructure to handle peak demand is expensive and inefficient - especially as high-end AI hardware becomes more costly and power-intensive. Organizations need to be able to scale workloads in real time across different environments and hardware tiers as operational needs change.

The future of AI infrastructure isn't exclusively centralized or decentralized. It is both. It's also more diverse, more distributed, and more specialized.

Organizations building cutting-edge AI systems can no longer simply decide which traditional cloud provider to use or which GPUs to purchase. Today there are more nuanced considerations.

For instance, can you access the right hardware for each workload? Can you deploy where your users are? Can you scale specific resources as requirements change? Can you experiment without long-term commitments?

These capabilities deliver genuine competitive advantages - but historically they've been accessible mainly to the largest players.

## AI Infrastructure for the Inference Era

Vast.ai changes the equation.

Rather than locking you into fixed hardware allocations or forcing you to overprovision for peak demand, Vast.ai provides on-demand, affordable access to globally distributed GPU compute.

You can match each workload to the most appropriate hardware configuration - whether that's high-performance accelerators like [H200s](https://vast.ai/pricing/gpu/H200) and [A100s](https://vast.ai/pricing/gpu/A100-SXM4), professional workstations like the [RTX PRO 6000](https://vast.ai/pricing/gpu/RTX-PRO-6000-WS), or consumer GPUs like the [RTX 5090](https://vast.ai/pricing/gpu/RTX-5090). And you only ever pay for the compute you actually use. There's no upfront investment and no overhead.

For production inference workloads, [Vast.ai Serverless](https://vast.ai/products/serverless) goes further by adding predictive autoscaling that automatically adjusts capacity based on demand patterns. This eliminates the need for manual capacity planning and ensures resources are available exactly when needed.

In the age of AI inference, infrastructure shouldn't be your bottleneck. Get started with Vast.ai today.


Explore the AI inference trends shaping 2026 infrastructure, from rising compute demand and specialized hardware to flexible, distributed GPU deployment.

The Future of AI Inference in 2026: Key Trends Shaping AI Infrastructure

Learn how to build a fully functional AI coding agent in under 100 lines of Python using Vast.ai GPU cloud. Covers the agent loop, Ollama setup, tool calling, and SSH tunneling.

How to Build an Agent on Vast.ai in Less Than 100 Lines of Python


If you're operating at the frontier of AI and high-performance computing, you already know that GPU selection can make or break your infrastructure strategy.

NVIDIA's H200 GPU set a new standard for performance when it arrived as Hopper's high-memory successor. But the Blackwell Ultra-based B300 raises an obvious question: is it finally time to upgrade?

Both GPUs are built for serious compute demands. On paper, the B300 is the more capable machine, but that does not automatically make it the better choice. Here is how the B300 and H200 compare side by side, and when Blackwell Ultra is worth the jump.

## NVIDIA H200: Hopper's Finest

The H200 GPU builds on the foundation of the H100, retaining the same Hopper architecture while improving one of the biggest bottlenecks in modern AI workloads: memory.

Equipped with [141 GB of HBM3e memory and 4.8 TB/s of memory bandwidth](https://www.nvidia.com/en-us/data-center/h200/), the H200 increases memory capacity and bandwidth over the H100. These gains matter for memory-intensive workloads such as large language model inference and retrieval-augmented generation.

The H200 is no slouch on throughput, either. In NVIDIA's Llama 2 70B benchmark discussion, H200 systems [reached](https://resources.nvidia.com/en-us-nim/nvidia-hopper-leaps-ahead) more than 31,000 tokens per second.

Some notable H200 features include:

- 141 GB of HBM3e memory at 4.8 TB/s bandwidth.
- Up to 1.9x the LLM inference performance of H100 in NVIDIA's H200 positioning.
- 900 GB/s of fourth-generation NVLink bandwidth per GPU for intra-node communication.
- A practical upgrade path for teams already built around Hopper systems and software.

Even after the launch of Blackwell, the H200 remains one of the most widely adopted GPUs for enterprise AI because it combines high memory capacity, a mature software ecosystem, and a manageable infrastructure profile.

But if your workload is moving into frontier inference, very long context windows, or exascale-style deployments, the B300 is a different class of accelerator.

## NVIDIA B300: The Arrival of Blackwell Ultra

The [Blackwell Ultra B300](https://vast.ai/article/everything-you-need-to-know-about-the-nvidia-blackwell-ultra-b300) is one of NVIDIA's highest-end accelerators for the inference era. It was designed for the surge in agentic AI systems, large-scale reasoning workloads, and models that need more memory than earlier generations can comfortably provide.

The B300's memory capacity gets attention first. Blackwell Ultra supports up to [288 GB of HBM3e memory with up to 8 TB/s of bandwidth](https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/). That is roughly twice the memory capacity of H200. A 70B model in FP16 can require about 140 GB of VRAM for weights alone, so B300 leaves far more room for KV cache, batching, and longer context windows.

Meanwhile, H200's 141 GB can be enough for many 70B workloads, but headroom gets tight when batch sizes, context length, or serving overhead grow. As workload complexity increases, B300's additional memory can help sustain throughput without forcing as much quantization or sharding.

Beyond memory, the B300 features fifth-generation Ultra Tensor Cores with native NVFP4 support. Blackwell Ultra pushes dense low-precision compute to 15 PFLOPS of dense NVFP4 performance, which is a major step up from Hopper's dense FP8 capability.

<div className="article-image-light-frame">
  <img src="/uploads/nvidia-blackwell-ultra-b300-low-precision-compute.png" alt="Dense low precision compute comparison across Hopper, Blackwell, and Blackwell Ultra GPUs" />
</div>

[Image source: NVIDIA](https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/)

However, the B300 also comes with a steep infrastructure profile. NVIDIA lists Blackwell Ultra max power at up to 1,400 W TGP, while [DGX B300 systems are specified at roughly 14 kW system power](https://www.nvidia.com/en-us/data-center/dgx-b300/). That means power delivery, cooling, rack design, and facility planning matter just as much as raw GPU performance.

Standout B300 features include:

- 288 GB of HBM3e memory and 8 TB/s memory bandwidth.
- Fifth-generation Ultra Tensor Cores with native NVFP4 support.
- Fifth-generation NVLink with 1.8 TB/s bandwidth per GPU.
- ConnectX-8 networking in DGX B300 systems for high-speed cluster connectivity.
- A memory and compute profile built for large-scale inference, reasoning, and multi-GPU deployments.

To really understand the tradeoff, it helps to compare the two GPUs directly.

## NVIDIA H200 vs B300: Full Spec Comparison

<div className="article-table-scroll gpu-comparison-table">
<table>
  <thead>
    <tr>
      <th scope="col">Feature</th>
      <th scope="col">H200</th>
      <th scope="col">B300</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Architecture</td>
      <td data-label="H200">Hopper</td>
      <td data-label="B300">Blackwell Ultra</td>
    </tr>
    <tr>
      <td>Memory</td>
      <td data-label="H200">141 GB HBM3e</td>
      <td data-label="B300">288 GB HBM3e</td>
    </tr>
    <tr>
      <td>Memory Bandwidth</td>
      <td data-label="H200">4.8 TB/s</td>
      <td data-label="B300">8 TB/s</td>
    </tr>
    <tr>
      <td>CUDA Cores</td>
      <td data-label="H200">16,896</td>
      <td data-label="B300">20,480+</td>
    </tr>
    <tr>
      <td>Tensor Cores</td>
      <td data-label="H200">528 fourth-gen Tensor Cores</td>
      <td data-label="B300">640 fifth-gen Ultra Tensor Cores</td>
    </tr>
    <tr>
      <td>Transformer Engine</td>
      <td data-label="H200">First generation</td>
      <td data-label="B300">Second generation</td>
    </tr>
    <tr>
      <td>Form Factor</td>
      <td data-label="H200">SXM</td>
      <td data-label="B300">SXM</td>
    </tr>
    <tr>
      <td>Interconnect</td>
      <td data-label="H200">NVLink 4, 900 GB/s per GPU</td>
      <td data-label="B300">NVLink 5, 1.8 TB/s per GPU</td>
    </tr>
    <tr>
      <td>Networking</td>
      <td data-label="H200">ConnectX-7 in DGX H200 systems</td>
      <td data-label="B300">ConnectX-8 in DGX B300 systems</td>
    </tr>
    <tr>
      <td>PCIe Generation</td>
      <td data-label="H200">Gen 5</td>
      <td data-label="B300">Gen 6</td>
    </tr>
    <tr>
      <td>Precision Formats</td>
      <td data-label="H200">FP8, FP16/BF16, TF32, FP32, FP64, INT8</td>
      <td data-label="B300">FP4, FP8, FP16/BF16, TF32, FP32, FP64, INT8</td>
    </tr>
    <tr>
      <td>Multi-Instance GPU</td>
      <td data-label="H200">Up to 7 instances</td>
      <td data-label="B300">Up to 7 instances</td>
    </tr>
    <tr>
      <td>Max Power</td>
      <td data-label="H200">Up to 700 W TGP</td>
      <td data-label="B300">Up to 1,400 W TGP</td>
    </tr>
    <tr>
      <td>Cooling Profile</td>
      <td data-label="H200">Air or liquid-cooled systems</td>
      <td data-label="B300">Purpose-built high-density systems</td>
    </tr>
  </tbody>
</table>
</div>

The B300 may look like the obvious winner. But real-world requirements are a whole other problem.

## H200 vs B300 Use Cases: Which GPU Is Right for Your Workload?

Whether you already have H200s or you're deciding between H200 and B300 instances, the right GPU depends on your workload, your infrastructure assumptions, and how quickly your memory needs are growing.

### The H200 is likely the right choice if you:

- Need a proven, high-performance GPU for enterprise-scale AI workloads without moving into the highest-density Blackwell Ultra tier.
- Are working with memory-bound workloads where 141 GB gives you enough headroom, including many 70B inference and long-context RAG deployments.
- Want to upgrade from existing Hopper deployments without rethinking your full power and cooling footprint.
- Have power, budget, or availability constraints but still need strong performance per dollar.

### The B300 may be worth the upgrade if you:

- Need to run 70B+ parameter models with more KV cache and batching room on each GPU.
- Need to serve large reasoning models where long context windows would overwhelm H200 memory.
- Need maximum inference throughput and scalability for high-volume or frontier workloads.
- Want to prepare for the next generation of agentic AI and larger multimodal model deployments.

## The Bottom Line

The NVIDIA H200 and B300 are both serious datacenter GPUs. The H200 delivers proven Hopper performance with a mature deployment path and a more manageable infrastructure profile. It remains a strong choice for many production AI teams.

The Blackwell Ultra B300 is in a higher tier for memory capacity, low-precision inference, and scaling headroom. But its power draw and system requirements are also higher, so the upgrade only makes sense when your workload can use that extra memory and compute.

For most teams, the decision comes down to more than benchmark numbers. Top-tier GPUs are expensive to buy, install, power, cool, and operate. That is where Vast.ai helps.

Whether you need H200s for enterprise AI inference, B300s for frontier model workloads, or other NVIDIA GPUs for development and scaling, Vast.ai gives you on-demand access without the upfront hardware burden.

[Compare current GPU pricing on Vast.ai](https://vast.ai/pricing), or explore available [B300](https://vast.ai/pricing/gpu/B300) and [H200](https://vast.ai/pricing/gpu/H200) instances to find the right fit for your workload.


Compare NVIDIA B300 vs H200: specs, memory, power requirements, and real-world use cases to find out if Blackwell Ultra is worth the upgrade for your AI workloads.

NVIDIA B300 vs. H200: Is Blackwell Ultra Worth the Upgrade?


June's updates bring NVIDIA B200 and B300 Blackwell Ultra GPU supply to the marketplace, plus new templates, guides, and platform improvements.

## NVIDIA B200 and B300 GPUs Now Available

Fresh GPU supply is live on Vast.ai, with the powerful NVIDIA B300 now available. Built on the Blackwell Ultra architecture, the B300 features 288GB of HBM3e memory, 8 TB/s of memory bandwidth, and 20,480 CUDA cores - designed for large-scale inference, reasoning models, distributed training, and any workloads that keep running out of VRAM. See up-to-date [B300 pricing](https://vast.ai/pricing/gpu/B300), and rent by the hour or reserve long-term.

We've also added hundreds of B200 GPUs to the marketplace. Rent individual instances or use [Vast.ai Serverless](https://vast.ai/products/serverless) to automatically scale workloads without managing infrastructure yourself.

## Platform Updates: Serverless, Billing, and Security Fixes

We've continued our platform updates with several improvements this month. Serverless metrics reporting has been revamped with increased granularity and new metrics, and serverless UI performance has been significantly improved to accommodate high instance count endpoints.

In addition, when you create a machine report, that machine now stays out of your search results for longer, so your results stay more relevant to you.

On the fixes side: we've improved support for manual invoices on billing transaction history exports, fixed some SSH key issues with key editing and stale data, addressed OAuth login and two-factor authentication (2FA) issues, and made a number of security improvements.

## New Templates

This month, we've added a template for [Qwen3.6 35B A3B](https://vast.ai/model/qwen36-35b-a3b). It's the first open-weight model in the Qwen3.6 series, combining a hybrid Gated DeltaNet and Gated Attention architecture with sparse Mixture-of-Experts (MoE) routing and a vision encoder optimized for agentic coding, frontend workflows, and repository-level reasoning.

Serverless templates for Qwen3.6 and Gemma 4 are now available. The Qwen3.6 template supports a 35B-total, 3B-active model.

We've also updated the vLLM template to include TurboQuant support and made some maintenance updates to the ComfyUI template.

## Our Commitment

We remain committed to keeping high-performance AI infrastructure accessible and affordable for everyone. With fresh B200 and B300 GPU supply live on Vast.ai, we're continuing to expand the resources available to our users while making it easier to build, train, fine-tune, and deploy AI workloads at scale.

Need help? Reach out anytime at [support@vast.ai](mailto:support@vast.ai) or join our [Discord](https://discord.gg/vast) server to connect with the community and keep up with our latest updates.

## Change Log

### New Features

- NVIDIA B200 and B300 GPUs are now available.
- Creating a machine report now removes the machine from your results for a longer duration.
- Revamp of serverless metrics reporting: increased granularity and new metrics.
- Significant improvement in the performance of serverless UI to accommodate high instance count endpoints.
- Billing page UI improvements.
- Search page performance improvements.
- B300 support.

### Issues Resolved

- Improved support for manual invoices on billing transaction history export.
- Fixed instance SSH key issues with key editing and stale data.
- OAuth login issues.
- Various 2FA fixes.
- Security improvements.

### New Templates

- [Qwen3.6 35B A3B](https://vast.ai/model/qwen36-35b-a3b): the first open-weight model in the Qwen3.6 series, built on direct community feedback and focused on stability and real-world utility. It combines a hybrid Gated DeltaNet and Gated Attention architecture with sparse Mixture-of-Experts routing and a vision encoder for unified multimodal reasoning.
- Serverless templates for Qwen3.6 and Gemma 4.

### Updated Templates

- vLLM template: TurboQuant support.
- ComfyUI maintenance.

### New Guides

- Step-by-step instructions for moving workloads from [Salad Cloud to Vast.ai](https://docs.vast.ai/examples/migrations/salad-to-vast), including instance setup and cost comparison.
- How to generate [video with LTX-2.3 and ComfyUI](https://docs.vast.ai/examples/video-generation/ltx-video-2-3) on Vast.ai.
- A practical guide to fine-tuning open-source LLMs on Vast.ai GPU instances using the [Axolotl](https://docs.vast.ai/examples/ai-ml-frameworks/axolotl-fine-tuning) framework.
- Deploy and run [NVIDIA Nemotron 3 Super](https://docs.vast.ai/examples/text-generation/nemotron-3-super) on Vast.ai for high-performance inference.

### Top Blog Posts

- [All-in-One App Studio: Your Complete Creative AI Toolkit in a Single Container](https://vast.ai/article/all-in-one-app-studio)
- [How Much Money Can You Earn Renting Out Your GPU on Vast.ai?](https://vast.ai/article/how-much-money-can-you-earn-renting-out-your-gpu-on-vast-ai)
- [TurboQuant Explained: How It Reduces LLM Memory by 5x and Speeds Up Inference](https://vast.ai/article/turboquant-explained-llm-memory-inference)
- [How to Use Unsloth Studio for Fast and Affordable AI Fine-Tuning](https://vast.ai/article/how-to-use-unsloth-studio-for-fast-affordable-ai-fine-tuning)


Vast.ai's June 2026 product update brings NVIDIA B200 and B300 Blackwell Ultra GPU supply to the marketplace, plus new templates, guides, and platform improvements.

June 2026 Product Update


What makes a powerful GPU? For years, GPU generations were often judged by how much faster they could train AI models. But today's AI workloads are increasingly dominated by inference, where memory capacity, latency, and efficiency matter as much as raw compute.

The [NVIDIA Blackwell Ultra B300](https://vast.ai/pricing/gpu/B300) reflects that reality. Designed to tackle the explosive growth of agentic AI and the most complex reasoning workloads, it enables larger models, longer context windows, and higher throughput than any single GPU before it.

Here's everything you need to know about the B300.

## What Is the NVIDIA Blackwell Ultra B300?

Available through partner systems beginning in the second half of 2025, the Blackwell Ultra B300 is one of the most powerful GPUs in NVIDIA's lineup. It features an enormous 288 GB of HBM3e memory. That's 50% more than the B200, about twice the capacity of the H200, and 3.6x that of the H100.

<div className="article-image-light-frame">
  <img src="/uploads/nvidia-blackwell-ultra-b300-memory-comparison.png" alt="NVIDIA Blackwell Ultra B300 memory capacity comparison" />
</div>

[Image source: NVIDIA](https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/)

Compared to Hopper and standard Blackwell, the Blackwell Ultra also delivers substantial gains in dense low-precision compute performance.

<div className="article-image-light-frame">
  <img src="/uploads/nvidia-blackwell-ultra-b300-low-precision-compute.png" alt="B300 vs B200 vs H200 vs H100 dense low precision compute comparison" />
</div>

[Image source: NVIDIA](https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/)

For reference, each B300 GPU features:

- 288 GB of HBM3e memory
- 160 streaming multiprocessors
- 8 TB/s memory bandwidth
- 15 PFLOPS dense FP4 performance
- 7 PFLOPS dense FP8 performance
- 3.5 PFLOPS dense FP16 performance
- 5th-generation Ultra Tensor Cores
- 20,480+ CUDA cores
- 1.8 TB/s 5th-gen NVLink bandwidth
- NVIDIA ConnectX-8 networking
- 1,400W TDP

In addition, the B300's Blackwell Ultra architecture incorporates several features designed for modern AI inference. These include fifth-generation Tensor Cores with native NVFP4 support, a second-gen Transformer engine, a dual-die design linked by a 10 TB/s on-package interface, and faster PCIe Gen 6 and NVLink interconnects for scaling across GPUs.

But the memory capacity is the big news when it comes to the NVIDIA B300.

## Why the B300's Memory Matters

Memory capacity increasingly determines what AI workloads can run efficiently. The massive 288 GB of memory attached to each B300 GPU is particularly valuable for frontier models, retrieval-augmented generation (RAG) systems, and trillion-parameter mixture-of-experts (MoE) models that require substantial memory for weights, activations, and key-value (KV) caches.

As context windows grow longer and reasoning models become more sophisticated, memory often becomes the limiting factor before compute does. When KV caches can't fit on-device, they spill onto external memory tiers like system RAM or NVMe storage, increasing latency and degrading throughput.

However, a single B300 can fit a full 70B-parameter model in FP16 with plenty of room remaining for KV cache storage and batching - something that previous-generation GPUs could typically only achieve through quantization and model sharding. The result is lower latency variability and higher sustained throughput as workload complexity increases.

This advantage compounds in multi-GPU systems. The B300 is the module that powers the NVIDIA DGX B300, a server combining eight B300 GPUs into a single system with over 2 TB of total GPU memory. More on-device memory per GPU means there's less need for complex model parallelism, which reduces the cross-GPU communication overhead that can constrain performance at scale.

With the DGX B300, organizations can run 400B+ parameter frontier models entirely within GPU memory. But as impressive as it is, the B300 does have some drawbacks.

## The Hidden Cost of B300 Performance

The NVIDIA Blackwell Ultra B300 represents a generational leap in memory and compute, but it's also a generational leap in infrastructure demand.

Each B300 GPU can draw between 1,100W and 1,400W of power depending on the chassis form factor, with a TDP of 1,400W. That's 40% higher than the B200's 1,000W. Not to mention, the eight-GPU DGX B300 draws about 14 kW under peak load.

At this level, liquid cooling is mandatory and has real implications for total cost of ownership.

Consider that a single B300 GPU costs around $50,000, and a fully configured DGX B300 costs in the range of $300,000 to $500,000 - before even factoring in the hidden cost of liquid cooling infrastructure, power delivery, and/or facility upgrades.

Granted, the B300 is energy-efficient per FLOP: according to NVIDIA, the rack-scale GB300 NVL72 [boasts](https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/) 5x throughput per megawatt versus Hopper-generation systems. But efficiency gains don't necessarily make up for the 2x higher power draw and greater rack-level density as deployment scale increases.

The bottom line is that cooling and power delivery must be addressed before you can even think about taking advantage of the B300's performance. Another consideration: when does it make sense to use the B300?

## What Workloads the B300 Is Built For

The B300 is particularly well suited for use cases such as:

- Large model inference with 70B+ parameters, with reduced reliance on quantization or sharding across multiple GPUs.
- Reasoning models with large KV cache requirements, allowing for higher output quality and lower latency while maintaining longer context windows.
- High-throughput inference APIs where the FP4 compute advantage means using fewer GPUs at a given request volume.
- Multi-node training environments where the high inter-node bandwidth eases communication bottlenecks.

These workloads are where the B300's performance can actually justify the infrastructure investment. But on-prem deployment isn't for everyone.

## Accessing the B300 Without the Infrastructure Burden

For many teams, the NVIDIA Blackwell Ultra B300 is the right hardware at the wrong price. The operational overhead of running these flagship datacenter-grade GPUs can be prohibitive for all but the largest organizations with the deepest pockets.

That's where Vast.ai comes in. Rather than owning and operating the hardware yourself, you get on-demand access to B300 GPUs from a globally distributed fleet at a fraction of the cost, with flexible configurations ranging from one to eight B300s. And if you prefer not to manage GPU instances at all, [Vast.ai Serverless](https://vast.ai/products/serverless) can automatically orchestrate the provisioning of GPU workers to match your dynamic workloads in real time.

Leverage Blackwell Ultra's massive memory capacity and impressive FP4 inference performance without the upfront capital expense - and experiment and scale faster while paying only for the compute you actually use.

Ready to get started? [Explore available B300 instances on Vast.ai today](https://vast.ai/pricing/gpu/B300).


The NVIDIA B300 features 288 GB HBM3e memory - 50% more than the B200. Here's what that means for inference, reasoning models, and cost.

Everything You Need to Know About the NVIDIA Blackwell Ultra B300


With AI came the technology to transform entire industries, but the infrastructure to run it was suddenly constrained. Hyperscalers secured much of the available advanced GPU capacity for their own needs, leaving many startups, research labs, and smaller enterprises scrambling for compute.

Demand for GPUs surged so high that pricing quickly became prohibitively expensive. In early 2024, renting four NVIDIA H100-powered instances from a hyperscaler for just one month could [cost](https://journal.uptimeinstitute.com/neoclouds-a-cost-effective-ai-infrastructure-alternative/) nearly $300,000.

This opened up a gap in the market, and into it stepped an entirely new type of cloud provider: the [neocloud](https://vast.ai/hosting/neoclouds).

## What Is a Neocloud? Definition and Core Models

Born out of GPU scarcity and cost barriers, as well as hyperscaler allocation bottlenecks, neoclouds are AI-first cloud infrastructure providers that specialize in GPU-as-a-Service (GPUaaS) and high-performance computing (HPC) workloads. Neoclouds serve a market that's [projected](https://www.abiresearch.com/blog/leading-neocloud-companies) to grow from $42B last year to $250B+ by 2030.

A neocloud is an AI-first cloud infrastructure provider that specializes in GPU compute for machine learning training and inference workloads. Unlike traditional hyperscalers, neoclouds are purpose-built for AI, offering faster provisioning, transparent pricing, and configurations optimized for high-performance computing at significantly lower cost.

Neoclouds are an emerging type of AI-first cloud infrastructure provider that emphasizes GPUs, faster networking, and optimized storage for the intensive demands of generative AI workloads. Inference, training, and other machine learning algorithms require more advanced infrastructure to keep their data pipelines filled. Otherwise, resources become idle and turnaround time suffers.

Unlike hyperscalers, which offer general-purpose infrastructure for a wide variety of workloads, neoclouds typically focus on AI. Many provide GPU compute through simpler pricing models. Another advantage: faster provisioning and configurations that are optimized for AI training and inference.

But not all neoclouds are built the same way. Here's a look at the three business models that neoclouds operate under.

### 1. Own Infrastructure

Larger neoclouds like CoreWeave and Nscale build AI infrastructure in their own data centers. This requires massive capital investment in the millions to billions of dollars and involves depreciation risk for aging hardware, but it also offers full-stack control. Neoclouds that invest in their own infrastructure can provide earlier access to next-gen GPUs and support larger-scale enterprise deployments.

### 2. Colocation Strategy

Under the colocation model, neoclouds take a hybrid approach. They partner with data centers to lease established infrastructure, reducing upfront capital expenditure. High-density, GPU-optimized colocation environments allow neoclouds to scale quickly without building facilities from scratch.

### 3. Asset-Light Marketplace

With this approach, neoclouds like Vast.ai aggregate GPU capacity without owning infrastructure. The marketplace model connects distributed compute providers with customers who need flexible, cost-effective access without long-term commitments, while the platform itself acts as the coordination layer.

## Neocloud Benefits for Enterprise AI: Cost, Speed, and Scale

No matter their business model, neoclouds offer numerous benefits to organizations working with AI.

First and foremost, better pricing. Because they run leaner and have less overhead, neoclouds can [deliver](https://www.mckinsey.com/capabilities/tech-and-ai/our-insights/the-evolution-of-neoclouds-and-their-next-moves) up to 85% cost savings compared to hyperscaler GPU instances. They also monetize differently. Rather than complex, layered billing, neoclouds tend to offer transparent, per-second pricing where you only pay for what you use.

Second, specialized infrastructure. Neoclouds are purpose-built for AI, so they can deliver faster provisioning with infrastructure optimized specifically for model training and inference.

Third, greater elasticity and scalability. AI workloads can be unpredictable. Neoclouds give organizations the ability to scale GPU resources up or down dynamically, so they get exactly what they need when they need it and avoid overprovisioning. Some neoclouds, like Vast.ai, even support [serverless autoscaling](https://vast.ai/article/vast-ai-serverless-automated-gpu-scaling) for workloads with variable demand.

Finally, better support for AI-specific compute requirements. AI workloads demand high-bandwidth interconnects, large memory pools, and massive parallel processing capacity. Because neoclouds are built around these requirements, they're often better suited for GPU-intensive workloads than general-purpose cloud environments.

Neoclouds like Vast.ai are also built to meet enterprise-grade security and compliance needs, so organizations don't have to choose between affordability and peace of mind.

However, not all neoclouds operate under the same economic or operational constraints.

## The Limitations of Traditional GPU Cloud Providers

Traditional neoclouds that own or lease infrastructure face a few major challenges:

- **Rapid price erosion:** Each new GPU generation decreases the value of older chips, with the price of a GPU hour [declining](https://www.mckinsey.com/capabilities/tech-and-ai/our-insights/the-evolution-of-neoclouds-and-their-next-moves) by up to 50% within five years.
- **Thin margins:** After depreciation, power, and labor costs, gross profit margins are often quite slim.
- **Difficulty differentiating:** To stay competitive, neoclouds may offer more than just raw compute, but developing higher-margin AI services can put them in direct competition with hyperscalers.

These pressures mean that traditional neoclouds have to maintain high utilization rates and constantly reinvest in new hardware while also navigating how to differentiate their services successfully.

The marketplace model, on the other hand, avoids these challenges.

## How GPU Compute Marketplaces Solve What Traditional Neoclouds Can't

By aggregating distributed GPU capacity rather than owning hardware, marketplace-based neoclouds eliminate the millions in upfront investment that would otherwise be needed, and never have to manage an aging fleet of depreciating hardware. There's also often little to no direct competition for the massive enterprise contracts that infrastructure owners depend on.

This means that the marketplace model is able to provide the same advantages as other neoclouds, including flexibility, elasticity, and on-demand access, while offering the most competitive rates possible.

One important byproduct of this approach is democratized compute. Organizations that can't afford higher rates can still leverage the latest hardware and pursue AI development.

### Neocloud vs. Hyperscaler: Key Differences

<div className="article-table-scroll neocloud-comparison-table">
  <table aria-label="Neocloud vs. hyperscaler key differences">
    <thead>
      <tr>
        <th>Neocloud</th>
        <th>Hyperscaler</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>GPU first</td>
        <td>CPUs first, with GPUs as add-ons</td>
      </tr>
      <tr>
        <td>Built for fast networking speed natively</td>
        <td>Mixed networking speeds</td>
      </tr>
      <tr>
        <td>Geared toward bare metal or thin VMs</td>
        <td>Virtualization overhead</td>
      </tr>
      <tr>
        <td>Simplified pricing</td>
        <td>Complex a la carte pricing</td>
      </tr>
      <tr>
        <td>Quick provisioning</td>
        <td>DIY provisioning or managed services</td>
      </tr>
    </tbody>
  </table>
</div>

## The Bottom Line

Both traditional neoclouds and hyperscalers carry overhead that drives up costs. Marketplace-based neoclouds like Vast.ai offer a better path forward, delivering the GPU compute your AI and HPC workloads need at prices that make experimentation and innovation possible for teams of any size.

For teams evaluating [GPU cloud](https://vast.ai/products/gpu-cloud) providers, from scrappy AI startups to enterprise ML teams, understanding the neocloud landscape is the first step toward smarter infrastructure decisions. Stop overpaying for GPU compute. Start building on Vast.ai today.


Neoclouds are AI-first GPU cloud providers built for machine learning workloads. Learn how they work, the three business models, and how they compare to hyperscalers.

What Is a Neocloud? The Business Model Explained


The NVIDIA GeForce RTX 5090 launched in January 2025 – and over a year later, it's still surprisingly difficult to get your hands on one.

Despite retail prices officially starting at $1,999 MSRP, limited availability and strong demand from both gamers and AI developers means stock sells out quickly and real-world pricing has pushed higher.

As NVIDIA's most powerful consumer GPU, the RTX 5090 is undoubtedly an impressive machine. Is it worth the steep price tag, though? That depends on more than just raw power – let's take a look.

### RTX 5090: The Consumer Powerhouse

For starters, here are some of the official specs for the RTX 5090:

-   **Architecture:** Powered by NVIDIA's cutting-edge Blackwell architecture.
-   **Gaming Chip:** Featuring the next-gen GB202 gaming chip.
-   **Memory:** 32GB of GDDR7 memory with a blazing-fast 1,792GB/sec bandwidth, supported by a super-wide 512-bit memory interface for even faster data transfer.
-   **CUDA Cores:** 21,760 cores for high-performance graphics workloads, complementing the 680 Tensor cores that accelerate AI inference and training.
-   **L2 Cache:** Expanded to 98MB, up from 73MB on the RTX 4090, improving memory efficiency and reducing latency.
-   **Display Support:** PCIe Gen 5 with DisplayPort 2.1b connectors supporting up to 8K at 165Hz.
-   **Power Requirements:** 575W total graphics power (TGP), with a 1000W power supply recommended for optimal performance.

Another bonus: the RTX 5090 has a redesigned **Founders Edition** with a three-slot design and a dual flow-through fan configuration – so it's more compatible with tighter builds, a remarkable shift from its predecessors.

For reference, here's a quick overview of the RTX 5090:

<table>
    <thead>
        <tr>
            <th>Spec</th>
            <th>RTX 5090</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><strong>CUDA Cores</strong></td>
            <td>21,760</td>
        </tr>
        <tr>
            <td><strong>Ray Tracing Cores</strong></td>
            <td>170 (4th-gen)</td>
        </tr>
        <tr>
            <td><strong>Tensor Cores</strong></td>
            <td>680 (5th-gen)</td>
        </tr>
        <tr>
            <td><strong>Base Clock</strong></td>
            <td>2,017 MHz</td>
        </tr>
        <tr>
            <td><strong>Boost Clock</strong></td>
            <td>2,407 MHz</td>
        </tr>
        <tr>
            <td><strong>L2 Cache</strong></td>
            <td>98,304KB</td>
        </tr>
        <tr>
            <td><strong>ROPs</strong></td>
            <td>176</td>
        </tr>
        <tr>
            <td><strong>VRAM</strong></td>
            <td>32GB 28Gbps GDDR7</td>
        </tr>
        <tr>
            <td><strong>VRAM Interface</strong></td>
            <td>512-bit</td>
        </tr>
        <tr>
            <td><strong>Memory Bandwidth</strong></td>
            <td>1,792 GB/s</td>
        </tr>
        <tr>
            <td><strong>Interface</strong></td>
            <td>16x PCIe 5.0</td>
        </tr>
        <tr>
            <td><strong>Power Connectors</strong></td>
            <td>1 x 16-pin (PCIe 5.0 / ATX 3.0) native connector; includes 4 x 8-pin adapter</td>
        </tr>
        <tr>
            <td><strong>Total Graphics Power (TGP)</strong></td>
            <td>575W</td>
        </tr>
    </tbody>
</table>

(For more specifications – and to see how the RTX 5090 stacks up against its younger siblings in the RTX 50-series lineup as well as the RTX 4090 – check out the [chart at the bottom of this page](#additional-specifications).)

#### RTX 5090: AI and Professional Workloads

The RTX 5090 is well-suited for high-performance AI and professional workloads. Its **680 Tensor cores** and **32GB of GDDR7** memory efficiently accelerate deep learning, 3D rendering, and other compute-intensive tasks. With **98MB of L2 cache** and a **512-bit memory interface**, the RTX 5090 delivers faster data access and reduced latency – crucial for AI training, scientific computing, and content creation.

The **5th-gen Tensor cores** boost AI inference efficiency in frameworks like TensorFlow and PyTorch, while **4th-gen RT cores** enhance real-time ray tracing for animation, visual effects, CAD work, and more.

For demanding AI and video workloads, the RTX 5090 more than meets the challenge. In Procyon's AI XL (FP16) test, the RTX 5090 was **40% faster** than the RTX 4090, and in PugetBench's DaVinci Resolve video processing test, it was **12% faster**.

#### Next-Gen Gaming Features

The biggest leap in technology for this GPU comes from DLSS 4's new **Multi Frame Generation**, an AI-driven frame interpolation technique exclusive to the RTX 50 series. It uses AI to generate up to three additional frames between traditionally rendered ones, significantly boosting frame rates with minimal visual artifacts.

Further, the RTX 5090 introduces a range of AI-driven enhancements in what's called **neural rendering**, designed to push in-game realism to new levels. **RTX Neural Shaders** improve texture compression and deliver film-quality lighting and shading in real time, while **RTX Neural Faces** uses generative AI to create lifelike skin, hair, and facial details.

Building on the power of DLSS 4, these neural rendering techniques integrate directly into the graphics pipeline, leveraging NVIDIA’s Tensor cores to enhance visual fidelity. As game support continues to expand, these technologies demonstrate the potential for deeper shadows, realistic translucency in materials, and an overall richer environment.

Another breakthrough is **Mega Geometry**, a feature that uses RT cores to significantly boost the number of triangles in ray-traced scenes. This allows game engines to maintain full geometric detail without sacrificing performance, resulting in more realistic object depth, shadows, and fine details.

The above features position the RTX 5090 as a powerhouse for next-generation gaming visuals... but there's a catch.

##### Caveat: Performance Claims vs. Real-World Results

The way that NVIDIA presents its performance numbers can be misleading. The company leans heavily on Multi Frame Generation-boosted FPS (frames per second) to compare these new GPUs against older ones.

Frame Generation and Multi Frame Generation (MFG) are powerful tools, but they aren't universal fixes. Their impact varies by game – some titles see massive FPS gains with little visual trade-off, while others may introduce artifacts or latency.

In addition, **not all games will support these features**, as DLSS adoption depends on individual developers. A GPU's raw performance still matters. MFG-inflated benchmarks can make performance gains seem more significant, but Frame Generation still needs a strong base frame rate to be effective. In CPU-limited games, or titles that don't support DLSS 4, the real-world performance gap may be far smaller.

Ultimately, while the RTX 5090 excels at AI-driven rendering, **real-world results will depend on game support and how well these features are implemented.**

### Worldwide RAM Shortage and Price Increases

Due to explosive growth in AI data centers and tech giants continuing to invest heavily in AI accelerators, there's an ongoing worldwide RAM shortage that shows no signs of slowing down. Some forecasts <a href="https://www.theverge.com/ai-artificial-intelligence/914672/the-ram-shortage-could-last-years" target="_blank" rel="noopener noreferrer">predict</a> that shortages could last until 2030.

It's no surprise that high demand and limited supply lead to rising prices. With the RTX 5090, though, there may be another factor at play. DeepSeek's rapid rise in popularity has sparked a huge <a href="https://www.theinformation.com/articles/deepseek-spurs-black-market-demand-for-nvidia-gaming-chips-in-china" target="_blank" rel="noopener noreferrer">surge</a> in demand for NVIDIA's gaming GPUs in China, as these cards can be repurposed to run DeepSeek's AI models – and provide a workaround for U.S. export restrictions on advanced GPUs.

See our guide on [how to run DeepSeek R1 on a GPU cloud](https://vast.ai/article/deepseek-r1-open-source-disruptor-or-overhyped-upstart).

With a global shortage of RAM and both gamers and AI developers vying for the same hardware, the strain on supply could push RTX 5090 prices even higher in the near future. Most buyers currently face inflated market prices ranging from $2,500 to over $4,000. And with NVIDIA set to <a href="https://wccftech.com/nvidia-passes-another-300-hit-to-rtx-5090-board-partners-due-to-vram-shortage/" target="_blank" rel="noopener noreferrer">increase prices</a> further on account of VRAM sourcing costs, those numbers could skyrocket to over $5,000.

### Final Thoughts: The Power of the RTX 5090 – Without the Hassle

The RTX 5090 remains the most powerful consumer GPU available today. For cutting-edge 4K gaming with DLSS 4 and Multi Frame Generation, as well as demanding AI and professional workloads, it delivers unmatched performance in a relatively accessible package.

At the same time, it may only be worth purchasing if you really want the best of the best and you're willing to shell out for it. The high price tag makes it a significant investment for the average user.

In the meantime, why wait? Skip the delays and high costs – [rent an RTX 5090](https://cloud.vast.ai/?gpu_option=RTX%205090) on Vast.ai for instant access to top-tier performance. Given its limited availability and its 575-watt power draw, our affordable rental options are the most practical way to access the RTX 5090's full potential without the upfront investment.

[Try the 5090 GPU](https://cloud.vast.ai/?gpu_option=RTX%205090) for yourself today.

<span id="additional-specifications"></span>

#### Additional Specifications:

<div className="article-table-scroll">
<table className="w-full border-collapse border border-gray-300">
    <thead>
        <tr className="bg-gray-100">
            <th className="border border-gray-300 px-4 py-2 text-left font-semibold">
                Spec
            </th>
            <th className="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 5090
            </th>
            <th className="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 5080
            </th>
            <th className="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 5070 Ti
            </th>
            <th className="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 5070
            </th>
            <th className="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 4090
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">GB202</td>
            <td className="border border-gray-300 px-4 py-2 text-center">GB203</td>
            <td className="border border-gray-300 px-4 py-2 text-center">GB203</td>
            <td className="border border-gray-300 px-4 py-2 text-center">GB205</td>
            <td className="border border-gray-300 px-4 py-2 text-center">AD102</td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">VRAM</td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                32GB GDDR7
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                16GB GDDR7
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                16GB GDDR7
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                12GB GDDR7
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                24GB GDDR6X
            </td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                VRAM Speed
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">28Gbps</td>
            <td className="border border-gray-300 px-4 py-2 text-center">30Gbps</td>
            <td className="border border-gray-300 px-4 py-2 text-center">28Gbps</td>
            <td className="border border-gray-300 px-4 py-2 text-center">28Gbps</td>
            <td className="border border-gray-300 px-4 py-2 text-center">21Gbps</td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                VRAM Bus Width
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">512</td>
            <td className="border border-gray-300 px-4 py-2 text-center">256</td>
            <td className="border border-gray-300 px-4 py-2 text-center">256</td>
            <td className="border border-gray-300 px-4 py-2 text-center">192</td>
            <td className="border border-gray-300 px-4 py-2 text-center">384</td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Bandwidth
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                1792GB/s
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                960GB/s
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                896GB/s
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                672GB/s
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                1.01TB/s
            </td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Streaming Multiprocessors (SMs)
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">170</td>
            <td className="border border-gray-300 px-4 py-2 text-center">84</td>
            <td className="border border-gray-300 px-4 py-2 text-center">70</td>
            <td className="border border-gray-300 px-4 py-2 text-center">48</td>
            <td className="border border-gray-300 px-4 py-2 text-center"></td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Cores
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">680</td>
            <td className="border border-gray-300 px-4 py-2 text-center">336</td>
            <td className="border border-gray-300 px-4 py-2 text-center">280</td>
            <td className="border border-gray-300 px-4 py-2 text-center">192</td>
            <td className="border border-gray-300 px-4 py-2 text-center">512</td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Ray Tracing Cores
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">170</td>
            <td className="border border-gray-300 px-4 py-2 text-center">84</td>
            <td className="border border-gray-300 px-4 py-2 text-center">70</td>
            <td className="border border-gray-300 px-4 py-2 text-center">48</td>
            <td className="border border-gray-300 px-4 py-2 text-center">128</td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Boost Clock
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                2407 MHz
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                2617 MHz
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                2452 MHz
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                2512 MHz
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                2520 MHz
            </td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Total Graphics Power (TGP)
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">575W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">360W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">300W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">250W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">450W</td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Recommended Power Supply
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">1000W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">850W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">750W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">650W</td>
            <td className="border border-gray-300 px-4 py-2 text-center">850W</td>
        </tr>
        <tr className="bg-gray-50">
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Launch Date
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                Jan. 30, 2025
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                Jan. 30, 2025
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                Feb. 20, 2025
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                Mar. 5, 2025
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">
                Sept. 20, 2022
            </td>
        </tr>
        <tr>
            <td className="border border-gray-300 px-4 py-2 font-semibold">
                Launch Price
            </td>
            <td className="border border-gray-300 px-4 py-2 text-center">$1,999</td>
            <td className="border border-gray-300 px-4 py-2 text-center">$999</td>
            <td className="border border-gray-300 px-4 py-2 text-center">$749</td>
            <td className="border border-gray-300 px-4 py-2 text-center">$549</td>
            <td className="border border-gray-300 px-4 py-2 text-center">$1,599</td>
        </tr>
    </tbody>
</table>
</div>

Post originally written March 7, 2025, updated May 21, 2026.


NVIDIA’s RTX 5090 remains hard to find as AI and gaming demand push prices higher. Review the specs, performance features, and rental options on Vast.ai.

NVIDIA GeForce RTX 5090: Everything You Need to Know


120,000+ AI developers are actively searching for compute. Your hardware is what they're looking for.

Whether you're running a single RTX 4090 in your home office or managing a datacenter full of H100s, your idle compute could be generating revenue for you right now.

So, how much money can you actually make renting out GPUs on Vast.ai? The short answer: it varies.

## What Hosts Are Earning on Vast.ai

Your earnings depend on your specific hardware, uptime, reliability, rental type, and how your prices compare to similar listings. That said, as a host, here's a snapshot of what you might expect:

- **Consumer GPUs** like the RTX 5090 typically earn $0.30-$0.60 per GPU-hour, depending on your rental type and your reliability score. This score is a measure of uptime and connection stability. A four-GPU 5090 rig running at 80% utilization can generate $700-$1,400 per month. Use the [host earnings calculator](https://vast.ai/hosting/calculator) to estimate your specific hardware.
- **Datacenter hardware** commands much higher rates. H100s and H200s bring in $2.15-$4.00+ per GPU-hour. If you operate a GPU farm, you can turn dedicated infrastructure into a profitable compute business. Vast.ai even offers financing and will match you with enterprise clients looking for reserved compute, so you can presell capacity and have paying customers from day one.

For current market rates across the Vast.ai platform, check the public [pricing page](https://vast.ai/pricing). Live GPU prices are typically about 25% above what hosts earn.

Once you log in and register at least one machine with Vast.ai, your host dashboard will show detailed market stats with occupancy rates and more, allowing you to optimize your pricing strategy and better estimate potential earnings.

## How Vast.ai Works for Hosts

From hobbyists at home to datacenters and GPU farms, Vast.ai supports every scale. This flexibility is what makes the platform work for hundreds of providers already earning steady revenue.

Many GPU rental platforms lock you into contracts and dictate pricing structures. Some only work with datacenters, requiring certifications and minimum machine counts before you can even list.

Vast.ai is different. You set your rates, select your rental terms, and decide exactly how much bandwidth and storage to offer, whether you're running one GPU or one hundred. *Note: hosting requires a separate account from your client account; a single account cannot be used for both.*

To accommodate your needs, Vast.ai supports three rental models, each designed for different workloads and revenue strategies.

### 1. On-Demand Instances: Premium Pricing for Guaranteed Access

On-demand is your highest-earning option. Set a fixed hourly rate and an offer end date. When a client rents your machine, those terms lock into a rental contract that cannot be changed -- not by modifying the offer, not by unlisting the machine, not by any other action. Once a rental contract is created, its terms -- price, end date, hardware specs -- are fixed. Unlisting your machine prevents new rentals but does not affect any active contracts. You must keep the machine available through every contract's rental end date.

This option attracts clients who are willing to pay a premium for reliability and guaranteed access, but you don't have to pick just one rental type. You can maximize utilization by offering interruptible capacity alongside on-demand instances.

### 2. Interruptible Instances: Lower Rates, Higher Flexibility

Interruptibles work in a bidding system. As the host, you set a minimum interruptible price for a client to rent your machine. Clients set a bid price for their instance; the current highest bid is the instance that runs, with lower bids paused. On-demand rentals always take priority over all interruptible instances regardless of bid price. Higher bids instantly preempt lower ones, and on-demand rentals always take priority.

Interruptible instances appeal to clients who are willing to trade guaranteed uptime for 50-80% cost savings. You'll earn less per hour with interruptible instances vs. on-demand, but they fill gaps in time that would otherwise generate nothing, so most hosts offer both.

### 3. Reserved Instances: Long-Term Contracts at Discounted Rates

Reserved instances let clients secure long-term rentals at a reduced rate. If you choose to offer reserved instances, you set your maximum possible discount off your normal on-demand price. That discount scales with rental duration -- longer commitments unlock larger discounts, up to your set maximum. You can also set this value to zero to opt out of reserved pricing entirely.

The reserved model works well for predictable, stable revenue if you're willing to trade some margin for higher occupancy. Customers commit upfront to weeks or months of utilization. That means you spend less time managing short-term rentals.

## How to Stand Out as a Host and Maximize Revenue

Once you're listed, a number of strategies can help you stand out and make more money:

- **Price competitively at launch.** New hosts often list their hardware at slightly below market rates to attract initial rentals and build a good reliability score faster. You can always adjust pricing later.
- **Offer reserved instance discounts.** Long-term contracts at discounted rates encourage more stable utilization and recurring revenue.
- **Keep machines online.** Strong uptime maintains your reliability score, which reflects connection stability. Note that reliability and verification are tracked independently; reliability drops don't affect your verification status. To avoid disruption, plan maintenance around active rental contracts. For unplanned maintenance, use the CLI's `schedule maint` command, which notifies affected clients to save their work.
- **Get and stay verified.** Vast.ai's automated verification process evaluates your machine's network stability, operational health, and performance. Verification runs on a randomized schedule. High-end hardware like H100s and 8x4090 rigs are typically tested fastest. Verified machines rank higher in search results and attract more rentals.
- **Apply for data center certification.** If you operate professionally managed infrastructure, you can become a [Certified Data Center Partner](https://docs.vast.ai/host/datacenter-status), giving you the blue trust label, priority placement, [Secure Cloud](https://vast.ai/compliance) classification for enterprise compliance, and increased customer traffic.

## Getting Started as a Host on Vast.ai

Becoming a host on Vast.ai is more straightforward than you might think. Setup complexity varies by hardware. The [Host Setup guide](https://cloud.vast.ai/host/setup/) walks you through Ubuntu installation, NVIDIA drivers, network port configuration, and other requirements. It can take as little as an hour.

But you don't need to figure everything out alone. Vast.ai maintains active host-only [Discord](https://discord.gg/vast) channels where providers share troubleshooting advice, setup guidance, and optimization tips. Get peer support from operators who've already solved the problems you might encounter.

Once your machines are online, hosting becomes less hands-on. Simply maintain your hardware and let the earnings roll in. Or you can take a more active approach and gradually optimize pricing, experiment with different rental models, and expand into multi-GPU systems or dedicated GPU infrastructure over time.

The choice is yours.

## The Bottom Line

On Vast.ai, hosts get a level of operational control and pricing transparency that's rare in the GPU cloud market. Whether you're hosting one GPU or an entire cluster, the platform is designed to let you monetize and grow on your own terms.

Here's what that means for you: 120,000+ developers are actively searching for compute on Vast.ai. That demand continues to grow, and it's waiting for your hardware. The only question is whether you're ready to start earning.

Want to put your hardware to work? Check [current GPU Pricing Rates](https://vast.ai/pricing), explore the [Host Setup guide](https://cloud.vast.ai/host/setup/), or jump straight in and list your first machine today.


Learn how much you can earn renting GPUs on Vast.ai - from RTX 4090s to H100s. Compare rental types, pricing strategies, and how to maximize your hosting revenue.

How Much Money Can You Earn Renting Out Your GPU on Vast.ai?


As AI demand grows, models are expected to perform increasingly complex tasks with pinpoint accuracy, often leaving founders and engineers scratching their heads. Hard-pressed to find ways to improve their systems, fine-tuning provides a novel solution. With tweaks and training, a model can be taught to produce high accuracy, consistent responses for all manner of specialized tasks. This is where tools such as Unsloth come in.

## Why use Unsloth Studio?

Unsloth, an open-source, code-driven framework for training and running AI models, released its no-code/low-code web UI, [Unsloth Studio](https://vast.ai/article/unsloth-studio-now-available-on-vast-ai), in early 2026. It's a convenient solution that makes fine-tuning a breeze, simplifying training, inference, data, and deployment.

Unsloth Studio currently supports over 500 models, training them 2x faster than traditional fine-tuning methods while requiring 70% less VRAM, all with no accuracy loss. It also runs, trains, and exports GGUF and safetensors models locally on Mac, Windows, and Linux. More efficient VRAM usage, faster training speeds, and local model support mean reduced computational needs and, as a result, cloud costs.

## Getting started with Unsloth Studio

The general process required to run and train your models in Unsloth Studio is straightforward:

1. Install Unsloth Studio
2. Choose a base model
3. Create or upload a dataset
4. Configure fine-tuning settings
5. Start training
6. Test your fine-tuned model
7. Export the model

For your convenience, we've built a [comprehensive guide](https://docs.vast.ai/examples/ai-ml-frameworks/unsloth-studio) that includes all of the information you'll need to get Unsloth Studio set up on your Vast.ai GPU instance. Be sure to use our pre-configured [Unsloth Studio template](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Unsloth%20Studio) in conjunction with the guide to make setup even easier.

## Fine-tuning at a glance

### The basics

Fine-tuning uses datasets to improve pre-trained models in key ways:

- **Update and learn:** Incorporate and learn domain-specific knowledge.
- **Customize behavior:** Engineers can adjust AI model tone, personality, or response style.
- **Tailor models for specific tasks:** Enhance accuracy and relevance for specialized workflows.

When compared to Retrieval-Augmented Generation (RAG), fine-tuning replicates all of RAG's capabilities, but not vice versa. There is a common misconception that fine-tuning doesn't perform as well as RAG and that it can't make a model learn new knowledge. This is untrue. A specialized model can train with fine-tuning and reinforcement learning (RL) while RAG only changes what the model sees at inference time.

### How does fine-tuning achieve memory savings?

Unsloth Studio adds two thin matrices to model weights and optimizes those, rather than changing all weights. This means that only a small fraction of weights, often well under 5%, depending on the LoRA rank and which layers are targeted, have to be optimized and quantization from 16-bit (LoRA) to 4-bit (QLoRA) saves 75% memory.

<div className="article-image-light-frame">
  <img
    src="/uploads/unsloth-studio-lora-memory-savings.png"
    alt="Diagram showing a model weight block and low-rank adapter matrices A and B"
  />
</div>

Image source: [Unsloth fine-tuning guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide)

### Choosing your model and method

When starting, it is recommended to use instruct models which require less data than base models. They also allow for direct fine-tuning using convenient conversational chat templates (ChatML, ShareGPT, etc.).

Regarding model training, you will need to choose between multiple methods:

- **Reinforcement learning (RL):** This method is most advantageous for training models to excel at specific behaviors using an environment and reward function instead of labeled data.
- **LoRA:** A parameter efficient training technique that trains a small set of low-rank adapter weights with 16-bit precision.
- **QLoRA:** This method augments LoRA with 4-bit precision to more efficiently handle large models.
- **Fine-tuning:** The Unsloth framework supports full fine-tuning (FFT) and pretraining, though both demand substantially more resources. If configured correctly, LoRA can match the gains of FFT.

Unsloth recommends QLoRA to start, as it is both accessible and highly effective at training models.

### Creating your datasets

Formatted and organized datasets are crucial for model fine-tuning success. Datasets should generally contain two columns, question and answer, and the data must be able to be tokenized.

Data can be generated synthetically and structured using an LLM. Unsloth Studio also provides a free Synthetic Dataset notebook that can parse documents, including PDFs and videos, generate QA pairs, and auto-clean data.

Generally, you should build a well-structured dataset rather than dumping data alone. This ensures learning, understanding, and response accuracy. Keep in mind, though, that if you're fine-tuning for code, dumping all of your code data can indeed bring significant performance improvements. Be sure to carefully consider your use case when building your fine-tuning strategy.

### AI training hyperparameters

Hyperparameters are essentially the settings when tuning AI models. They control how Low-Rank Adaptation fine-tunes LLMs and can be quite involved. Therefore, Unsloth provides a comprehensive [guide to hyperparameters](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide) for your convenience.

The key hyperparameter choices one needs to consider are as follows:

- **Learning rate:** Controls how aggressively the model adjusts its weights during training.
- **Epochs:** Specifies how many complete passes the model makes through the training dataset.
- **LoRA or QLoRA:** LoRA fine-tunes models using 16-bit precision, while QLoRA uses 4-bit quantization to reduce VRAM usage and hardware requirements.

## Ready to start improving your models?

We hope that this fine-tuning breakdown has given you the confidence to train and customize your AI models with Unsloth Studio. At Vast.ai, our mission goes beyond providing high-performance cloud GPU infrastructure and support; we also strive to educate so that engineers and founders have the knowledge and resources they need to create, build, and scale effectively.

For additional information on how to fine-tune your LLM with Unsloth Studio, check out Unsloth's [fine-tuning guide](https://unsloth.ai/docs/get-started/fine-tuning-llms-guide).

## Resources

- [Unsloth Studio Documentation](https://unsloth.ai/docs/new/studio)
- [Unsloth GitHub](https://github.com/unslothai/unsloth)
- [Vast.ai Instance Guide](https://docs.vast.ai/documentation/instances/overview)


Learn how Unsloth Studio helps teams fine-tune AI models faster, use less VRAM, and lower cloud GPU costs on Vast.ai.

How to Use Unsloth Studio for Fast and Affordable AI Fine-Tuning


Creative AI workflows can get pretty complex.

Let's say your project requires image and video generation with 3D modeling and animation, plus you want to include a voiceover track and music in the background. Or maybe you're fine-tuning a custom language model and immediately testing it through an API while generating supporting visuals in [ComfyUI](https://vast.ai/article/getting-started-with-comfy-UI).

Normally, projects like these require juggling multiple GPU instances for different AI workflows, which can get complicated and expensive fast.

Vast.ai's [All-in-One App Studio](https://cloud.vast.ai/template/readme/1d77da4218e4f86b1d2d0fe755d9b953) offers a unified solution that's elegantly simple: you get a complete creative AI studio with eight powerful applications, a GPU-accelerated remote desktop with KDE Plasma, and Blender, all in a single container.

## Everything You Need in One GPU Instance

The All-in-One App Studio template lets you spin up a single GPU instance to generate images and videos, create music, synthesize voices, transcribe audio, train LoRAs, fine-tune and serve LLMs, and work inside a full desktop environment with 3D rendering abilities.

The template includes:

- ComfyUI for node-based image and video workflows.
- SD Forge for Stable Diffusion-based image generation.
- Wan2GP for advanced video generation accessible even to lower-VRAM machines.
- ACE Step 1.5 for commercial-grade AI music generation with optional vocals.
- Voicebox for text-to-speech generation, voice cloning, and audio editing.
- Whisper WebUI for transcribing audio and video files and generating subtitles.
- Ostris AI Toolkit for LoRAs and model training.
- Unsloth Studio for fine-tuning and serving LLMs.
- A full KDE Plasma remote desktop environment with Blender support.

## Built for Multimodal AI Workflows

With the All-in-One App Studio template, everything works together seamlessly. You can make the most of your creative pipelines without constantly spinning up and shutting down separate environments.

ComfyUI and [SD Forge](https://cloud.vast.ai/template/readme/94b6e3bbbc8a968fef0bae6ced1e39f2) share the same model directories, so you can download a checkpoint once and use it in both tools. After you train a LoRA in the AI Toolkit, immediately test it in ComfyUI or Forge through a shared model library.

Fine-tune a language model in Unsloth Studio and then instantly interact with it through the chat interface, or connect tools like Open WebUI and SillyTavern via the built-in OpenAI-compatible API. There's no separate deployment step required.

The desktop service provides a full KDE Plasma remote desktop streamed to your browser. Blender comes fully configured with GPU-accelerated Cycles rendering, so you can create and render 3D scenes on powerful cloud hardware. You also get Google Chrome, LibreOffice, VLC media player, and a complete desktop environment for GUI-based workflows.

Having all of these tools in a single container significantly reduces setup headaches and makes experimentation so much easier.

## Flexible Workloads Without Wasting VRAM

Start what you need, stop what you don't. That's how All-in-One App Studio handles GPU resources. Nothing auto-starts.

Instead, the template's Instance Portal has a Supervisor tab where you click "Start" next to the applications you want to use at any given time. If you want to work in ComfyUI, start it up with a click. Need the desktop for Blender? Start that, too. Done with video generation? Stop Wan2GP and free up VRAM for another task.

If you want more automation, you can set specific services to auto-start on boot or run your own provisioning script to configure your workspace with custom models or software.

The template is adaptable to your budget, as well. Lower-VRAM GPUs with 8-12 GB can comfortably handle lighter workloads like transcription or voice synthesis, while 24-48 GB instances are better suited for FLUX image generation, LLM fine-tuning, training, and video workflows.

## Getting Started with All-in-One App Studio

It only takes a few minutes to get up and running with the [All-in-One App Studio template](https://cloud.vast.ai/template/readme/1d77da4218e4f86b1d2d0fe755d9b953). [Launch an instance](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=All-in-One%20App%20Studio) with the VRAM you need via the Vast.ai console: 24 GB or more for most workflows, or 48 GB or more if you're planning to run multiple applications simultaneously.

Then open the Instance Portal, navigate to the Supervisor tab, and start up the apps you want to use. It's that easy.

One instance. Eight creative AI tools. A full desktop with Blender. Endless possibilities.

Try All-in-One App Studio on Vast.ai today.


Launch eight creative AI tools, a GPU-accelerated KDE desktop, and Blender in one Vast.ai container with the All-in-One App Studio template.

All-in-One App Studio: Your Complete Creative AI Toolkit in a Single Container


This month's updates bring new templates and guides, major upgrades to two-factor authentication, a benchmarking tool to help you pick the right GPU for your workload, and a complete creative AI studio that runs in a single container.

## NVIDIA Cloud GPU Updates

We've introduced several platform-level improvements this month. Two-factor authentication (2FA) now supports authenticator apps like Google Authenticator and 1Password instead of relying solely on SMS. Additional upgrades include CLI 2FA support, automatic backup codes for account recovery, multiple authentication methods per account, and 2FA for team roles and API key permission groups.

> **Note:** Action is required if you use legacy SMS 2FA. Simply log in normally, go to the Settings page, and click "Regenerate" on the backup codes section to migrate to the new 2FA in less than a minute.

For developers, the Vast [pip package](https://pypi.org/project/vastai/) has been upgraded, and the SDK and CLI repositories are now merged into a single, easy-to-grab repo. We've also introduced an agent skill for coding assistants like Claude and Codex to drive our CLI more effectively.

## Highlighted Feature: Benchmark Before You Deploy

Choosing the right production GPU is now faster and easier, without any guesswork involved. Vast's [benchmarking CLI](https://docs.vast.ai/guides/serverless/choosing-gpus) will take your workload and test it on H100s, A100s, 5090s, 4090s, or any selection you define to compare performance and cost efficiency across GPU classes.

The CLI rents one instance of each GPU type you specify, or a default set for a reasonable starting point if you don't specify any, and then runs the template's built-in benchmark workload, reports performance per dollar, and tears down each rental automatically when finished. Multiple GPU types can be tested in parallel, making it much faster to evaluate your options before committing to longer rentals or production traffic.

Because each run uses real instances and consumes account credit, you get measured performance and cost per hour on actual hardware, not estimates. This benchmarking tool is built for the moment you're staring at a pricing page, wondering whether an H100 is actually worth 4x the cost of a 5090. Sometimes it is, and sometimes it isn't. Now you can find out before you deploy.

## New Templates and Guides

We've rolled out another batch of templates and guides to support creative workflows, fine-tuning, experimentation, and multimodal reasoning.

The standout addition this month is the **All-in-One App Studio** template. It packs eight creative AI applications and a GPU-accelerated remote desktop with KDE Plasma and Blender into a single container, giving you a complete AI production environment without having to manage separate instances.

### New Templates

- [All-in-One Creative AI Studio](https://cloud.vast.ai/template/readme/1d77da4218e4f86b1d2d0fe755d9b953) - One GPU instance: image generation, video, music, voice, transcription, LoRA training, LLM fine-tuning, and KDE desktop with Blender, each activated only when you need it.
- [Unsloth Studio](https://cloud.vast.ai/template/readme/6444356291eabeb423440bb946ccc197) - No-code UI for running and fine-tuning 500+ open-source models with 70% less VRAM.
- [Kimi K2.6](https://vast.ai/model/kimi-k2.6) - 1T-parameter MoE, 32B active, multimodal agentic model with 256K context.
- [Qwen3.5 27B](https://vast.ai/model/qwen35-27b) and [Qwen 3.5B-A3B](https://vast.ai/model/qwen36-35b-a3b) - MoE vision-language models with unified multimodal reasoning and 256K context.
- [Gemma 4 26B-A4B](https://vast.ai/model/gemma-4-26b-a4b-it) and [Gemma 31B IT](https://vast.ai/model/gemma-4-31b-it) - Google's vision-language models with thinking mode and 256K context.
- [Autoresearch](https://docs.vast.ai/examples/ai-agents/autoresearch) - Let Claude Code run ML experiments for you, now in a [template](https://cloud.vast.ai/?ref_id=146798&creator_id=146798&name=Autoresearcher%20%28Claude%20Code%29).

### New Guides

- [Unsloth Studio Guide](https://docs.vast.ai/examples/ai-ml-frameworks/unsloth-studio)
- [CLI Hello World](https://docs.vast.ai/cli/hello-world)
- [SDK Hello World](https://docs.vast.ai/sdk/python/quickstart)

## Other Improvements

This update also brings fleet-wide mitigations for the recently disclosed CopyFail exploit. No action is required from users, and running workloads remain unaffected.

## Our Commitment

As other clouds are out of GPU capacity or charging a premium for what's left, we remain focused on making high-performance AI infrastructure available, affordable, and easy to deploy at scale.

Need help? Contact us anytime at [support@vast.ai](mailto:support@vast.ai) or join our [Discord](https://discord.gg/vast) server for tips, community support, and platform updates in real time.

## Change Log

### New Features

- [Benchmarking CLI](https://docs.vast.ai/guides/serverless/choosing-gpus) for GPU performance and cost comparison.
- [All-in-One App Studio](https://cloud.vast.ai/template/readme/1d77da4218e4f86b1d2d0fe755d9b953): eight creative AI tools in one GPU instance.
- Authenticator app support: use any TOTP app, including Google Authenticator, 1Password, and Microsoft Authenticator, instead of SMS.
- CLI 2FA: 2FA now works through the Vast CLI.
- Backup codes: automatically generate one-time recovery codes so you're never locked out.
- Multiple authentication methods: register more than one per account.
- 2FA for team roles and API key permission groups.

Set up and enable 2FA in [Settings](https://cloud.vast.ai/settings/) -> [Full setup guide](https://docs.vast.ai/guides/reference/account-settings#two-factor-authentication).

### Issues Resolved

- CopyFail exploit: fleet-wide mitigations rolled out with no impact on running workloads and no user action required.
- SSH key input validation, error message, and link fixes.
- Cloud copy fixes.
- Mobile UI fixes.
- Notifications fixes.
- General proactive security fixes.

### API Changes

- The Vast [pip package](https://pypi.org/project/vastai/) has been upgraded, and the SDK and CLI repos have been merged into a single, easy-to-grab repo. Use: `pip install vastai`.
- Install our agent skill to help Claude, Codex, etc., drive our CLI: `npx skills add vast-ai/vast-cli`.

### Updated Templates

- Wan2GP - migrated from provisioning installer to Docker.
- Whisper UI - migrated from provisioning to Docker image.
- FluxGym - Dockerized to prevent provisioning failures.
- vLLM - updated build actions for new CU130 image tag.

### New Blog Posts

- [Deploy LLM Inference Using Vast.ai Serverless](https://vast.ai/article/deploy-llm-inference-using-vast-ai-serverless)
- [Mistral Small 4 Just Dropped - Run It on Affordable H200s with Vast.ai](https://vast.ai/article/mistral-small-4-just-dropped-run-it-on-affordable-h200s-with-vast-ai)
- [Unsloth Studio Is Now Available on Vast.ai](https://vast.ai/article/unsloth-studio-now-available-on-vast-ai)


Vast.ai's May 2026 product update includes authenticator app support for 2FA, a benchmarking CLI for GPU selection, new creative AI templates, and platform security improvements.

May 2026 Product Update


## What Is TurboQuant?

TurboQuant has arrived, and already it has decisively bent the trajectory of LLM inference research. Savings of over 5x in memory requirements and faster token generation speed? Yes, please.

When [Google introduced TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) to the wider world by highlighting it on its research blog, it set the LLM community ablaze almost overnight. The claims seemed almost too good to be true. Memory-chip stocks sold off. Researchers began shipping patches and prototype implementations within days. This time, the promise was real. TurboQuant is not just another clever optimization paper. It has redrawn the practical landscape of transformer inference by attacking one of its most expensive pressure points: the KV cache.

For anyone renting GPUs and trying to ship something that actually works, that is the part that matters. This is no longer theoretical hand-waving. [The paper](https://arxiv.org/abs/2504.19874) has been public since April 2025, but Google brought it to a much wider audience in March 2026, and the idea has already spread into the local-inference stack, including [llama.cpp discussions](https://github.com/ggml-org/llama.cpp/discussions/20969), [vLLM feature work](https://github.com/vllm-project/vllm/pull/39890), and [experimental Ollama-adjacent efforts](https://github.com/ollama/ollama/issues/15051).

## Why Long-Context Inference Gets Expensive

People talk about AI cost as if it were mostly a compute problem. Sometimes it is. But for long-context inference, the uglier constraint is often GPU memory.

The culprit is the KV cache. During prefill and generation, the model stores keys and values from earlier tokens so it can attend back to prior context efficiently instead of recomputing everything from scratch. That is what makes long prompts, chat history, and retrieval-heavy workflows feasible. It is also what makes VRAM usage climb as context grows. [Google's writeup](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) explicitly frames KV cache as a central memory bottleneck for LLM inference, and the [paper](https://arxiv.org/abs/2504.19874) reports TurboQuant results specifically for KV-cache quantization.

That matters because once the KV cache swells, the consequences are brutal and immediate. VRAM headroom disappears, concurrency drops, deployment options narrow, and if you push too far, the job can fail outright. You may need a higher-VRAM card than the model weights alone would suggest. You may have to cut concurrency. You may even end up in a multi-GPU setup just to hold context, which complicates setup and can be a latency footgun. This is why long-context jobs can feel disproportionately expensive: the model is not merely "doing more thinking." It is dragging around a bigger and bigger memory footprint until VRAM pressure starts dictating what hardware is viable at all.

## Cracking the Memory Bottleneck

[Google describes TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) as an extreme compression method that can reduce KV-cache storage to roughly the 3-bit range while preserving quality in the reported benchmarks and accelerating attention-logit computation. The [paper](https://arxiv.org/abs/2504.19874) states the result more precisely, reporting absolute quality neutrality at about 3.5 bits per channel and only marginal degradation at about 2.5 bits per channel for KV-cache quantization. It also says the method is suitable for online applications, which is exactly what you want here. The cache is not a file you compress once and forget about. It sits in the hot path of inference.

That "3-bit" headline is why people noticed, but the deeper point is more practical. TurboQuant is not just trying to make the cache smaller on paper. It is trying to keep it small enough that your GPU spends less time moving KV data around and less of its VRAM budget just holding onto context. The market reaction made for drama, but the implementer's version is simpler: more context on the same card, or more headroom on the card you already use.

## Why "Just Quantize It" Is Not Enough

It is easy to say "compress the cache and save memory." It is harder to do that without giving back the win somewhere else. Low-bit schemes can lose fidelity, and some of them rely on enough extra metadata, scaling factors, or correction machinery that the real savings look less impressive once everything is counted. The TurboQuant paper calls this out directly by focusing not only on mean-squared error, but also on inner-product distortion, which matters because attention depends on those inner products.

There is also a runtime angle. A compression scheme is not useful here if it saves VRAM by making generation slower. If encoding, decoding, or correction is too expensive, you have not removed a bottleneck. You have moved it. Google's blog is notable because it does not present TurboQuant as a pure "memory down, speed down a little" trade. It reports substantial speedups in attention-logit computation, which strongly suggests the compressed representation is cheap enough to use in practice.

## What TurboQuant Does Differently

At a high level, TurboQuant is a two-stage scheme. The first stage rewrites the vector into a form that is easier to compress efficiently. Google's public explanation leans on the PolarQuant framing here. The important point is not to turn the article into a geometry lecture. It is that the representation is rearranged so that low-bit quantization behaves unusually well and avoids some of the per-block overhead conventional methods carry around.

The second stage uses a Quantized Johnson-Lindenstrauss transform, or QJL, on the residual. This is where the method gets more interesting than a generic "make the numbers smaller" story. The paper's point is that a quantizer can look decent on ordinary reconstruction error and still distort the inner products on which attention relies. TurboQuant spends a tiny additional correction budget to preserve those inner products more faithfully. In plain English, it is not just compressing the cache. It is compressing it in a way that tries not to confuse attention.

## Does It Reduce the "Math Tax"?

People sometimes describe TurboQuant as reducing generation cost by moving a math tax into prefill. That is not quite wrong, but it is not quite right either. A better description is that TurboQuant pays a small cost when KV entries are created so later attention steps have less data to move around. That includes prefill, but it does not stop there, because new KV entries are also created during generation. The win is not really that generating tokens no longer has to do the math; it is that generation keeps touching a smaller working set. In practice, the more important tax is often memory bandwidth. Google's published results support exactly that framing: they report attention-logit speedups, which points to reduced memory traffic and a friendlier working set rather than a mere reshuffling of arithmetic.

TurboQuant does not make the model attend to fewer tokens. It makes each attention step cheaper by keeping the KV cache compact enough that the GPU spends less time hauling memory around.

## Why Cheap Decompression Matters

A scheme like this only helps if the compressed cache is cheap to use. If the decoding or correction path is expensive, you can save memory and still make generation slower. That is why generic file-compression instincts do not really apply here. Inference is not trying to pack data for archival storage; it is trying to keep a hot data structure small and fast enough to touch constantly.

That is why TurboQuant is more interesting than a raw compression ratio would suggest. The published results indicate the overhead is low enough for the overall attention path to speed up rather than slow down. Saving VRAM is nice, but saving VRAM without sabotaging decoding speed is actually what changes system behavior. Google's blog reports up to 8x speedup in attention-logit computation on H100 for 4-bit TurboQuant, exactly the kind of result people notice.

## What It Means to You on Vast.ai

If you are renting GPUs on Vast.ai, you already know the real tradeoffs: VRAM, throughput, latency, context length, and hourly price. TurboQuant matters because it changes one of the nastiest constraints in that equation.

When these published gains carry over to your stack, a given workload may fit on a smaller card than before. A long-context job that used to push you toward a high-VRAM premium instance might stay viable on something cheaper. You may also be able to preserve more context or support more concurrent requests before memory becomes the thing that ruins your day. [Google reports](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) up to 6x KV-cache compression and substantial attention-logit speedups, making it possible to run larger models on the same hardware without breaking the bank.

## Bottom Line

TurboQuant matters because it attacks one of the fastest-growing costs in long-context inference: the KV cache. If that cost can be cut this aggressively without wrecking quality or slowing generation, then a lot of assumptions about what hardware a workload needs start to break. For people renting GPUs, this is not a side effect. This is **leverage**.

## Useful Resources

- [TurboQuant paper on arXiv](https://arxiv.org/abs/2504.19874)
- [Google Research: TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/)
- [llama.cpp TurboQuant KV-cache discussion](https://github.com/ggml-org/llama.cpp/discussions/20969)
- [vLLM TurboQuant KV-cache PR](https://github.com/vllm-project/vllm/pull/39890)
- [Ollama TurboQuant/RotorQuant issue](https://github.com/ollama/ollama/issues/15051)
- [Reuters coverage of AI memory-chip demand](https://www.investing.com/news/stock-market-news/samsung-flags-eightfold-jump-in-q1-profit-as-ai-chip-demand-drives-up-prices-4599550)


TurboQuant reduces LLM KV-cache memory pressure while speeding attention-logit computation, changing the economics of long-context inference on rented GPUs.

TurboQuant Explained: How It Reduces LLM Memory by 5x and Speeds Up Inference


Teams running inference-heavy workloads almost always run into the same problem: the high cost of compute. They're often forced to pay for capacity they're not using, or struggle to keep up when demand spikes.

What teams really need is scalable LLM deployment without expensive hyperscaler pricing or overprovisioned infrastructure that sits idle between peak loads.

That's where [Vast.ai Serverless](https://vast.ai/products/serverless) comes in. It's a new way to run AI inference on GPUs with automated scaling that solves three problems traditional cloud providers often exacerbate.

## Why Traditional GPU Clouds Aren't Ideal for LLM Inference

LLM inference workloads are fundamentally different from training jobs. They're bursty and latency-sensitive, so they amplify traditional GPU infrastructure challenges, such as:

- Unpredictable costs when compute usage varies but capacity must be provisioned ahead of time.
- Laggy cold starts when scaling up to meet spikes in demand.
- Infrastructure overhead from manual capacity planning, instance management, and vendor lock-in.

These limitations make it difficult to run LLM inference efficiently at scale. However, Vast.ai Serverless solves this problem by routing workloads to the most cost-effective hardware and scaling compute automatically. In doing so, it eliminates the need to manage infrastructure altogether.

## What Makes Vast.ai Serverless Different

The following are a few ways Vast.ai Serverless stands out:

### Predictive Optimization That Anticipates Demand

Instead of reacting to spikes in demand after they occur, Vast.ai Serverless anticipates them.

The platform proactively provisions reserve GPU workers based on historical usage patterns, real-time load, and ongoing market benchmarking. This approach minimizes laggy cold starts for latency-sensitive LLM inference.

### One Endpoint, Powered by a Global GPU Fleet

Vast.ai continuously benchmarks over 17,000 GPUs across 500+ locations worldwide. Our network includes everything from RTX-class consumer GPUs to enterprise-grade machines like A100s, H100s, and B200s.

But rather than locking you into a single hardware profile, Vast.ai Serverless lets you deploy one endpoint backed by our entire globally distributed GPU fleet. [Multiple Workergroups per Endpoint](https://docs.vast.ai/documentation/serverless/architecture) enable requests to be routed dynamically based on workload requirements.

This real-time hardware selection ensures your unique, inference-heavy workloads run on the most cost-effective GPUs available at any given moment while also improving fault tolerance due to reduced single-point risk.

### Lowest-Cost Transparent Serverless Pricing

Vast.ai bills by the second across three pricing options: On-Demand, Interruptible, and Reserved. You can start with just $5, and there are no hidden fees, tiers, or usage caps.

Plus, Vast.ai Serverless delivers AI inference cost savings of up to 75% or more compared to traditional providers with centralized infrastructure. It's the lowest-cost autoscaling GPU cloud on the market today.

How does it work? Let's take a look at the deployment process.

## How to Deploy LLM Inference Serverless on Vast.ai

Running LLM inference on Vast.ai Serverless takes five steps:

### 1. Bring Your Own Large Language Model

Launch open-source LLMs like Llama 3 or DeepSeek, or deploy your own custom fine-tuned models. Vast.ai provides prebuilt images, including vLLM, TGI, and Oobabooga, so you can serve models with minimal setup.

### 2. Send an Inference Request

Call your endpoint via API to send inference requests. Vast.ai handles all orchestration behind the scenes.

### 3. Vast.ai Serverless Routes and Provisions Compute on Ready GPUs

Our platform automatically selects the most efficient hardware based on your performance targets as well as current availability. Reserve workers are already provisioned and ready to handle your request immediately.

### 4. GPU Executes and Returns Results

The selected GPU worker executes the inference request and returns results through the same API endpoint. This entire process happens seamlessly.

### 5. Serverless Predictively Scales Up and Down Based on Usage

As demand fluctuates, Vast.ai Serverless automatically scales capacity up or down to match. Billing stops immediately when resources are released, so you only pay for actual compute time.

## Get Started with Vast.ai Serverless

Deploying LLM inference at scale doesn't have to mean choosing between performance and affordability. Vast.ai Serverless gives you both. You can run production-ready inference that automatically scales when you need it and stops charging when you don't.

Ready to build? Check out our [Serverless Overview](https://docs.vast.ai/documentation/serverless) and our [Model Library](https://vast.ai/model-library), and get started today with just $5.


Learn how Vast.ai Serverless deploys cost-efficient LLM inference with predictive autoscaling, global GPU routing, and per-second billing.

Deploy LLM Inference Using Vast.ai Serverless


# Mistral Small 4 Just Dropped — Run It on Affordable H200s with Vast.ai

[Mistral Small 4](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) is the first Mistral model to unify three previously separate families — instruct, reasoning (Magistral), and coding (Devstral) — into a single set of weights. It scores 78.0 on MMLU-Pro and 71.2 on GPQA Diamond, outperforms GPT-OSS 120B on LiveCodeBench with 20% less output, and delivers 40% lower latency and 3x higher throughput than Mistral Small 3.

Under the hood, it's a 119B-parameter mixture-of-experts model that activates just 6.5B parameters per token. It includes a Pixtral vision encoder, a 256K-token context window, native function calling, and a per-request `reasoning_effort` parameter to toggle between fast responses and deep chain-of-thought. Weights are stored in FP8 and it fits on 2x H200 141 GB GPUs with no quantization required. It's Apache 2.0 licensed.

This guide walks through deploying Mistral Small 4 on [Vast.ai](https://vast.ai) with vLLM.

## Model Overview

| Property | Value |
|---|---|
| Developer | [Mistral AI](https://mistral.ai/news/mistral-small-4) |
| Model | Mistral-Small-4-119B-2603 |
| Architecture | MoE with Multi-head Latent Attention (MLA), 128 experts, 4 active per token |
| Total Parameters | 119B |
| Active Parameters | 6.5B per token |
| Context Length | 256K tokens |
| Vision | Pixtral encoder |
| License | Apache 2.0 |
| HuggingFace | [mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) |

## Deploy Mistral Small 4 on Vast.ai with vLLM

The model weights are ~111 GB in FP8. With KV cache and overhead, you need 2x H200 141 GB (282 GB total) or 4x H100 80 GB. We'll use 2x H200 since they offer more headroom per card.

### Prerequisites

- A [Vast.ai](https://cloud.vast.ai) account with credits
- [`jq`](https://jqlang.org/) for parsing JSON responses
- The Vast CLI and OpenAI Python client installed:

```bash
pip install --upgrade vastai openai
vastai set api-key <YOUR_API_KEY>
```

Generate an API key to secure your model endpoint:

```bash
export VLLM_API_KEY=$(openssl rand -hex 24)
echo $VLLM_API_KEY  # save this somewhere
```

### Find a GPU

```bash
vastai search offers "gpu_name=H200 num_gpus=2 direct_port_count>=1 rentable=true disk_space>=500 cuda_vers>=12.2"
```

### Launch the Instance

Mistral Small 4 requires a [custom vLLM Docker image](https://hub.docker.com/r/mistralllm/vllm-ms4) with fixes for tool calling and reasoning parsing. Upstream support is tracked in [vLLM PR #37081](https://github.com/vllm-project/vllm/pull/37081).

Pick an offer ID from the search results and deploy:

```bash
vastai create instance <OFFER_ID> \
  --image mistralllm/vllm-ms4:latest \
  --env '-p 8000:8000' \
  --disk 500 \
  --onstart-cmd "vllm serve mistralai/Mistral-Small-4-119B-2603 \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 2 \
    --max-model-len 131072 \
    --attention-backend FLASH_ATTN_MLA \
    --tool-call-parser mistral \
    --enable-auto-tool-choice \
    --reasoning-parser mistral \
    --api-key $VLLM_API_KEY"
```

The `--reasoning-parser mistral` flag separates the model's chain-of-thought reasoning from the final answer in API responses. `--max-model-len 131072` sets a 128K context window as a practical default — the model supports up to 256K natively if you have the VRAM headroom.

Check progress with `vastai logs <INSTANCE_ID>`.

## Call the API

Find your instance's IP and port:

```bash
vastai show instance <INSTANCE_ID> --raw | jq -r '"\(.public_ipaddr):\(.ports["8000/tcp"][0].HostPort)"'
```

You can also find this in the [Vast console](https://cloud.vast.ai/instances/).

### Text Generation

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://<IP>:<PORT>/v1",
    api_key="<YOUR_VLLM_API_KEY>",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {"role": "user", "content": "What are three advantages of mixture-of-experts architectures over dense transformers? Be concise."}
    ],
    max_tokens=512,
    temperature=0.1,
)

print(response.choices[0].message.content)
```

### Vision

The built-in Pixtral encoder handles images alongside text — pass image URLs directly in the message content:

```python
response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this chart. What benchmarks are being compared, and which model performs best?"},
                {"type": "image_url", "image_url": {"url": "https://cms.mistral.ai/assets/7d5b181d-776d-4406-aabc-b88414edf567.png"}},
            ],
        }
    ],
    max_tokens=1024,
    temperature=0.1,
)

print(response.choices[0].message.content)
```

### Configurable Reasoning

Toggle between fast responses and step-by-step reasoning on a per-request basis with `reasoning_effort`:

```python
response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {"role": "user", "content": "Find all integers n such that n^2 + 2n + 4 is divisible by 7."}
    ],
    max_tokens=2048,
    temperature=0.7,
    extra_body={"reasoning_effort": "high"},
)

# Reasoning trace is separated from the final answer
data = response.model_dump()
thinking = data["choices"][0]["message"].get("reasoning")
answer = data["choices"][0]["message"]["content"]

print("Thinking:", thinking[:500] if thinking else "N/A")
print("Answer:", answer)
```

Use `reasoning_effort="none"` for fast, everyday tasks. Use `"high"` for math, logic, and complex analysis.

## Cleanup

When you're done, destroy the instance to stop charges:

```bash
vastai destroy instance <INSTANCE_ID>
```

## Conclusion

Mistral Small 4 packs 119B parameters into a model that activates 6.5B per token, unifying instruct, reasoning, and coding capabilities behind a single endpoint. With vision, a 256K context window, and native function calling, it covers a wide range of use cases. The [NVFP4 checkpoint](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4) (~66 GB) fits on 2x H200, and a trained [Eagle speculative decoding head](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle) is available for faster generation.

## Resources

- [Mistral Small 4 on Hugging Face](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)
- [Mistral Small 4 NVFP4 on Hugging Face](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)
- [Mistral Small 4 Announcement](https://mistral.ai/news/mistral-small-4)
- [vLLM Documentation](https://docs.vllm.ai/)
- [Vast.ai Documentation](https://docs.vast.ai/)


Mistral Small 4 is the first Mistral model to unify three previously separate families — instruct, reasoning (Magistral), and coding (Devstral) — into a single set of weights.

Mistral Small 4 Just Dropped — Run It on Affordable H200s with Vast.ai


It's been a busy few months at Vast.ai, with updates spanning performance and reliability improvements, new templates and guides, and a major step forward for serverless deployment workflows.

## NVIDIA Cloud GPU Updates

Managing large numbers of rented instances is now faster and more responsive, thanks to pagination and performance improvements when fetching instance data. Monitoring serverless workloads is easier as well, with improved serverless metric visualization so you can track usage and performance at a glance.

We've also updated the Host Setup page with clearer instructions for new hosts, and extended two-factor authentication via SMS to support additional countries.

## Highlighted Feature: Deploy Serverless GPU Endpoints from Python

You can now create, configure, and manage serverless GPU endpoints directly from code, without relying on the dashboard.

With the open beta of the Vast SDK deployment workflow, you can define Docker images, apt packages, autoscaling settings, and more entirely in Python. The goal is simple: one install, one deployment flow, and a much faster path from local code to a live serverless endpoint.

Where this gets especially powerful is the `@remote` programming model. Instead of building HTTP wrappers and managing infrastructure by hand, you can decorate a Python function and call it like local code while Vast handles the serverless execution layer underneath.

This is the fastest way to spin up serverless GPUs at Vast's price-performance point, and it's available now to all users as part of the open beta.

Everything you need is in the [deployment docs](https://docs.vast.ai/documentation/serverless/deployments) and [Python examples](https://docs.vast.ai/documentation/serverless/deployments/examples/square).

![Python serverless deployment example](/uploads/april-2026-serverless-sdk.png)

## New Templates and Guides

We also shipped a fresh round of templates, model pages, and deployment guides to help you get up and running faster across training, inference, multimodal workflows, and migrations.

### New Templates

- [Unsloth Studio](https://cloud.vast.ai/?ref_id=62897&template_id=2fba903698162c83cca06d926a016764) - LLM fine-tuning and inference environment.
- [GLM 5](https://vast.ai/model/glm-5) - 744B MoE model for agentic reasoning, coding, and tool use.
- [Kimi K2.5](https://vast.ai/model/kimi-k2.5) - Next-generation multimodal agentic model.
- [vLLM Omni](https://cloud.vast.ai/?ref_id=62897&template_id=10d5f684d6da7700379b16a7bcb74faa) - Multimodal inference engine.
- [Voicebox TTS](https://cloud.vast.ai/?ref_id=62897&template_id=c2325adb01ae87b2a3e8f6f68f88fabe) - Multi-model text-to-speech with voice cloning.
- [LTX 2.3 via ComfyUI](https://vast.ai/model/ltx-2-3) - Video generation workflow.

### New Guides

- [Budget-Friendly Alternative to Claude Code - Overnight Ralph Loop](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop)
- [OpenClaw AI Assistant with vLLM on Vast.ai](https://docs.vast.ai/examples/ai-agents/openclaw)
- [Autonomous AI Research with Autoresearch on Vast.ai](https://docs.vast.ai/examples/ai-agents/autoresearch)
- [Migrate from Runpod to Vast.ai](https://docs.vast.ai/examples/migrations/runpod-to-vast)

## Other Improvements

This update also brings a number of platform-level changes, including BitPay support for crypto payments, serverless OpenAI-compatible endpoints, pagination for large rented-instance result sets, and a long list of quality-of-life and reliability fixes.

## Our Commitment

As we continue to improve platform performance and expand support for deployment workflows, our focus remains on making AI infrastructure easier to use at scale.

Need assistance? Reach out anytime at [support@vast.ai](mailto:support@vast.ai) or join our [Discord](https://discord.gg/vast) server for tips, discussion, and the latest updates as they happen.

## Change Log

### New Features

- BitPay payment gateway for crypto payments, including Lightning Network support, flat fees, and refunds across hundreds of wallets including Coinbase.
- Serverless OpenAI API-compatible endpoints released.
- Serverless metric visualization improvements.
- 2FA SMS support expanded to many additional countries.
- Pagination and performance improvements when fetching large numbers of rented instances.
- Host Setup page updated with improved instructions for new hosts.

### Issues Resolved

- Local volume deletion issue resolved.
- Copy error visibility improved.
- Copy progress bar issue resolved.
- Disk quota issues resolved.
- Machines missing geolocation resolved.
- Open button token persistence after instance recreation improved.
- Missing DLPerf on the instance page resolved.
- Various security improvements.
- Various quality-of-life improvements.
- Various serverless edge-case scaling improvements.
- Host CLI schedule maintenance command issue resolved.

### Feature Changes

- Coinbase is sunsetting Coinbase Commerce on March 31, so payment support is moving accordingly.

### API Changes

- Upcoming breaking change: `get instances` will soon return paginated results, with responses limited to a maximum of 25 instances per request. Update API, CLI, and SDK integrations to handle pagination.
- Deprecated: `disable_bundling` for `search offers` will be removed from the API, CLI, and SDK.
- Serverless SDK updated for enhanced security.

### Updated Templates

- PyTorch
- Open WebUI
- Ollama
- Ostris Toolkit
- ComfyUI
- Llama.cpp (ssl)
- SGLang (onstart script fixed)
- Pinokio (v6.x)

### New GPUs

- RTX PRO 5000

### New Blog Posts

- [Affordable Claude Code Alternative: Run Autonomous Coding Agents on Vast.ai](https://vast.ai/article/affordable-claude-code-alternative-ralph-loop)
- [Vast.ai Named Among Fastest Growing Vendors by Ramp and Brex](https://vast.ai/article/ramp-brex-fastest-growing-vendor)
- [Liquid AI's LFM2 Just Dropped - Here's How to Run It on Vast.ai](https://vast.ai/article/liquid-ai-lfm2-run-on-vast-ai)
- [Running Qwen 3.5 Medium Models on Vast.ai](https://vast.ai/article/running-qwen-3-5-medium-models-on-vast-ai)
- [Vast.ai GPUs Can Now Be Rented Through SkyPilot](https://vast.ai/article/vast-ai-gpus-can-now-be-rentend-through-skypilot)
- [7 Popular Model Templates for AI Workloads on Vast.ai Right Now](https://vast.ai/article/popular-templates-for-ai-workloads-on-vast-ai)
- [Running Nemotron-Cascade-2 on Vast.ai](https://vast.ai/article/running-nemotron-cascade-2-on-vast.ai)


Vast.ai's April 2026 product update includes Python-based serverless deployments in open beta, new templates and guides, payment updates, and platform reliability improvements.

April 2026 Product Update


[Unsloth Studio](https://unsloth.ai/docs/new/studio) is an open-source, no-code web UI for running and fine-tuning more than 500 open-source AI models across text, vision, audio, and embeddings. It delivers 2x faster training with 70% less VRAM than standard approaches, which makes it possible to fine-tune larger models on more affordable hardware.

Vast.ai now offers a pre-configured [Unsloth Studio template](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Unsloth%20Studio). Rent a GPU, click Open, select Unsloth Studio, and start running or training models without CLI setup or manual Docker configuration.

## What You Can Do

- **Run models** — Search and download any of 500+ models from Hugging Face. Chat with built-in tool calling, code execution, and web search.
- **Compare models** — Use Model Arena to evaluate two models side by side on the same prompt.
- **Fine-tune** — Train with QLoRA, LoRA, or full fine-tuning. Upload datasets or build them from raw documents using Data Recipes. Monitor training with real-time loss curves and GPU stats.
- **Export** — Save fine-tuned models as merged safetensors, LoRA adapters, or GGUF files. Upload directly to Hugging Face Hub.

## Get Started

Read the full guide: **[Run and Fine-Tune AI Models with Unsloth Studio on Vast.ai](https://docs.vast.ai/examples/ai-ml-frameworks/unsloth-studio)**

Or launch the template directly: **[Open Unsloth Studio Template](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Unsloth%20Studio)**

## Resources

- [Unsloth Studio Documentation](https://unsloth.ai/docs/new/studio)
- [Unsloth GitHub](https://github.com/unslothai/unsloth)
- [Vast.ai Instance Guide](https://docs.vast.ai/documentation/instances/overview)


Run and fine-tune 500+ open-source AI models with the Unsloth Studio template on Vast.ai.

Unsloth Studio Is Now Available on Vast.ai


# Inside GTC 2026: A Bright Future for AI and GPU Computing

NVIDIA's flagship conference, [GTC](https://www.nvidia.com/gtc/), concluded in San Jose with much fanfare. We attended and want to share our key takeaways and what they mean for the future of AI compute. GTC's sessions generated a buzz in the industry, ranging from NVIDIA CEO Jensen Huang's keynote, which introduced the powerful (and controversial) DLSS 5, to announcements of new hardware technologies and the shift from generative to agentic systems. The industry event sparked excitement about the future of AI and GPU computing.

## AI Infrastructure Becomes the Product

The focus has shifted as greater attention is now being paid to AI inference versus yesteryear's training. As AI systems mature and vie for market saturation, the need for sustained computing power grows, and with it, AI infrastructure. NVIDIA and others look to reshape the narrative around AI datacenters, framing them as "AI Factories" where training and inference converge and full-stack systems bring cost efficiency and reliability.

## From Generative to Agentic AI

With the announcement of NemoClaw, a secure, enterprise-grade wrapper for the agentic framework OpenClaw, NVIDIA has broadened the market. NemoClaw was built to address the issue of trust in AI agent adoption, making it safer for companies to utilize agentic systems at scale. This is a crucial development for founders as infrastructure will increasingly need to support long-lived processes with memory and retrieval for 24/7 persistent workloads.

## The Evolution of AI in Gaming

The latest iteration of NVIDIA's gaming tech, DLSS 5, was announced during the show and raised eyebrows in more ways than one. Some critics quickly panned it as "AI slop" but behind the graphical changes that are sure to improve over time lies the "Real-Time Neural Rendering" architecture. The advancement expands possibilities for AI use in creative fields as developers will now be able to precisely control AI rendering parameters. As barriers to AI use in creative projects fall and pressure for workflow efficiency rises, demand for GPU compute power in creative professions will grow.

## Changes in the Global AI Market

Jensen Huang had a lot to say at this year's GTC but some select comments about the broader market and NVIDIA's renewed growth within China stood out. "Our supply chain is getting fired up," Huang said, revealing that NVIDIA is in the process of restarting manufacturing of H200 chips for the Chinese market. After receiving licenses from the U.S. government to once again sell the chips in China, NVIDIA will begin fulfilling orders from Chinese companies. This will have a broader impact on the GPU market as demand for advanced chips expands once again.

## Powering the Next Era of AI Compute

GTC 2026 brought together major and emerging players who showcased new technologies, shared industry developments, and made predictions for the future of AI. What comes next will be defined by how efficiently and broadly AI compute can be deployed.

At Vast.ai, we value the fair distribution of AI computing resources. Our goal is to help you access the infrastructure you need to build, scale, and compete. [Speak to our team](https://vast.ai/contact-sales) today to learn how we can support your AI workloads.


Vast.ai attended NVIDIA GTC 2026 and shares key takeaways on what they mean for the future of AI compute.

Inside GTC 2026: A Bright Future for AI and GPU Computing


Andrej Karpathy recently released [autoresearch](https://github.com/karpathy/autoresearch) -- a framework that lets an AI agent autonomously run ML research experiments while you sleep. Point it at a GPU, give it a training script, and it will iterate on model architecture and hyperparameters for hours, keeping improvements and discarding regressions. No human in the loop.

The idea is simple but powerful: a single H100 running overnight can execute ~100 five-minute training experiments. Each one modifies the model code, trains, evaluates, and decides whether to keep or revert. By morning you have a log of findings from every experiment.

## Why This Matters

ML research has always been bottlenecked by the experiment loop -- tweak, train, evaluate, repeat. Autoresearch collapses that loop by handing it to an AI agent ([Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview)) that can run continuously. It's not replacing the researcher -- it's running the tedious overnight grid searches and ablations so you can focus on the ideas that require human intuition.

Autoresearch includes a simplified single-GPU implementation of [nanochat](https://github.com/karpathy/nanochat) -- a small GPT trained on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) and evaluated on validation bits-per-byte (`val_bpb`). The agent has full freedom to modify the training script -- architecture, optimizer, learning rate schedules, attention patterns -- within a fixed 5-minute time budget per experiment.

## Running It on Vast

We put together a step-by-step guide for running autoresearch on Vast.ai. It covers everything from renting a GPU instance to launching the autonomous research loop:

**[Autonomous AI Research with Autoresearch on Vast.ai ->](https://docs.vast.ai/examples/ai-agents/autoresearch)**

The guide walks through:

- Renting a GPU instance with the Vast CLI
- Installing dependencies (`uv`, Node.js, Claude Code)
- Preparing the training data and running a baseline experiment
- Configuring Claude Code for unattended overnight operation
- Monitoring experiments as they run

All you need is a Vast.ai account and a Claude Code account. The whole setup takes about 15 minutes before you can kick off the research loop and walk away.


Run Andrej Karpathy's autoresearch on Vast.ai H100 GPUs to automate overnight ML experiments and training loops.

How to Run Karpathy's Autoresearch on Vast H100s


If you’ve ever dabbled in machine learning, data science, or coding in general, chances are you’ve heard of Google Colaboratory – better known as **Google Colab**. But what exactly is it, what are its limitations, and how can you make the best use of it?

Let’s take a closer look at these questions and more...

## What Is Google Colab?

In a nutshell, **Google Colab** is a browser-based platform that lets you run and write code in a Jupyter notebook environment without having to set up anything locally on your machine. Offering both free and paid plans, it’s widely used for Python-based projects, especially in machine learning and data analysis.

Colab is an easy starting point for experiments, since you can use the Jupyter notebook interface to write and execute code, visualize results, and keep everything neatly organized in one place. This type of interactive programming allows you to experiment with code in real time, making it ideal for data science and research projects that require immediate feedback and iteration.

Some other benefits include:

- **Accessibility** – The browser-based platform allows you to access your projects from anywhere.
- **Integration with Google Drive** – Save your notebooks and datasets directly to your Google Drive.
- **Collaboration** – Easily share your notebooks with others for real-time collaboration across teams.
- **Wealth of Resources** – Plenty of tutorials, sample notebooks, and community support are available in the Colab ecosystem, making it easier for you to learn and troubleshoot along the way.

These features make Google Colab a convenient tool for a wide range of users. However, it does have some significant limitations that may prompt you to look for other solutions – such as running a Jupyter instance right here on **Vast** and uploading your notebook directly.

## Why Choose Vast Over Google Colab?

While Google Colab is helpful for smaller-scale projects or initial experimentation, **Vast's** cloud GPU platform offers a much more flexible, powerful, and reliable experience through our Jupyter integration. Here’s why you should consider using Vast instead.

### No Resource Limits

With Google Colab, free-tier users face dynamic usage limits with idle timeout periods, very limited access to GPUs, and resource caps that sometimes fluctuate. The paid plans have their own set of restrictions as well.

For example, notebooks can only run for up to 12 hours on a free Colab plan. The highest-tier paid plan supports continuous background execution for at most 24 hours, with output saved to Google Drive even if your browser is closed. Lower-tier paid plans have some extended execution based on availability and may not automatically save output to Drive.

But with **Vast**, there are no imposed time limits, so your projects can run uninterrupted for as long as you need. Whether your project takes hours or days, you won’t be cut off or forced to restart – and your data will stay on the Vast instance, available to be copied at all times.

> **Note:** Google Colab does include the option to pay for a dedicated virtual machine (VM) to get around usage limits, but Vast’s pricing is extremely competitive as we are the market leader in low-cost cloud GPU rental.

### Customizable GPU Power for Every Project

Google Colab offers different VMs based on the plan you choose. The free tier only allows access to VMs with a standard system memory profile, restricting the compute power you can use. Paid versions of Colab provide access to machines with a higher system memory profile, but you still don't get to select specific GPUs unless you opt to manually pay for a dedicated machine.

**Vast** gives you access right from the start to powerful and customizable hardware options that are also highly affordable. You can select the exact GPU setup you need, whether you're running small tasks or large-scale machine learning models. For high-end performance, Vast supports multi-GPU configurations, making it easy to scale up your projects without hitting resource bottlenecks.

### Simplicity and Reliability

**Vast** simplifies the entire process by running your Jupyter notebook directly in the cloud with no need for SSH or complex setups. Jupyter by itself is much more reliable than Google Colab and doesn’t require setting up SSH keys – you can always open a terminal right inside the browser.

And while Google Colab does facilitate sharing and collaboration, **Vast** offers the same, with the added benefit of running on a more stable platform without timeouts or interruptions. Team members can collaborate in real time, sharing notebooks and results without worrying about session limits.

## Making the Switch to Vast

If you’ve been using Google Colab for simple notebooks and want to step up your workflow, switching to Vast is straightforward. Simply download your notebook from Colab as a `.ipynb` file, run a Vast Jupyter instance with the recommended [PyTorch template](https://cloud.vast.ai/?ref_id=43484&template_id=f5540ef1a1398b8499546edb53dae704), and upload your notebook into Jupyter directly.

Check out the step-by-step guide on running [Jupyter notebooks](https://docs.vast.ai/google-colab) on Vast.

While Google Colab is a useful starting point for basic tasks, it often falls short when workflows become more demanding and compute costs start adding up. **Vast** provides a more cost-effective and customizable solution through its Jupyter notebook integration, with greater control, higher performance, and fewer interruptions.

With **Vast.ai**, you’ll enjoy the freedom to push your projects even further. **Explore the possibilities today!**


Google Colab is a browser-based platform that lets you run and write code in a Jupyter notebook environment without having to set up anything locally on your machine.

Google Colab Explained: Simplifying Your Workflow with Cloud Tools


# Running Nemotron-Cascade-2 on Vast.ai

NVIDIA released [Nemotron-Cascade-2-30B-A3B](https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B), a reasoning model that achieves gold-medal-level scores on international math and programming competitions. It does this with 30B total parameters and only 3B active per token - 20x fewer than competing models - outperforming both Qwen3.5-35B-A3B and larger open-weight alternatives.

The model fits on a single 80 GB GPU at about 60 GB in BF16 with no quantization or multi-GPU setup required. It includes built-in chain-of-thought reasoning, tool-integrated reasoning for math and code, and an OpenAI-compatible API.

This guide walks through deploying the model on [Vast.ai](https://vast.ai) with vLLM. Vast.ai offers A100 and H100 GPUs at 3-5x less than traditional cloud providers, with no contracts or minimums.

## Model Overview

| Property | Value |
| --- | --- |
| Developer | [NVIDIA](https://research.nvidia.com/labs/nemotron/nemotron-cascade-2/) |
| Model | Nemotron-Cascade-2-30B-A3B |
| Architecture | Hybrid Mamba-Attention MoE |
| Total Parameters | 30B |
| Active Parameters | 3B per token |
| Training | Cascade RL post-training from Nemotron-3-Nano base |
| License | [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| HuggingFace | [nvidia/Nemotron-Cascade-2-30B-A3B](https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B) |

## Deploy Nemotron-Cascade-2 on Vast.ai with vLLM

The model requires a single 80 GB GPU (A100 or H100) with about 150 GB disk for weights and cache.

### Prerequisites

- A [Vast.ai](https://cloud.vast.ai) account with credits
- The Vast CLI installed and configured:

```bash
pip install --upgrade vastai
vastai set api-key <YOUR_API_KEY>
```

Generate an API key to secure your model endpoint:

```bash
export NEMOTRON_API_KEY=$(openssl rand -hex 24)
echo $NEMOTRON_API_KEY  # save this somewhere
```

### Find a GPU

Search for the cheapest single 80 GB GPU with direct port access:

```bash
# -o dph sorts results by price (dollars per hour, cheapest first)
vastai search offers "gpu_ram>=80 num_gpus=1 direct_port_count>=1 cuda_vers>=12.4 cuda_vers<13" -o dph
```

### Launch the Instance

Pick an instance ID from the search results and deploy with vLLM:

```bash
vastai create instance <INSTANCE_ID> \
  --image vllm/vllm-openai:latest \
  --env '-p 8000:8000' \
  --disk 150 \
  --onstart-cmd "vllm serve nvidia/Nemotron-Cascade-2-30B-A3B \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 32768 \
    --api-key $NEMOTRON_API_KEY \
    --trust-remote-code"
```

This pulls the model from HuggingFace and starts an OpenAI-compatible API server on port 8000. Initial startup takes several minutes while the model downloads.

Once the instance is running, get your connection details:

```bash
vastai show instances --raw | jq '.[] | select(.id == <INSTANCE_ID>) | {id, actual_status, public_ipaddr, ports}'
```

You can also find the IP and port mapping in the [Vast console](https://cloud.vast.ai/instances/).

## Call the API

Send a request to test the model:

```bash
curl http://<VAST_IP>:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $NEMOTRON_API_KEY" \
  -d '{
    "model": "nvidia/Nemotron-Cascade-2-30B-A3B",
    "messages": [{"role": "user", "content": "Find all integers n such that n^2 + 2n + 4 is divisible by 7."}],
    "max_tokens": 2048
  }'
```

The model thinks before it answers - you will see its step-by-step reasoning in the response, followed by the final answer. This chain-of-thought reasoning is what drives its competition-level performance on math and code.

## Cleanup

When you're done, destroy the instance to stop charges:

```bash
vastai destroy instance <INSTANCE_ID>
```

## Conclusion

Nemotron-Cascade-2 delivers reasoning performance that previously required models 20x its size, and it runs on a single GPU. That combination of capability and efficiency makes it a practical option for math, code, and general reasoning workloads. With Vast.ai, you can have it running in minutes at a fraction of what traditional cloud providers charge.

## Resources

- [Nemotron-Cascade-2-30B-A3B on HuggingFace](https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B)
- [Nemotron-Cascade-2 Research Page](https://research.nvidia.com/labs/nemotron/nemotron-cascade-2/)
- [Nemotron-Cascade-2 Paper (arXiv:2603.19220)](https://arxiv.org/abs/2603.19220)
- [Vast.ai Documentation](https://docs.vast.ai/)
- [vLLM Documentation](https://docs.vllm.ai/)


Deploy NVIDIA's Nemotron-Cascade-2-30B-A3B on Vast.ai with vLLM and run a powerful reasoning model on a single 80GB GPU.

Running Nemotron-Cascade-2 on Vast.ai


When you're getting started with an AI project, choosing the right model is only half the battle. You also need to configure environments and install dependencies while connecting the tools that make your workflows run smoothly. Setting things up can quickly become a bottleneck.

Prebuilt templates simplify the process – and **Vast.ai** gives you plenty to choose from!

Our [Model Library](https://vast.ai/model-library) offers a model-first way to deploy AI, with templates that launch ready-to-run environments for some of the most-used AI tools today. Vast.ai also offers [lower-level templates](https://cloud.vast.ai/templates) for frameworks and inference engines like PyTorch and vLLM for more customizable setups.

Each pre-configured template is organized by modality and use case so you can quickly find what you need – and you retain full custom control over the underlying runtime, parameters, and infrastructure.

### Popular Templates in the Vast.ai Model Library

From image, text, and audio-video generation to multimodal AI workflows, the following are a few of the most popular templates being deployed on Vast.ai right now.

#### 1. Flux.2-Dev – Rectified Flow Image Generation

[**FLUX.2-Dev**](https://vast.ai/model/flux.2-dev) is a 32-billion parameter rectified flow transformer built for advanced text-to-image generation and editing. The model processes text prompts alongside optional reference images, so you can modify existing images or combine multiple visual references without additional fine-tuning.

Its ability to maintain consistent characters, objects, and styles across generations makes it ideal for character design, visual storytelling with scene continuity, digital art with precise style control, and design prototyping.

#### 2. LTX-2 – Native Audio-Video Generation in a Single Workflow

[**LTX-2**](https://vast.ai/model/ltx-2) is a 19-billion parameter Diffusion Transformer (DiT) model that generates audio and video together in a single pass. It creates synchronized audiovisual output from text prompts, images, and other inputs – keeping dialogue, motion, and ambient sound aligned throughout the clip.

One of LTX-2's biggest advantages is its light and iterative workflow. Since the model generates video in a compressed latent space before converting it to full resolution, it supports faster iteration and more efficient memory usage than many traditional video models.

#### 3. WAN 2.2 – Efficient Text-to-Video and Image-to-Video Generation

[**WAN 2.2**](https://vast.ai/article/wan-2-2-explained-new-approach-ai-video-generation) is a video generation model that produces short cinematic clips from either text prompts or static images. What sets it apart is its Mixture-of-Experts (MoE) architecture. The model switches between specialized experts that handle early scene structure and later visual refinement. This results in smoother motion and higher visual fidelity along with more efficient use of compute.

Available on Vast.ai in both text-to-video ([T2V](https://vast.ai/model/wan-2.2-t2v-a14b)) and image-to-video ([I2V](https://vast.ai/model/wan-2.2-i2v-a14b)) variants – with the latter offering optional guidance via text prompts – WAN 2.2 gives you fine control over motion, lighting, and scene composition. Popular use cases include rapid prototyping of video concepts based on text descriptions and product animations generated from still images.

> Not sure whether **WAN 2.2** or **LTX-2** fits your workflow best? Our detailed [comparison post](https://vast.ai/article/wan-2-2-vs-ltx-2-which-ai-video-model-should-you-use) explores the strengths of each model and which one to choose for your specific needs!

#### 4. ACE-Step V1 – Fast Text-to-Music Generation

[**ACE-Step V1**](https://vast.ai/model/ace-step-v1-3.5b) is an open-source music generation model that creates full musical compositions from text prompts in **17 languages**. Thanks to its diffusion-based architecture and efficient audio compression pipeline, it generates music about **15x faster** than comparable alternatives.

Musical quality is high, with controllable duration (up to five minutes works best) and coherent output across melody, harmony, and rhythm. You can even edit lyrics and manipulate vocals. Overall, ACE-Step V1 is a practical tool for fast experimentation with musical ideas, as well as voice cloning applications and music remixing and style transfer.

#### 5. Dia 1.6B – Realistic Dialogue Generation from Text

[**Dia 1.6B**](https://vast.ai/model/dia-1.6b) is a text-to-speech model that produces highly realistic dialogue in English from given transcripts. Using [S1] and [S2] speaker tags, it creates multi-speaker conversations that can also include nonverbal expressions like laughter, coughing, and gasps.

Audio conditioning enables precise control over emotion and tone, and the model even supports voice cloning functionality. Dia 1.6B's applications include conversational audio content, accessibility tools, and voice synthesis applications that require consistent, natural-sounding speech.

#### 6. DeepSeek V3.2 Exp – Long-Context Sparse Attention LLM

[**DeepSeek V3.2 Exp**](https://vast.ai/model/deepseek-v3.2-exp) is an open-source experimental large language model designed for long-context reasoning and advanced text generation. It introduces DeepSeek Sparse Attention, a novel mechanism that enables efficient processing of extended documents and conversations without sacrificing output quality.

The model is well suited for long-form document analysis and generation, research that requires extended reasoning, code generation and debugging, and even customer support with context-aware responses. Notably, it has a license that offers full commercial use permissions.

#### 7. Qwen3.5 397B A17B – Multimodal Reasoning and Agentic Workflows

[**Qwen3.5 397B A17B**](https://vast.ai/model/qwen35-397b-a17b) is the open-weight of the first model in the new Qwen3.5 series. It's a frontier-level native vision-language model with outstanding results across numerous benchmarks.

Built with a sparse mixture-of-experts (MoE) architecture and hybrid attention mechanisms, Qwen3.5 397B A17B supports highly complex tasks in areas like mathematical reasoning and problem solving, multi-turn agentic workflows, code generation and debugging, document analysis, video understanding, visual question answering, and multilingual interaction across **200+** languages and dialects.

### Launch Templates in Minutes on Vast.ai

Today's AI workloads span an incredibly wide range of modalities. The templates above represent just a small snapshot of what developers are building with on Vast.ai right now.

With the **Vast.ai** [Model Library](https://vast.ai/model-library), getting started on your own AI projects is easy. Simply browse templates by use case, select the recommended GPU hardware, and launch a production-ready environment in minutes!


Discover the most popular prebuilt templates on Vast.ai for image, video, audio, and text AI workloads — from Flux.2-Dev to Qwen3.5.

7 Popular Model Templates for AI Workloads on Vast.ai Right Now


[SkyPilot](https://skypilot.co/) is an open-source framework from UC Berkeley that abstracts away cloud infrastructure, letting you define AI workloads once and run them on any provider. It handles resource discovery, provisioning, cost optimization, and failover across 25+ clouds, including Vast.ai.

With Vast.ai as a SkyPilot backend, you get access to Vast's GPU marketplace, often the lowest-cost option for on-demand GPU compute, through SkyPilot's unified interface. SkyPilot automatically finds the cheapest available instance matching your requirements, provisions it, and manages the full lifecycle.

This guide covers setup, configuration, deploying a model, and the advanced Vast-specific options available in SkyPilot.

## Prerequisites

- A [Vast.ai account](https://cloud.vast.ai/) with credits
- Python 3.7-3.13 (on Python 3.12+, install `skypilot[vast]` explicitly - it is excluded from `skypilot[all]`)
- Terminal access

## Installation

Install SkyPilot with Vast.ai support:

```bash
pip install -U "skypilot[vast]"
```

If you use `uv`, you can also install via:

```bash
uv pip install "skypilot[vast]"
```

This pulls in all required dependencies, including the Vast SDK.

## Configuring Credentials

Get your API key from the [Vast.ai Account page](https://cloud.vast.ai/account/), then store it where SkyPilot expects:

```bash
mkdir -p ~/.config/vastai
echo "<YOUR_VAST_API_KEY>" > ~/.config/vastai/vast_api_key
```

Verify that SkyPilot detects your credentials:

```bash
sky check
```

You should see `Vast: enabled` in the output.

## Deploying a Model: DeepSeek R1 on vLLM

This section walks through deploying DeepSeek R1-0528-Qwen3-8B, an 8B parameter model with step-by-step reasoning capabilities, on a Vast.ai H100 GPU using vLLM.

### Create the YAML Configuration

Create a file named `deepseek-r1-inference.yaml`:

```yaml
resources:
  accelerators: H100:1
  cloud: vast
  image_id: docker:vllm/vllm-openai:latest
  disk_size: 100

run: |
  vllm serve deepseek-ai/DeepSeek-R1-0528-Qwen3-8B \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.9 \
    --reasoning-parser deepseek_r1
```

| Parameter | Purpose |
|-----------|---------|
| `accelerators: H100:1` | Request 1x H100 GPU |
| `cloud: vast` | Use Vast.ai as the cloud provider |
| `image_id: docker:vllm/vllm-openai:latest` | Pre-built vLLM Docker image |
| `disk_size: 100` | 100GB storage for model weights and container |
| `--reasoning-parser deepseek_r1` | Enable DeepSeek's chain-of-thought reasoning output |
| `--gpu-memory-utilization 0.9` | Use 90% of GPU VRAM for model serving |

### Launch the Cluster

Deploy the service with SkyPilot:

```bash
sky launch deepseek-r1-inference.yaml --cluster deepseek-r1
```

SkyPilot will find the cheapest available H100 on Vast.ai, provision it, download the model (~15GB), and start the vLLM server. This takes 5-10 minutes.

Monitor progress in real time:

```bash
sky logs deepseek-r1 --follow
```

Confirm the cluster is ready:

```bash
sky status deepseek-r1
```

Look for `Status: UP`.

### Access the Service via SSH Tunnel

Vast.ai instances provisioned through SkyPilot don't expose ports directly. Use SSH tunneling to access the vLLM server on your local machine:

```bash
ssh -L 8000:localhost:8000 deepseek-r1
```

Keep this terminal open - the tunnel must remain active for API access.

### Test the Deployment

In a new terminal, verify the server is running:

```bash
curl http://localhost:8000/health
curl http://localhost:8000/v1/models
```

Send a test inference request:

```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    "messages": [
      {"role": "user", "content": "What is 15 * 8? Show your reasoning step by step."}
    ],
    "max_tokens": 300,
    "temperature": 0.1
  }'
```

### Python Client Example

The vLLM server exposes an OpenAI-compatible API, so you can use the standard OpenAI Python client:

```python
import openai

client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="sk-fake-key"  # vLLM doesn't require a real API key
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{
        "role": "user",
        "content": "Write a brief history of Los Angeles in one paragraph."
    }],
    max_tokens=300,
    temperature=0.7
)

print(response.choices[0].message.content)
```

## Managing Your Cluster

```bash
# Check status
sky status deepseek-r1

# View logs
sky logs deepseek-r1

# SSH into the instance
ssh deepseek-r1

# Stop the cluster (preserves data, stops billing)
sky stop deepseek-r1

# Restart a stopped cluster
sky start deepseek-r1

# Auto-stop after 30 minutes of idle time
sky autostop deepseek-r1 --idle-minutes 30

# View all clusters and costs
sky status --all

# Tear down completely
sky down deepseek-r1
```

Always stop or tear down clusters when you're done to avoid unnecessary charges.

## Advanced: Vast Configuration in SkyPilot

SkyPilot supports Vast-specific configuration in `~/.sky/config.yaml`. These options let you control instance selection and pass parameters directly to the Vast API.

### Datacenter-Only Instances

Filter to professional datacenter-hosted machines only. This excludes consumer-grade or home-hosted GPUs, which can improve reliability:

```yaml
vast:
  datacenter_only: true
```

Note that some GPU types may only be available on non-datacenter offers, so enabling this may reduce availability.

### Custom Instance Parameters

The `create_instance_kwargs` block passes parameters directly to the Vast API when creating instances. This gives you access to Vast features beyond what SkyPilot's standard YAML exposes:

```yaml
vast:
  datacenter_only: true
  create_instance_kwargs:
    python_utf8: true
    lang_utf8: true
    extra: "--shm-size=16g"
    onstart_cmd: "echo 'Instance started'"
```

#### Supported Parameters

| Parameter | Description |
|-----------|-------------|
| `image` | Docker image override (e.g., `vastai/base-image:@vastai-automatic-tag`) |
| `env` | Environment variables and port mappings (e.g., `"-e KEY=value -p 8080:8080"`) |
| `price` / `bid_price` | Maximum bid price for preemptible (spot) instances |
| `disk` | Disk size in GB (overrides `disk_size` in task YAML) |
| `label` | Custom instance label |
| `extra` | Extra `docker run` arguments (e.g., `"--shm-size=16g"`) |
| `onstart_cmd` | Shell command to run on instance start |
| `onstart` | Path to a local script to run on instance start |
| `login` / `image_login` | Docker registry credentials for private images |
| `python_utf8` | Enable Python UTF-8 mode |
| `lang_utf8` | Enable system UTF-8 locale |
| `jupyter_lab` | Start JupyterLab on the instance |
| `jupyter_dir` | Jupyter notebook directory path |
| `template_hash_id` | Use a pre-configured Vast template by hash ID |
| `args` | Custom Docker command arguments |
| `user` | Run the container as a specific user |
| `vm` | Use VM mode instead of container mode |

### Using Vast Templates

If you have a saved Vast template, you can reference it by hash ID. When using a template, `image` and `disk` are inherited from the template:

```yaml
vast:
  create_instance_kwargs:
    template_hash_id: "abc123def456"
    price: 0.50
```

### Spot (Preemptible) Instances

Request spot instances for lower costs. SkyPilot handles preemption recovery automatically when using managed jobs:

```yaml
resources:
  accelerators: H100:1
  cloud: vast
  use_spot: true
```

You can set a maximum bid price via `create_instance_kwargs`:

```yaml
vast:
  create_instance_kwargs:
    bid_price: 0.50
```

## Current Limitations

Vast.ai support in SkyPilot does not currently include:

- **Multi-node clusters** - interconnection between nodes is non-trivial on Vast
- **Object store mounting** - no direct mounting of S3, GCS, etc.
- **Port opening after launch** - ports must be configured at launch time
- **Custom disk or network tiers** - Vast does not expose these options

## Troubleshooting

### "Vast: not enabled" After `sky check`

Verify your API key file exists and is readable:

```bash
cat ~/.config/vastai/vast_api_key
```

If the file is missing, re-run the credential setup steps above. If you previously stored the key at `~/.vast_api_key`, note that SkyPilot now expects it at `~/.config/vastai/vast_api_key`.

### SSH Tunnel Connection Refused

Confirm the cluster is running before attempting the tunnel:

```bash
sky status deepseek-r1
```

If the status is `UP`, recreate the tunnel:

```bash
ssh -L 8000:localhost:8000 deepseek-r1
```

### Model Not Loading or Out of Memory

SSH into the instance and check GPU utilization:

```bash
ssh deepseek-r1
nvidia-smi
```

If the model exceeds available VRAM, reduce `--gpu-memory-utilization` to `0.7` or `--max-model-len` to a smaller value in your YAML, then redeploy.

### Slow Startup

Large Docker images (vLLM is ~15GB) take time to download. Model weights add another 5-30GB depending on the model. First launch on a new instance can take 10+ minutes. Use `sky logs <cluster> --follow` to monitor progress.

## Cleanup

Tear down the cluster when you're done to stop all charges:

```bash
sky down deepseek-r1
```

## Next Steps

- **Try other models**: Swap the model in the YAML to serve Llama 3, Mistral, or Qwen
- **Use managed jobs**: Run `sky jobs launch` instead of `sky launch` for automatic recovery from preemptions
- **Explore spot instances**: Add `use_spot: true` to your resource block for lower costs
- **Datacenter filtering**: Set `datacenter_only: true` in `~/.sky/config.yaml` for production workloads
- **SkyPilot docs**: Full reference at [docs.skypilot.co](https://docs.skypilot.co/)

## Additional Resources

- [SkyPilot Documentation](https://docs.skypilot.co/)
- [SkyPilot GitHub](https://github.com/skypilot-org/skypilot)
- [SkyPilot Vast Configuration Reference](https://docs.skypilot.co/en/latest/reference/config.html#vast)
- [Vast.ai Console](https://cloud.vast.ai/)
- [Vast.ai Documentation](https://vast.ai/docs/)
- [DeepSeek R1 on HuggingFace](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B)


Learn how to use SkyPilot with Vast.ai to provision low-cost GPUs, deploy vLLM workloads, tune Vast-specific settings, and troubleshoot common issues.

Vast.ai GPUs Can Now Be Rented Through SkyPilot


Qwen 3.5 is Alibaba's latest model family, featuring a hybrid architecture that combines Gated DeltaNet (linear attention) with standard attention in a 3:1 ratio. This design enables dramatically faster inference — up to 8.6x at 32K context and 19x at 256K — while maintaining strong reasoning capabilities. The medium models released February 24, 2026 include:

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Total Params</th>
      <th>Active Params</th>
      <th>Architecture</th>
      <th>VRAM (BF16)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Qwen3.5-122B-A10B</td>
      <td>122B</td>
      <td>10B</td>
      <td>MoE</td>
      <td>~244 GB</td>
    </tr>
    <tr>
      <td>Qwen3.5-35B-A3B</td>
      <td>35B</td>
      <td>3B</td>
      <td>MoE</td>
      <td>~66 GB</td>
    </tr>
    <tr>
      <td>Qwen3.5-27B</td>
      <td>27B</td>
      <td>27B</td>
      <td>Dense</td>
      <td>~54 GB</td>
    </tr>
  </tbody>
</table>

The MoE models route each token through 8 of 256 experts plus 1 shared expert, so only a fraction of the total parameters are active per token. The 35B-A3B model activates just 3B parameters per token while having 35B total — giving you large-model quality at small-model inference cost. All three models are Apache 2.0 licensed, so no HuggingFace token is needed.

In this guide, we'll deploy Qwen3.5-35B-A3B on Vast.ai using SGLang, which has day-0 Qwen3.5 support as of v0.5.9, on a single A100 80GB GPU.

## Deploying on Vast.ai

Install the Vast.ai CLI and set your API key:

```bash
pip install --upgrade vastai
export VAST_API_KEY="YOUR_KEY_HERE"
vastai set api-key $VAST_API_KEY
```

Search for a suitable GPU. The model needs ~66 GB VRAM in BF16, so an 80 GB card works:

```bash
vastai search offers "gpu_ram>=80 num_gpus=1 direct_port_count>=1 rentable=true disk_space>=200 cuda_vers>=12.2" -o dph
```

Deploy the model using SGLang's Docker image. Replace `YOUR_INSTANCE_ID` with an offer ID from the search results:

```bash
vastai create instance YOUR_INSTANCE_ID \
    --image lmsysorg/sglang:latest \
    --env '-p 8000:8000' \
    --disk 200 \
    --onstart-cmd "python3 -m sglang.launch_server \
        --model-path Qwen/Qwen3.5-35B-A3B \
        --host 0.0.0.0 --port 8000 \
        --tp-size 1 \
        --context-length 32768 \
        --reasoning-parser qwen3 \
        --mem-fraction-static 0.85"
```

Key flags:
- `--reasoning-parser qwen3` enables the model's built-in thinking mode, separating reasoning from the final answer
- `--mem-fraction-static 0.85` allocates 85% of GPU memory for KV cache
- `--context-length 32768` sets a 32K context window (the model supports up to 262K natively)

The model will download (~70 GB) and load automatically. Check `vastai logs YOUR_INSTANCE_ID` — you'll see "The server is fired up and ready to roll!" when it's serving.

## Calling the Model

Find your instance's IP and port in the [Instances tab](https://cloud.vast.ai/instances/) of the Vast.ai console. Click the IP address button to see the port mapping for 8000/tcp.

Install the OpenAI SDK:

```bash
pip install --upgrade openai
```

Then call the model:

```python
from openai import OpenAI

VAST_IP_ADDRESS = "YOUR_IP"
VAST_PORT = "YOUR_PORT"

client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B",
    messages=[
        {"role": "user", "content": "What are three benefits of mixture-of-experts models? Be concise."}
    ],
    max_tokens=1024,
    temperature=0.7
)

print(response.choices[0].message.content)
```

### Response

```
1. **Computational Efficiency:** Only a subset of parameters is activated per input,
   reducing inference costs while maintaining a massive total parameter count.
2. **Enhanced Performance:** The larger effective model capacity allows for better
   accuracy on complex and diverse tasks compared to dense models of similar compute.
3. **Specialization:** Individual experts can learn to handle specific input patterns
   or domains, improving the model's robustness and adaptability.
```

With `--reasoning-parser qwen3`, the model's thinking process is automatically separated from the final answer. The thinking tokens appear in `reasoning_content` in the API response, while the clean answer appears in `content`.

## Other Models

You can swap `Qwen/Qwen3.5-35B-A3B` for the other medium models:

- **Qwen3.5-122B-A10B**: Larger MoE model, 122B total / 10B active. At BF16 the full weights need ~244 GB, so you'll need multiple GPUs (e.g., 4× A100 80GB with `--tp-size 4`). Alternatively, GGUF quantizations are available — the Q4_K_M variant is ~70 GB and fits on a single 80GB card.
- **Qwen3.5-27B**: Dense model, all 27B parameters active. Needs ~54 GB VRAM. Fits on a single A100 or H100.

## Conclusion

Qwen3.5-35B-A3B packs 35B parameters into a model that only activates 3B per token, and its hybrid DeltaNet architecture delivers substantially faster inference than standard transformers. With SGLang and a single A100 on Vast.ai, you can have it serving in under 10 minutes.

For production use, consider increasing `--context-length` — the model supports up to 262K natively, with 128K+ recommended for best reasoning performance. GGUF quantizations (Q4_K_M at ~22 GB) are also available for running on consumer GPUs like the RTX 4090.

## Useful Resources

- [Qwen3.5-35B-A3B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)
- [Qwen3.5-122B-A10B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)
- [Qwen3.5-27B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-27B)
- [SGLang Documentation](https://docs.sglang.ai/)
- [Vast.ai Documentation](https://docs.vast.ai/)


Deploy Qwen 3.5 medium models on Vast.ai using SGLang, including setup, launch commands, and API usage.

Running Qwen 3.5 Medium Models on Vast.ai


Liquid AI just released [LFM2-24B-A2B](https://huggingface.co/LiquidAI/LFM2-24B-A2B), a 24 billion parameter foundation model that only activates 2.3 billion parameters per token. That ratio — less than 10% of total parameters active at inference time — makes it one of the most aggressively sparse mixture-of-experts models available today.

What makes LFM2 different from other MoE models isn't just the sparsity. The architecture is a hybrid design that replaces most of the attention layers with gated short convolution blocks. Out of 40 total layers, only 10 use grouped query attention — the other 30 are convolutional. This is a fundamentally different approach from transformer-only models, and it's what allows Liquid AI to push the active parameter count so low while maintaining competitive quality across benchmarks like MMLU-Pro, GPQA Diamond, IFEval, and MATH-500.

Liquid AI, an MIT spinoff, has scaled this architecture from 350M to 24B parameters following log-linear scaling laws. LFM2 is the first model in the family large enough to be broadly useful — and with day-one support for vLLM, SGLang, and llama.cpp, it's ready to deploy today.

> **Note**: LFM2-24B-A2B is an early checkpoint trained on 17 trillion tokens without reinforcement learning or extensive post-training. A post-trained version (LFM2.5-24B-A2B) is coming.

## Model Overview

<table>
  <thead>
    <tr>
      <th>Property</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Developer</td>
      <td><a href="https://www.liquid.ai">Liquid AI</a></td>
    </tr>
    <tr>
      <td>Model</td>
      <td>LFM2-24B-A2B</td>
    </tr>
    <tr>
      <td>Architecture</td>
      <td>Hybrid MoE: 30 convolutional + 10 GQA attention layers, 64 experts, top-4 routing</td>
    </tr>
    <tr>
      <td>Total Parameters</td>
      <td>24B</td>
    </tr>
    <tr>
      <td>Active Parameters</td>
      <td>2.3B per token</td>
    </tr>
    <tr>
      <td>Context Length</td>
      <td>32,768 tokens</td>
    </tr>
    <tr>
      <td>Training Data</td>
      <td>17T tokens (mixed BF16/FP8)</td>
    </tr>
    <tr>
      <td>License</td>
      <td><a href="https://www.liquid.ai/lfm-open-license-v1-0">LFM Open License v1.0</a></td>
    </tr>
    <tr>
      <td>HuggingFace</td>
      <td><a href="https://huggingface.co/LiquidAI/LFM2-24B-A2B">LiquidAI/LFM2-24B-A2B</a></td>
    </tr>
  </tbody>
</table>

## Deploy LFM2-24B-A2B on Vast.ai with vLLM

At BF16, the model weights are ~48 GB — it fits on a single A100-80GB with room for a 32K context KV cache. No quantization or multi-GPU setup needed.

### Prerequisites

- A [Vast.ai](https://cloud.vast.ai) account with credits
- The Vast CLI installed and configured:

```bash
pip install --upgrade vastai  # make sure you're on v0.5.0+
vastai set api-key <YOUR_API_KEY>
```

### Find a GPU

Search for a single GPU with at least 80 GB VRAM and direct port access:

```bash
vastai search offers "gpu_ram>=80 num_gpus=1 direct_port_count>=1 cuda_vers>=12.4 cuda_vers<13" -o dph
```

> **Important**: Use a machine with **CUDA 12.x** (not 13.x). The vLLM nightly image is built against CUDA 12 and will fail with a driver mismatch on CUDA 13 machines.

### Launch the Instance

Pick an instance ID from the search results and deploy with vLLM. LFM2 requires a newer version of the `transformers` library than what ships in the vLLM image, so we upgrade it at startup:

```bash
vastai create instance <INSTANCE_ID> \
  --image vllm/vllm-openai:nightly \
  --env "-p 8000:8000" \
  --disk 100 \
  --onstart-cmd "pip install 'transformers>=5.0.0' && \
    vllm serve LiquidAI/LFM2-24B-A2B \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 32768 --dtype auto \
    --trust-remote-code"
```

This upgrades the tokenizer library, pulls the model from HuggingFace, and starts an OpenAI-compatible API server on port 8000. Initial startup takes several minutes while the model downloads (~48 GB for the BF16 weights).

Once the instance is running, find your IP and port in the [Vast console](https://cloud.vast.ai/instances/) — click the IP button on your instance to see the port mapping.

> **Why `nightly` and `transformers>=5`?** LFM2's tokenizer uses the `TokenizersBackend` class introduced in `transformers 5.0`. The stable vLLM image pins `transformers<5`, so the tokenizer fails to load. Using `nightly` with a transformers upgrade resolves this. This was tested with vLLM nightly build `v0.16.0rc2`. Once a stable vLLM release ships with `transformers>=5.0.0` support, you can switch to `vllm/vllm-openai:latest` and drop the `pip install` step.

## Call the API

```bash
curl http://<VAST_IP>:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "LiquidAI/LFM2-24B-A2B",
    "messages": [{"role": "user", "content": "What makes mixture-of-experts models efficient?"}],
    "max_tokens": 300
  }'
```

## Cleanup

When you're done, destroy the instance to stop charges:

```bash
vastai destroy instance <INSTANCE_ID>
```

## Resources

- [LFM2-24B-A2B on HuggingFace](https://huggingface.co/LiquidAI/LFM2-24B-A2B)
- [LFM2-24B-A2B GGUF Quantizations](https://huggingface.co/LiquidAI/LFM2-24B-A2B-GGUF)
- [Liquid AI Blog: LFM2-24B-A2B](https://www.liquid.ai/blog/lfm2-24b-a2b)
- [Liquid AI Models](https://www.liquid.ai/models)
- [Vast.ai Documentation](https://docs.vast.ai/)
- [vLLM Documentation](https://docs.vllm.ai/)


Liquid AI released LFM2-24B-A2B, a sparse 24B MoE model with 2.3B active parameters per token.

Liquid AI's LFM2 Just Dropped — Here's How to Run It on Vast.ai


Claude Code is one of the most capable agentic AI tools for software development – but it comes with a practical constraint: usage limits and spend. It's easy to run into daily or weekly quotas, and API costs can add up quickly.

Fortunately, there's a workaround.

### Ralph Loop: Running an Open-Source Coding Agent on Vast.ai

Instead of relying on API-limited services or local hardware, you can run open-source coding agents on Vast.ai for about **$1.50 per hour** – even continuously overnight! One example of this is Ralph, an agentic coding loop designed to implement software projects from a Product Requirements Document (PRD).

Ralph picks a user story, writes the code, runs tests, fixes any failures, and moves on to the next story. This process repeats until the project is complete and everything passes.

To jump straight in, follow our [Overnight Ralph Loop guide](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop) to get started. Otherwise, let's explore some more about what you can accomplish with Ralph and why.

### Why Run the Ralph Loop on Vast.ai?

There are several practical reasons to run the Ralph loop on a self-hosted model using our dedicated GPU infrastructure.

First, cost becomes more manageable. On Vast.ai, you pay for GPU time directly, at cost-effective rates that make overnight runs both predictable and affordable. Ralph is built for sustained workflows, so you can run it continuously without the usage caps of API-based tools like Claude Code.

Running Ralph on Vast.ai also gives you direct control over your development environment and supports stricter security and [compliance](https://vast.ai/compliance) needs. At the same time, you can match model size and GPU compute to the complexity of the task, ensuring your resources scale up only when the workload requires it.

### The Ralph Loop in Practice: Model & Use Cases

On Vast.ai, you can run the Ralph loop on Qwen3-Coder-Next, a large Mixture-of-Experts (MoE) model trained specifically for agentic coding tools like Claude Code, Aider, and Cline. It boasts a 256K context length and strong performance across multi-step coding tasks involving large codebases.

Once deployed, Ralph operates continuously and autonomously on real software projects, making it well suited for overnight development workflows. Common use cases include:

- Building a full CLI application
- Implementing REST APIs with authentication and validation
- Generating complete test suites for existing codebases
- Creating a web scraper with multiple site adapters, rate limiting, and data export

You simply define the requirements and allow the agent to carry out the work unattended.

### Getting Started with the Overnight Ralph Loop

With Vast.ai, a typical setup using Qwen3-Coder-Next on a 4x RTX 4090 instance costs approximately $1.50 per hour, with an overnight run often coming in **under $18** .

Our [Overnight Ralph Loop guide](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop) walks you through deploying the model, connecting the agent, and running the Ralph loop on Vast.ai. Start your first overnight run today!


Run autonomous coding agents on Vast.ai for about $1.50/hour with the Ralph Loop workflow.

Affordable Claude Code Alternative: Run Autonomous Coding Agents on Vast.ai


Welcome back to another round of updates at Vast.ai. We've kicked off the new year with improvements focused on expanding orchestration support, refining billing and serverless workflows, and introducing new templates and guides to help you deploy faster and more securely.

Another exciting update: we recently launched the [Vast.ai Startup Program](https://vast.ai/startup), designed to support early-stage teams building on our platform.

Startups can now get up to **$2,500 in free GPU credits** when they move their workloads to Vast.ai. With 24/7 priority support and access to Vast's full range of GPUs, the program delivers high-performance compute to power the next wave of AI innovation without long-term commitments. For more info, see our blog post [here](https://vast.ai/article/vast-ai-startup-program).

Not to mention we have a comprehensive new [Model Library](https://vast.ai/model-library), giving you a faster path from model selection to deployment.

## NVIDIA Cloud GPU Updates

### SkyPilot Enhancements

For teams using [SkyPilot](https://vast.ai/article/vast-ai-gpus-can-now-be-rentend-through-skypilot) to manage deployments, we've made updates that expand configuration options on Vast.ai. SkyPilot users can now filter capacity by Secure Cloud to ensure that workloads run only on verified datacenter-hosted GPUs.

We've also added full support for instance creation parameters via SkyPilot, enabling precise control over images, environment variables, disk size, pricing, startup commands, and other instance-level settings at launch. See [SkyPilot docs](https://docs.skypilot.co/en/latest/reference/config.html#vast) for full details.

### Fresh New Templates and Guides

Our [Model Library](https://vast.ai/model-library) introduces a model-first approach to deploying AI. Templates are organized by modality and use case and include recommended hardware presets and clear dedicated GPU pricing.

Here are some of our newest templates:

- [Kimi K2.5](https://vast.ai/model/kimi-k2.5) - An open-source, native multimodal agentic model that combines vision and language understanding with agentic capabilities.
- [LTX-2](https://vast.ai/model/ltx-2) - A DiT-based audio-video generation model that produces synchronized video and sound.
- [SGLang](https://cloud.vast.ai/template/readme/1fce7860e11071cae9c357fb1c8b7560) - A high-performance LLM inference framework with a hosted API server template.

And our latest guides:

- [dstack + vLLM](https://docs.vast.ai/examples/serving-infrastructure/dstack-vllm): Deploy language models using vLLM with dstack for automated provisioning and cost control.
- [DR-Tulu](https://docs.vast.ai/examples/mcp/dr-tulu): Deploy a research agent trained with an MCP server and web tools.
- [GLiNER2](https://docs.vast.ai/examples/ner/gliner2): Run GLiNER2 for named entity recognition with custom entity types.
- [BrowseSafe](https://docs.vast.ai/examples/ai-agents/browsesafe): Deploy BrowseSafe to detect prompt injection attacks in HTML.
- [Overnight Ralph Loop](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop): Budget-friendly alternative workflow to Claude Code.

### Other Improvements

If you missed it late last year, we also launched [Vast.ai Serverless](https://vast.ai/article/vast-ai-serverless-automated-gpu-scaling). It enables automatic, pay-per-use scaling for AI workloads through a fully serverless API on Vast's globally distributed GPU cloud.

## Our Commitment

With new SkyPilot enhancements, startup opportunities, a model-first deployment experience, and improved inference workflows, we're continuing to build a platform that gives you more control with less friction.

Need help? Contact our customer support team anytime at [support@vast.ai](mailto:support@vast.ai) or join our [Discord](https://discord.gg/vast) server for tips, real-time updates, and ongoing community discussion.

## Change Log

### SkyPilot Enhancements

- Advanced configuration options for SkyPilot on Vast.ai.
- Support for filtering by Secure Cloud (`datacenter_only`).
- Full support for instance creation parameters via SkyPilot.
- See [SkyPilot docs](https://docs.skypilot.co/en/latest/reference/config.html#vast).

### New Templates

- [Kimi K2.5](https://vast.ai/model/kimi-k2.5): Multimodal agentic model.
- [LTX-2](https://vast.ai/model/ltx-2): Video and audio generation model.
- [SGLang](https://cloud.vast.ai/template/readme/1fce7860e11071cae9c357fb1c8b7560): High-performance LLM inference engine.
- Find them in the [Model Library](https://vast.ai/model-library).

### Template Updates

- Base Image and PyTorch now support **CUDA 13.1** and updated NVML libraries.
- Ostris AI Toolkit updates.
- ComfyUI updates.
- Llama.cpp (ssl) updates.
- SGLang onstart script fix.
- Pinokio v6.x updates.

### New Guides

- [dstack](https://docs.vast.ai/examples/serving-infrastructure/dstack-vllm): Serving infrastructure.
- [DR-Tulu](https://docs.vast.ai/examples/mcp/dr-tulu): MCP.
- [GLiNER2](https://docs.vast.ai/examples/ner/gliner2): Named Entity Recognition.
- [BrowseSafe](https://docs.vast.ai/examples/ai-agents/browsesafe): AI agents and security.
- [Overnight Ralph Loop](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop): Budget-friendly alternative to Claude Code.

### Platform Improvements

- Fixed serverless GPU filter UI.
- Fixed serverless template selection UI.
- Billing Transaction History Invoice PDF Export now excludes failed and pending payments and other non-payment activity.
- Billing Transaction History now supports easy individual invoice download.
- Security improvements.
- Local volume deletion issue resolved.
- Copy progress bar issue resolved.
- Disk quota issues resolved.
- Machines missing geolocation resolved.
- Host CLI schedule maintenance command issue resolved.


Vast.ai's January and February 2026 updates include new SkyPilot controls, Startup Program credits, Model Library additions, fresh guides, and major platform fixes.

January & February 2026 Product Updates


### Introducing the Vast.ai Startup Program: AI Infrastructure Without the Burn Rate

Startups can now get [$2,500 in free GPU credits](https://vast.ai/startup) when they move workloads to Vast.ai.

To apply, get in touch with our team to share a bit about your project and compute needs. From there, we'll finalize your credit offer, help you configure the right setup, and get you up and running quickly. You'll also have access to **24/7 priority support** throughout the credit period.

Compute credits apply across Vast.ai's global GPU marketplace, with access to everything from consumer-grade cards to high-end accelerators like A100s and H200s.

The Vast.ai Startup Program gives startups an on-ramp to the same platform that teams rely on well beyond the early stage. That's because Vast.ai is designed to scale alongside you - without long-term commitments or runaway costs.

### Built for Real AI Workloads

From day one, Vast.ai supports the tools and workflows that teams already use. Launch preconfigured environments with Jupyter, Docker, RAPIDS, and Hugging Face to train deep learning models, run large-scale experiments, and process massive datasets on demand.

We offer a growing selection of ready-to-run AI models in our [Model Library](https://vast.ai/model-library), plus thousands of [templates](https://cloud.vast.ai/templates) to get started immediately - or deploy your own custom model anytime. Your team can even collaborate across departments and campuses, avoiding duplicated infrastructure and fragmented workflows.

Most importantly, with Vast.ai, your team can cut costs without cutting capability. Reduce infrastructure expenses by **up to 81%** compared to major cloud GPU providers, while paying only for what you need, when you need it.

Plus, most startups can migrate existing workloads with minimal changes and begin running jobs shortly after approval.

### Secure, Compliant, and Production-Ready

Vast.ai is **SOC 2 Type II** certified and designed to support all of your [security and compliance](https://vast.ai/compliance) needs from the start.

Run workloads in isolated instances with end-to-end encryption and full control over compute environments to keep your sensitive data protected. Our [Secure Cloud](https://cloud.vast.ai/?secure_cloud=true) option lets you filter for GPUs from our vetted datacenter partners for even higher security needs.

Notably, these safeguards support reproducible research via consistent environments, as well as compliance-friendly configurations aligned with GDPR and HIPAA requirements.

With [Vast.ai Serverless](https://vast.ai/article/vast-ai-serverless-automated-gpu-scaling), moving from development to production is even easier. Inference workloads run through a fully serverless API that automatically routes to the most cost-effective GPUs available, according to your performance requirements. This eliminates manual instance management and capacity planning - so you can leverage our distributed GPU fleet with minimal overhead.

### Get Started with Vast.ai Today

For early-stage teams, access to affordable GPU compute is crucial. But there's more to it than that. You also want infrastructure that scales as you grow and supports evolving security and compliance needs - all without locking you into long-term contracts.

The Vast.ai Startup Program helps you get there faster.

[Apply today](https://vast.ai/startup) to get up to 2,500 free compute hours and start building on Vast.ai on your own terms.


Startups can get up to $2,500 in free GPU credits when moving workloads to Vast.ai, with priority support and production-ready infrastructure.

Introducing the Vast.ai Startup Program


If you've ever tried generating AI video and thought, *This looks cool, but it's not what I asked for*, you're not alone!

**Wan 2.2** was designed to address scaling and efficiency challenges in video generation. It is among the first open-source video generation models to leverage a Mixture-of-Experts (MoE) architecture, enabling more efficient training. Created by Alibaba Tongyi Lab, the model gives creators more control over how their ideas turn into motion – following prompts more closely, with smoother movement and cinematic-quality results – without drastically increasing compute requirements.

On Vast.ai, Wan 2.2 is available in two variants in our [Model Library](https://vast.ai/article/vast-ai-model-library-choosing-best-ai-models):

- **[Text-to-video (T2V)](https://vast.ai/model/wan-2.2-t2v-a14b)** – Generating short videos directly from written prompts.
- **[Image-to-video (I2V)](https://vast.ai/model/wan-2.2-i2v-a14b)** – Transforming a static image into a dynamic video sequence.

Both models share the same underlying architecture, which is where things get interesting...

## What Makes Wan 2.2 Different

Most diffusion-based video models use a single neural network to handle the entire denoising process. Wan 2.2's MoE design, on the other hand, uses two specialized "experts" that strategically separate the video generation workflow:

- A **high-noise expert** that handles the early denoising stages, focusing on layout, composition, and overall scene structure and motion patterns.
- A **low-noise expert** that takes over later, refining video details and enhancing cinematic elements like lighting, contrast, and color tone.

This separation helps reduce common issues with video generation like unstable camera motion and inconsistent frames. It also makes the process more efficient. Even though each model has approximately **27 billion parameters** in total, only about **14 billion are active** per inference step.

The models automatically switch between experts via signal-to-noise ratio (SNR) thresholds – a measure of how much meaningful visual information remains versus how much noise is still present at each stage of generation. By activating the right expert at the right moment, Wan 2.2 avoids unnecessary computation. This keeps performance much more efficient, comparable to smaller single-expert models.

In short, you get more natural motion dynamics, superior visual fidelity, reduced artifacts, and less wasted compute.

## One Architecture, Two Workflows

While their underlying architecture is the same, the two Wan 2.2 variants have different creative entry points.

### Wan2.2 T2V A14B (FP8)

The text-to-video (T2V) model uses text prompts to generate **5-second clips at 480P or 720P** resolution, with granular control over lighting, composition, contrast, and color tone.

A few other capabilities of the T2V variant include:

- Superior performance on Wan-Bench 2.0 evaluations compared to leading commercial video generation systems
- Integration with Qwen models or DashScope API for enhanced prompt elaboration
- Support for parameter-type conversion, leading to improved inference speed

Overall, the text-to-video model is well suited for storyboarding, concept visualization, marketing clips, and rapid prototyping of video ideas from plain language.

### Wan2.2 I2V A14B (FP8)

The image-to-video (I2V) model begins with a static image input and turns it into a short video sequence. It offers both **480P and 720P output**, and can optionally support text prompts to guide content and motion in detail.

Some strengths of the I2V variant include:

- Handling a wide range of aesthetic preferences, from photorealistic to stylized
- Processing at **24 frames per second** (FPS) for smooth, high-definition output
- Automatic prompt derivation that makes it possible to generate videos from an image alone *without* text

The image-to-video model is especially useful for turning concept art, product photos, and illustrations into animated content.

## From Idea to Output: Designed for Real-World Use

Whether you start from a text prompt or from an image, Wan 2.2 is built for real-world creative and production workflows.

You don't even need enterprise-grade hardware to use the model. Wan 2.2's MoE architecture keeps things so efficient, you can achieve high-quality results on consumer GPUs like the RTX 4090. For larger workloads, multi-GPU inference is also an option. The model supports FSDP and DeepSpeed Ulysses for accelerated generation.

Wan 2.2 pairs particularly well with Vast.ai's flexible GPU infrastructure. On Vast, you can spin up exactly the hardware you need and run the model on a single GPU, or scale across multiple GPUs for faster generation as your video projects grow.

Both the [T2V](https://vast.ai/model/wan-2.2-t2v-a14b) and [I2V](https://vast.ai/model/wan-2.2-i2v-a14b) models are compatible with [ComfyUI](https://docs.vast.ai/documentation/serverless/comfyui-wan-2.2) workflows, making them easy to integrate into tools you may already be using – which are also available directly on Vast.ai.

Ready to start creating with Wan 2.2? Launch it on [Vast.ai](https://vast.ai/model-library) in just a few clicks and start generating cinematic video today!


Wan 2.2 uses a Mixture-of-Experts architecture to deliver efficient, high-quality AI video generation. Learn how it works and why it matters.

Wan 2.2 Explained: A New Approach to AI Video Generation


Wouldn't it be great if you could just think of a scene and instantly turn it into a video the way it appears in your head? Technology isn't quite there yet, but today’s most advanced AI video generation models are getting us closer.

Today we’re taking a look at **WAN 2.2** and **LTX-2**, two open-source/open-weights models that transform text and images into short-form video.

## What WAN 2.2 and LTX-2 Are — and How They Differ

From the outside, WAN 2.2 and LTX-2 are pretty similar tools. They’re both open-source/open-weights diffusion-based video generation models designed to turn images or text prompts into short video clips. Their underlying architecture, however, is very different.

### WAN 2.2: Prompt Fidelity and Cinematic Control

Developed by Alibaba Tongyi Lab, WAN 2.2 is built around a Mixture-of-Experts (MoE) architecture. Instead of using a single neural network to manage the entire denoising process, it employs two specialized “experts”: a **high-noise expert** for overall structure and layout, and a **low-noise expert** for refining textures and details like lighting and color tone.

Switching between these two experts means the model can allocate compute depending on what it needs to do at any given moment—focusing on broader structure first and finer details later. It also boosts efficiency by avoiding unnecessary computation.

WAN 2.2 comes in three main variants, each designed for different workflows:

- **Text-to-Video (T2V):** Generates 5-second video clips at 480P to 720P from a text prompt written in plain language. This is a flexible option for scenes where everything needs to be synthesized from scratch.
- **Image-to-Video (I2V):** Begins with a single image and turns it into a short video. It uses automatic prompt derivation to generate video from an image without text input at all, but can also support text prompts for more directed results.
- **Hybrid:** A compact model with 5 billion parameters that handles both text-to-video and image-to-video generation. It delivers high-definition results at up to 720P and 24 FPS, but is designed for users with lower VRAM.

The base WAN 2.2 models generate video only, without native audio output. However, there is a specialized speech-to-video version (WAN 2.2 S2V) that transforms static images and audio inputs into synchronized videos.

### LTX-2: Native Audio-Video Generation

Created by Lightricks, LTX-2 is a DiT-based (Diffusion Transformer) audio-video generation model. It produces audio and visuals together in one pass, keeping dialogue, lip movements, and ambient sound aligned coherently.

Its architecture is based on latent diffusion, which means the model works in a compressed version of the video first, before converting it into full resolution. This makes it more memory efficient and enables faster iteration, translating to quicker experimentation and lower hardware overhead.

LTX-2 can generate up to **~20 seconds of synchronized audio and video**, with support for high resolutions and high frame rates depending on configuration and available compute. The model offers fine-grained control options—such as LoRA-based customization and multimodal inputs including text, image, video, and audio—for precise creative direction.

This makes LTX-2 a highly flexible model. In short, it supports text-to-video, image-to-video, and native audio-visual generation, along with cross-modal workflows like audio-to-video, text-to-audio, and video-to-audio—all within a single model.

## Choosing the Right Model for Your Workflow

How the two models are designed directly affects what you experience as a user.

For instance, WAN 2.2's MoE design prioritizes structured generation and motion consistency. It boasts strong prompt adherence with high-fidelity output and is more likely to preserve scene intent across frames, sticking closely to what you asked for—albeit at the cost of slightly longer generation times.

LTX-2's latent diffusion approach emphasizes speed and accessibility. It’s faster to iterate with, easier to experiment on, and even offers native audio-video sync. However, it may require more prompt tuning to get exactly what you want.

### Choose **WAN 2.2** if you want:

- Cinematic or narrative-style clips where composition and camera motion are critical
- Strong prompt fidelity for complex scenes with multiple elements
- More deliberate, production-oriented outputs and professional video content

### Choose **LTX-2** if you prefer:

- Rapid prototyping of video concepts and creative exploration for lengthier scenes
- Visual storytelling or character-driven video with synchronized dialogue or sound
- A lighter and more iterative workflow where speed matters more than precision

Both models also integrate with ComfyUI, so you can jump right into testing them out with an intuitive node-based visual workflow.

## Final Thoughts

Neither WAN 2.2 nor LTX-2 is objectively superior to the other. The two open-source/open-weights models are designed for different kinds of workflows and creative goals. The best way to get a feel for them is to actually try them out.

The good news is that both models run well on high-end consumer GPUs, making them far more accessible than many people might expect. With Vast.ai, it’s even easier: you can spin up the right hardware on demand and experiment on your own terms, paying only for the compute you need—and save up to 80% over traditional clouds.

Try [WAN 2.2 T2V](https://vast.ai/model/wan-2.2-t2v-a14b) and [WAN 2.2 I2V](https://vast.ai/model/wan-2.2-i2v-a14b), or [LTX-2](https://vast.ai/model/ltx-2) (or both!) in our [Model Library](https://vast.ai/model-library), and build your own creative pipeline on Vast.ai today.


A practical comparison of WAN 2.2 and LTX-2—two open-source/open-weights AI video models—covering architecture, strengths, and use cases.

WAN 2.2 vs. LTX-2: Which AI Video Model Should You Use?


dstack is an open-source GPU orchestration platform that automates instance provisioning and lifecycle management across cloud providers. This guide walks through using dstack with Vast.ai as the backend, combining declarative infrastructure with competitive GPU marketplace pricing.

## What Makes dstack Special?

Key features include:
- **Infrastructure as Code**: Define GPU requirements, pricing limits, and workloads in YAML
- **Automatic Provisioning**: dstack finds and provisions the best available instances
- **Cost Controls**: Set `max_price` to cap hourly costs automatically
- **Built-in Proxy**: Access services through dstack's authenticated endpoint

## What's in the New Guide

Our latest documentation walks you through deploying language models with dstack and vLLM on Vast.ai.

**Complete Setup Instructions**
- Installing and configuring dstack with your Vast.ai API key
- Starting the dstack server and CLI
- Creating service configurations for vLLM deployments

**Working Service Configurations**
- Ready-to-use YAML for deploying Qwen3-30B-A3B
- GPU memory and pricing parameters
- Real deployment outputs showing what to expect

**API Integration Examples**
- Python code using the OpenAI SDK
- cURL examples for testing
- Streaming response implementations

## Why dstack + Vast.ai?

The combination gives you orchestration on top of a GPU marketplace:

- **Simplified Workflow**: No manual instance hunting or environment setup
- **Cost Optimization**: dstack finds the cheapest instance meeting your requirements
- **Flexible Pricing**: Access Vast.ai's competitive rates with automatic cost caps
- **Production-Ready API**: vLLM provides OpenAI-compatible endpoints

The guide demonstrates deploying Qwen3-30B-A3B on an H100 80GB—provisioned with a single command.

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Teams** wanting reproducible, version-controlled GPU deployments
- **Developers** building LLM applications who want simple infrastructure management
- **Anyone** tired of manually provisioning and configuring GPU instances

## Get Started

The complete guide is now available in our documentation:

**[Read: Deploy LLMs with dstack and vLLM on Vast.ai →](https://docs.vast.ai/examples/serving-infrastructure/dstack-vllm)**

Whether you're new to GPU orchestration or looking for a better way to manage LLM deployments, this guide provides everything you need to get started with dstack on Vast.ai.

Ready to try it? [Sign up for Vast.ai](https://vast.ai) and follow the guide to deploy your first model.


Learn how to deploy language models with dstack and vLLM on Vast.ai, combining declarative infrastructure with competitive GPU marketplace pricing.

Deploy LLMs with dstack on Vast.ai


DR-Tulu is AI2's open-source research agent—an 8B model that autonomously plans research strategies, searches the web, reads pages, and synthesizes comprehensive answers with citations. Unlike typical LLMs with tools bolted on, DR-Tulu was trained end-to-end with its MCP tools, making it a compelling open alternative to proprietary research APIs. This guide walks through deploying DR-Tulu on Vast.ai, where you can run the full agent stack on affordable GPU instances and integrate it into your own agentic applications.

## What Makes DR-Tulu Special?

Key features include:
- **Native MCP Integration**: Trained with web search and page reading tools from the start
- **Autonomous Research**: Plans strategies, gathers sources, and synthesizes cited answers
- **Efficient Architecture**: 8B parameters with strong research capabilities

## What's in the New Guide

Our latest documentation walks you through deploying DR-Tulu on Vast.ai with a split architecture: vLLM for GPU inference on Vast.ai, MCP backend running locally for tool execution.

**Complete Deployment Instructions**
- Instance selection with the Vast.ai CLI
- vLLM configuration with the correct version (critical for compatibility)
- MCP backend setup with required API keys
- Workflow configuration to connect the components

**Three Ways to Use DR-Tulu**
- Interactive chat for exploratory research
- Batch evaluation against built-in datasets
- Python API for programmatic integration

## Why This Architecture?

The split deployment keeps GPU-intensive inference on Vast.ai while running tool orchestration locally. This means:

- Your API keys (Serper, Jina) stay on your machine (or wherever you deploy the MCP backend)
- Model inference doesn't require sending data to third-party APIs
- You can modify the MCP backend without redeploying the model
- Lower Vast.ai costs since you only need GPU for inference

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Researchers** who need cited, source-backed answers at scale
- **Developers** building applications that require web research capabilities
- **Teams** evaluating open alternatives to proprietary research APIs

## Get Started

The complete guide is now available in our documentation:

**[Read: Running DR-Tulu on Vast.ai →](https://docs.vast.ai/examples/mcp/dr-tulu)**

Whether you're exploring agentic AI architectures or ready to deploy your own research assistant, this guide provides everything you need to get DR-Tulu running on Vast.ai infrastructure.

Ready to try it? [Sign up for Vast.ai](https://vast.ai) and follow the guide to deploy your first instance.


Learn how to deploy DR-Tulu, AI2's open-source research agent, on Vast.ai for autonomous web research with citations.

Deploy Deep Search with DR-Tulu on Vast.ai


As 2025 comes to a close, we're looking back at the milestones and updates that shaped the past year for us here at Vast.ai.

What a year it's been! From new hardware to major security achievements and product launches, here are our top moments of 2025.

## Expanding Our Global GPU Fleet

The past year saw some of the biggest hardware expansions in Vast.ai's history.

In 2025, we added NVIDIA's latest RTX 5000-series GPUs – including the RTX 5090 – followed by high-performance machines like the RTX PRO 6000 WS and the H200 and B200. These additions supported higher-throughput workloads and more efficient AI inference across a [wide range](https://cloud.vast.ai/) of hardware profiles.

Another highlight: our RTX 5090 inventory surpassed 1,000 machines this year!

## Security and Compliance Milestones

This year, Vast.ai achieved both **SOC 2 Type I** and **Type II** certification. These independent audits validated that the security infrastructure we've built over 6+ years of trusted service continues to meet rigorous compliance standards.

We take this responsibility seriously. Data privacy and integrity remain core priorities in everything we build. As we continue to advance our [security and compliance](https://vast.ai/compliance) roadmap, we're committed to upholding the trust you place in us.

## New Templates and Tools

We introduced a number of ready-to-use templates this year, to help you move from idea to execution quickly and seamlessly. Some notable ones included:

- [Langflow + Ollama](https://docs.vast.ai/langflow-ollama) for visual AI workflow design
- [Ostris AI Toolkit](https://cloud.vast.ai/?ref_id=62897&template_id=81e6e9dd266d9bdbcbb0fd48b9afe357) for custom Stable Diffusion and LoRA training
- [Open-Sora](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Open-Sora) and [LTX Video](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%2520%252B%2520LTX%2520Video) (ComfyUI) for video generation workflows
- [ComfyUI ACE Step](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ACE%2520Step%2520Music%2520Generation) for AI-assisted music creation
- [InvokeAI](https://cloud.vast.ai/?ref_id=62897&template_id=c309df384010cf9d3dc9abb9e0a5a6fd) for image generation in a professional web interface
- [Dia 1.6b TTS](https://cloud.vast.ai/?ref_id=62897&template_id=6ba071acb5f590913c5eaf6c2dc3d0db) for lifelike text-to-speech

But the biggest update of the year was still ahead...

## Introducing Vast.ai Serverless

The headline story of 2025 was the [recent launch](https://vast.ai/article/vast-ai-serverless-automated-gpu-scaling) of **Vast.ai Serverless** – a fully automated GPU compute layer designed for AI inference at scale.

With our Serverless offering, you can easily run workloads without managing instances on your own or overpaying for capacity. You simply send inference requests and the platform automatically selects the right hardware, routes workloads across regions as needed, and predictively scales up and down based on usage.

In short, Vast.ai Serverless gives you access to a diverse, globally distributed GPU fleet with a serverless model that applies predictive optimization and flexible scaling – with none of the overhead. We're thrilled to finally make truly serverless AI inference a reality on Vast.ai!

## Looking Ahead

To all of our users and the wider Vast.ai community, thank you for being part of our journey this year. It's been an incredible year of growth, and we're grateful to everyone building and innovating with us.

Here's to an even stronger year ahead!


As 2025 comes to a close, we're looking back at the milestones and updates that shaped the past year for us here at Vast.ai.

Vast.ai's 2025 Year in Review


With so many AI model libraries to choose from, making the right decision requires research and testing. Luckily, [Vast.ai](http://vast.ai) partners with some of the leading innovators across a variety of generative and assistive models. Along with our GPU rentals, we empower businesses and individuals to produce their best work without latency issues or breaking the bank. Not only are these AI models reliable, they are also very affordable. You can scale production and workload with ease. Whether looking for an image generator, video production, audio, text or computer vision research, these are some of the best AI models available.

Model Library is Vast's new, model-first way to deploy AI. Instead of starting from the runtime-level (like vLLM, ComfyUI, or PyTorch), you can browse and discover model-level templates, sorted by modality and use case, then launch a running instance in seconds with full custom control over the underlying runtime, parameters, and infrastructure.

Every template includes recommended hardware presets and clear dedicated GPU rental pricing, so you can quickly get started with the right cost/performance setup and deploy with confidence, or tweak to fit your needs.

Coming soon: more runtimes, more model variations, and Vast Serverless model templates.

## The Best AI Image Generation Models

With text-based prompts, you can produce images from a range of popular AI models. From hyper-realistic renderings to cartoonish depictions, these models are among your best bets for generating images.

- **[HiDream I1 Full](https://vast.ai/model/hidream-i1-full)**: Released in May 2025, HiDream produces images in a variety of styles. Whether you want realistic photo imagery or stylized cartoon graphics, HiDream can deliver. With speed and precision, it can produce images without sacrificing quality.

- **[Qwen Image (FP8)](https://vast.ai/model/qwen-image)**: With a specific focus on generating images containing text, Qwen Image can produce high-quality images that provide a strong eye for detail. This is true not only of images with English characters, but even intricate characters for multilingual projects. It has a particular focus on generating images containing Chinese characters.

- **[FLUX.1 [dev]](https://vast.ai/model/flux.1-dev)**: Whether prototyping concepts, marketing, or digital art, FLUX.1 is an open-source tool that rivals many closed-source competitors in its functionality and precision. Developed by Black Forest Labs, this 12-billion-parameter image generator serves for a range of use cases from concepting to marketing materials and scientific visualization.

- **[Juggernaut XI v11](https://vast.ai/model/juggernaut-xi-v11)**: With a particular emphasis on photorealistic images, Juggernaut XI v11 is trained to provide exceptional detail on challenging human features like hands and facial expressions. It also provides exceptional detail for landscapes and other detailed imagery that have historically fallen into the realm of "uncanny valley."

- **[RealVisXL V5.0](https://vast.ai/model/realvisxl-v5.0)**: Developed by Evgeny, RealVisXL renders photorealistic images and produces anatomically accurate and high-fidelity visuals thanks to inference parameters like DPM++ SDE Karras sampling. This makes it equally adept at producing architectural designs and photorealistic images.

- **[Stable Diffusion XL Base 1.0](https://vast.ai/model/stable-diffusion-xl-base-1.0)**: Developed by Stability AI, Stable Diffusion XL Base improves image quality by providing two models. With a brainstorming mode in its base model, you can parse through a variety of images and then hone in on specific imagery with the more detailed refinement module. This holistic approach provides you with everything you need to craft the specific images you need.

## The Best AI Video Generation Models

[Vast.ai](http://vast.ai) offers some of the best video generation options available on the market. With options from a variety of AI engines and developers, you can produce stunning videos that rival cost-and-labor-intensive CGI and video productions.

- **[LTX Video](https://vast.ai/model/ltx-video)**: The first Diffusion Transformer (DiT)-based video generation model, LTX Video, produces high quality videos at resolutions and framerates that exceed much of the competition. Videos are available at 30 FPS and a 1216×704 resolution. This gives it an advantage over many of its competitors.

- **[Mochi 1 Preview](https://vast.ai/model/mochi-1-preview)**: Specializing in photorealistic video generation, Mochi 1 has 10 billion parameters that are built on an innovative Asymmetric Diffusion Transformer architecture. This helps it process over 44,000 visual tokens to produce high-quality videos.

- **[Wan2.2 I2V A14B (FP8)](https://vast.ai/model/wan-2.2-i2v-a14b)**: Wan2.2 I2V generates 720p videos from static image prompts. With a high-noise expert option for background imagery and a low-noise expert option for movement, you can make still images come to life with striking detail.

- **[Wan2.2 T2V A14B (FP8)](https://vast.ai/model/wan-2.2-t2v-a14b)**: Released in July 2025, Wan2.2 T2V generates 5-second videos at both 480P and 720P resolutions. With cinematic aesthetics and complex motion capabilities, it provides greater detail than many previous open-source and commercial models.

## The Best Audio Generation AI Models

Whether you need to produce a podcast, narrate an audiobook, engineer music production, or just explore the possibilities of ai-generated sounds, [Vast.AI](http://vast.ai) provides affordable audio generation options from some of the best-known models available.

- **[ACE Step V1 3.5B](https://vast.ai/model/ace-step-v1-3.5b)**: ACE Step V1 produces realistic audio thanks to its 3.5-billion-parameter model. It can be leveraged for a range of use cases: podcasts, audiobooks, and a wide range of music production prompts.

- **[Dia 1.6B](https://vast.ai/model/dia-1.6b)**: Developed by Nari Labs, Dia 1.6B is an audio generation model that specializes in multi-speaker production. This works great for podcasts and other text-to-audio prompts and excels at inflecting speakers' voices and producing sighs, laughter, and other non-verbal flourishes.

## The Best Text Generation AI Models Available

Whether you need agentic support, assistance on scientific research, or multilingual text translation, there are many incredible choices available. These are the best text generation AI models available from [Vast.AI](http://vast.ai).

- **[GLM 4.6](https://vast.ai/model/glm-4.6)**: This model provides support for language models that work on multi-step workflows. With over 200,000 tokens and sophisticated reasoning logic, it can excel at anything from coding applications to agentic capabilities.

- **[Kimi K2 Thinking](https://vast.ai/model/kimi-k2-thinking)**: Released under a Modified MIT License, Kimi K2 Thinking supports both commercial and research applications. Its chain-of-thought reasoning allows it to operate through hundreds of steps without degrading its response quality.

- **[DeepSeek V3.2 Exp](https://vast.ai/model/deepseek-v3.2-exp)**: Built for complex research and debugging scenarios, DeepSeek V3.2 Exp is an incredible assistant for very labor-intensive processes. With the ability to parse and analyze large bodies of text, you can automate some of the most tedious parts of your day in seconds.

- **[Kimi K2 Instruct 0905](https://vast.ai/model/kimi-k2-instruct-0905)**: With over a trillion parameters, Kimi K2 Instruct excels at language models including agentic intelligence and software development. Its massive 256k token window provides advanced decision-making capabilities for a range of issues and use cases.

- **[Qwen3 235B A22B Thinking 2507](https://vast.ai/model/qwen3-235b-a22b-thinking-2507)**: Developed by Alibaba, this 235-billion-parameter Mixture-of-Experts (MoE) language model is specifically engineered for extended reasoning tasks by activating 8 experts and 22 billion parameters per token. This makes it ideal for complex projects including mathematics, scientific research, and code generation.

- **[Qwen3 Coder 480B A35B Instruct](https://vast.ai/model/qwen3-coder-480b-a35b-instruct)**: This 480-billion-parameter Mixture-of-Experts (MoE) model developed by Alibaba is designed for advanced code generation and is optimized for actionable code output by operating exclusively in an efficient non-thinking mode.

- **[DeepSeek R1 0528](https://vast.ai/model/deepseek-r1-0528)**: This advanced reasoning language model developed by DeepSeek AI utilizes advanced chain-of-thought logic for complex mathematical and logical problems. This makes it a great choice for tool and API integrations.

- **[DeepSeek V3.1](https://vast.ai/model/deepseek-v3.1)**: This hybrid language model operates in both thinking and non-thinking modes. This dual-mode architecture is equally suited for deep reasoning with visible thought processes or generating fast responses without intermediate reasoning.

- **[GPT-OSS-120b](https://vast.ai/model/gpt-oss-120b)**: GPT-OSS-120b is an open-weight reasoning model from OpenAI designed for production use cases. With a range of low, medium, and high reasoning settings it balances speed and accuracy for agentic functions.

- **[GPT OSS 20b](https://vast.ai/model/gpt-oss-20b)**: Built for lower latency and specialized use cases. For everything from web browsing to function calling and API integration tasks, GPT OSS 20b is a great choice for agentic support.

- **[Llama 4 Maverick 17B 128E Instruct](https://vast.ai/model/llama-4-maverick-17b-128e-instruct)**: With seamless processing of both text and visual inputs, Llama 4 Maverick 17B supports up to a 10 million token context length and can process up to 5 input images simultaneously. It supports 12 languages for a wide array of agentic functions including English, Spanish, Tagalog, French, German, Hindi, Indonesian, Italian, Portuguese, Arabic, Thai, and Vietnamese.

- **[Llama 4 Scout 17B 16E Instruct](https://vast.ai/model/llama-4-scout-17b-16e-instruct)**: Released in April 2025, Scout 17B 16E Instruct employs a mixture-of-experts (MoE) and has a total of 109 billion parameters. It functions as a great assistant for analyzing data, parsing large quantities of data and is compatible with 12 languages including English, Spanish, Tagalog, French, German, Hindi, Indonesian, Italian, Portuguese, Arabic, Thai, and Vietnamese.

- **[Qwen3 VL 235B A22B Instruct](https://vast.ai/model/qwen3-vl-235b-a22b-instruct)**: The most powerful vision-language model in the Qwen series, VL 235B A22B Instruct combines 236 billion parameters across visual understanding, agent applications, context processing, and multimodal reasoning tasks. It excels at recognizing a range of items from landmarks, to plants and animals, to works of art and celebrities. This makes it a great research tool for imaging, robotics, and even technical documentation.

## The Best Computer Vision Models Available

When it comes to choosing an AI model that can interpret visual elements from real-world scenarios, these are among the best:

- **[DeepSeek OCR](https://vast.ai/model/deepseek-ocr)**: This vision-language model specializes in optical character recognition and document understanding. This makes it great for extracting text from images or scanned documents, processing PDFs, and much more.

- **[GLM 4.5V](https://vast.ai/model/glm-4.5v)**: Whether summarizing longform videos, analyzing data, or scanning documents for text, GLM 4.5V provides support for advanced reasoning tasks. With so many use cases, you can even toggle between quick responses or deeper analysis of data to make it work for you the way you want.

- **[InternVL3 78B](https://vast.ai/model/internvl3-78b)**: Whether analyzing technical documentation, interpreting results from medical images, or providing agentic support, InternVL3 78B has many use cases from text-only applications to a mixture of visual applications.

- **[Llama 4 Maverick 17B 128E Instruct](https://vast.ai/model/llama-4-maverick-17b-128e-instruct)**: Featuring 17 billion activated parameters distributed across 128 total experts, this model combines text and image understanding capabilities within a unified architecture. This model provides everything from an assistant-like conversational experience to image captioning, document analysis, and multilingual support.

- **[Llama 4 Scout 17B 16E Instruct](https://vast.ai/model/llama-4-scout-17b-16e-instruct)**: With efficiency being front and center, this model runs much more leanly than Maverick. This gives it a more conversational tone while still providing agentic support, robust analysis of charts, multilingual functionality.

- **[Qwen3 VL 235B A22B Instruct](https://vast.ai/model/qwen3-vl-235b-a22b-instruct)**: Qwen3's most powerful vision-language model, it provides incredible PC and mobile GUI navigation for automation tasks, visual coding support across a range of coding languages, and agentic support.

---

If you're ready to take your AI workflows to the next level, [Vast.ai](http://vast.ai) is here to help. From an [AI model library](https://vast.ai/model-library) with a host of tools and use cases to [GPU servers](https://cloud.vast.ai/) that can accelerate your research, we're here to help empower and scale your discovery every step of the way.

---


Explore Vast.AI's comprehensive model library featuring the best AI models across image generation, video production, audio, text, and computer vision.

Vast.AI's Model Library: Choosing from the Best AI Models Available


# Running ComfyUI on Vast.ai Serverless: Simple, Scalable Image Generation

ComfyUI has become one of the most popular open-source AI tools for image generation, for good reason.

Rather than typing a prompt and hoping for the best, you build the entire process yourself through a **node-based interface that gives you full, visual control** over how images are created. Every part of the workflow – from models, prompts, and parameters – can be adjusted or branched however you like, and workflows themselves are also shareable and reusable.

While ComfyUI runs well locally for smaller projects, it's a whole other challenge to use it for heavier custom workflows or at production scale. Complex ComfyUI graphs can exceed consumer GPU memory or demand more throughput than a single machine can handle – or simply take too long to iterate on locally.

That's where **Vast.ai Serverless** comes in.

### ComfyUI Meets Vast.ai Serverless

Vast.ai Serverless makes it easy to run your ComfyUI workflows at scale, thanks to a ready-to-use template that handles all of the setup for you – and no need to manage GPUs or monitor capacity.

Simply send requests to ComfyUI and have your generated assets automatically uploaded to S3-compatible storage. The template will return pre-signed URLs in response to requests, as well as detailed process updates while the workflow runs.

It's a straightforward way to integrate ComfyUI into your applications or pipelines without worrying about infrastructure overhead.

### What Comes in the ComfyUI Serverless Template

Our ComfyUI Serverless template provides [everything you need](https://www.google.com/url?q=https://docs.vast.ai/documentation/serverless/comfy-ui&sa=D&source=editors&ust=1766188232265630&usg=AOvVaw0k4v0rLcf_0MltEhly3_ZW) right out of the box:

- **ComfyUI** and its **API wrapper** (which manages asset uploads)
- **Stable Diffusion 1.5** for benchmarking
- **PyWorker** for processing JSON workflows
- **Provisioning script** for custom configuration

The template saves you from building or maintaining your own Docker image and ensures you get a consistent environment every time a worker spins up. Once the template is in place, several capabilities come built into the workflow by default.

#### 1. Built-In Asset Handling with S3 Storage

Generated assets are automatically uploaded to your private S3-compatible storage buckets, and pre-signed URLs provide secure, temporary access to download results directly. This keeps storage and retrieval simple, even as workloads scale.

Just add your credentials once in your account settings, and every worker can use them. If you need different buckets for different jobs, you can override these variables on a per-request basis.

#### 2. Flexible Configuration When You Need It

With ComfyUI Serverless, you can keep things simple or tailor your setup with optional configurations. For instance:

- Add a webhook to be notified after generation completion or failure
- Use a custom PyWorker git repository URL
- Adjust benchmark settings to better reflect your workload

With these configuration tools, you're free to adapt the template to your unique needs.

#### 3. Distributing Work Through Benchmarking

Each worker runs a quick benchmark when it initializes, to validate GPU performance and calculate a performance score. This score is what determines how requests are distributed across workers.

The template includes a default Stable Diffusion 1.5 benchmark for this purpose, but you can configure it to match your actual workload complexity or even supply your own custom benchmark.

### Try It Out Interactively Before Going Serverless

The ComfyUI Serverless template is designed for serverless execution, of course, but you can test it as an interactive instance first. Interactive testing helps confirm that everything works as expected before you start sending requests through Serverless.

You'll have to enable port forwarding to access applications over the public interface, but the benchmarking process will be visible in the instance logs either way.

### A Smoother Path to Production-Scale ComfyUI

Vast.ai Serverless and our ComfyUI template lets you go from a local workflow to a scalable, on-demand system with minimal setup. The environment is preconfigured – with customizable adjustments as needed – and assets are handled automatically while workloads are routed to the right GPUs based on real performance.

For creative work beyond images, we offer Serverless templates for [text-to-video](https://www.google.com/url?q=https://docs.vast.ai/documentation/serverless/comfyui-wan-2.2&sa=D&source=editors&ust=1766188232275066&usg=AOvVaw33TRQTBXX80tZ040XucSdk) and [text-to-music](https://www.google.com/url?q=https://docs.vast.ai/documentation/serverless/comfyui-acestep&sa=D&source=editors&ust=1766188232275242&usg=AOvVaw3Dnzq-bbQYJjA76I3vHcU3) workflows, as well.

Ready to see ComfyUI in action? Just launch the [ComfyUI Serverless template](https://www.google.com/url?q=https://cloud.vast.ai/?ref_id%3D62897%26creator_id%3D62897%26name%3DComfyUI%2520(Serverless)&sa=D&source=editors&ust=1766188232275749&usg=AOvVaw0clIXR2u6vdhxqRdqoPmHE) on Vast.ai and start running workflows at scale. It's that easy.


ComfyUI has become one of the most popular open-source AI tools for image generation, for good reason.

Running ComfyUI on Vast.ai Serverless: Simple, Scalable Image Generation


Vast.ai is thrilled to announce the launch of our Serverless offering for GPU workloads, enabling automatic, pay-per-use scaling for AI inference without hyperscaler pricing or fixed infrastructure constraints.

With [Vast.ai Serverless](https://vast.ai/products/serverless), users run inference workloads through a fully serverless API on Vast's globally distributed GPU cloud. That means no manual instance management and no capacity planning. Instead, the platform automatically applies predictive optimization and flexible scaling, with access to a diverse GPU fleet -- so teams can deploy production AI systems that scale when needed and yet stay cost-efficient at all times.

How does this serverless model work in practice? Let's take a closer look!

<div style={{ margin: "48px auto", maxWidth: "960px", width: "100%" }}>
  <div style={{ position: "relative", paddingBottom: "56.25%", height: 0, overflow: "hidden", borderRadius: "12px", background: "#000" }}>
    <iframe
      src="https://www.youtube-nocookie.com/embed/0PAPzSZa3tA?rel=0"
      title="Vast.ai Serverless overview"
      allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
      allowFullScreen
      referrerPolicy="strict-origin-when-cross-origin"
      loading="lazy"
      style={{ position: "absolute", top: 0, left: 0, width: "100%", height: "100%", border: 0 }}
    ></iframe>
  </div>
</div>

### Where GPU Cloud Meets Serverless AI

Vast.ai Serverless provides access to a wide range of GPUs, from RTX-class consumer GPUs to enterprise-grade machines like A100s, H100s, and B200s. Rather than managing individual instances, all you have to do is define performance targets, and Vast handles the rest.

Behind the scenes, Vast.ai continuously benchmarks GPUs across a global network of more than **17,000 GPUs hosted by 1,400+ providers in over 500 locations worldwide**. As workloads run, the platform dynamically selects, provisions, and routes jobs to the most efficient hardware available at that moment.

The key is predictive optimization.

### Predictive Optimization Built for Real-World Workloads

Most serverless platforms react to demand after it appears. Vast.ai Serverless goes further.

Our predictive optimization feature analyzes historical usage patterns, real-time load, and ongoing market benchmarking to anticipate demand before it peaks. Based on those signals, the platform **proactively provisions GPU workers** that balance cost and latency -- ready to activate on demand rather than scrambling to scale once performance has already degraded.

These reserve workers help avoid the laggy cold starts and unpredictable costs common with other GPU serverless offerings. Not to mention having to pay for excess GPU capacity that isn't actively being used.

### Multiple Workergroups per Endpoint

Unlike traditional serverless systems that restrict endpoints to a single hardware profile, Vast.ai Serverless [supports multiple Workergroups per Endpoint](https://docs.vast.ai/documentation/serverless/architecture). A Workergroup is a collection of GPU workers that are managed as a logical unit to automatically adjust capacity to meet demand.

In our Serverless offering, each Workergroup in an Endpoint can specify different GPU types or hardware configurations. This lets a single API endpoint serve workloads using whichever Workergroup is most cost-effective or the best fit for your performance targets. For example, lighter requests might route to consumer GPUs, while heavier inference jobs scale onto H100s -- with no manual intervention needed.

In practice, this makes it possible to optimize performance and cost-efficiency in real time within a single deployment. And that brings us to pricing.

### Lowest-Cost Transparent Serverless Pricing

Vast.ai Serverless is the **lowest-cost autoscaling GPU cloud** on the market today.

Our Serverless workloads are billed per second, with support for On-Demand, Interruptible, and Reserved pricing. There are [no tiers and no limits](https://docs.vast.ai/documentation/serverless/pricing) -- and all you need is $5 to get started.

Because Vast draws from a competitive global market of GPU providers, pricing reflects real supply and demand rather than preset SKUs. As demand scales up, Vast.ai Serverless automatically selects the most cost-effective GPUs available. When demand drops, resources are released immediately, and billing stops.

As a result, teams can run production workloads at a fraction of the cost compared to traditional providers with centralized infrastructure. However, lower costs don't mean lower standards. Security and compliance remain core to our platform.

### Enterprise-Ready Security and Compliance

Vast.ai is built with security and compliance as foundational principles. Our platform is backed by **SOC 2 Type II** [certification](https://vast.ai/article/vast-soc2-typeII-certification), with ongoing audits every 12 months to ensure continual coverage.

For customers with the highest security requirements, Vast.ai offers a tiered compliance structure with our [Secure Cloud](https://cloud.vast.ai/?secure_cloud=true) offering. In the Secure Cloud, workloads run on isolated instances with direct SSH, CLI, and API access, hosted by our vetted datacenter partners that meet **ISO 27001** standards at minimum. Many partners maintain additional certifications. (Full details are available on our [compliance page](https://vast.ai/compliance).)

Other enterprise security features -- including private VPN access, optional audit trails, and enterprise-grade compliance support -- can be enabled as needed.

In all cases, data sovereignty remains fully in your control. Models, data, and workloads persist only as long as you choose and can be deleted at any time.

### How to Automatically Scale GPU Compute for AI Inference

With Vast.ai Serverless, you can go from [zero to compute](https://docs.vast.ai/documentation/serverless) in seconds. In short, here's how the process works:

1. Bring your own machine learning model
2. Send an inference request
3. Vast.ai Serverless routes and provisions compute on ready GPUs
4. GPU executes and returns results
5. Serverless predictively scales up and down based on usage

That's all there is to it! From request to response, Vast.ai handles all of the heavy lifting. You also gain access to ample metrics and debugging tools, including logs and Jupyter/SSH.

### Get Started with Vast.ai Serverless

Our Serverless offering makes it easy to run AI inference without managing infrastructure or overpaying for capacity. Vast.ai Serverless gives you a simple and straightforward path from experiment to production -- and it's ready when you are.

Leverage Vast's distributed GPU fleet on your own terms. Check out our [Serverless Overview](https://docs.vast.ai/documentation/serverless) and start building today!


Announcing Vast.ai Serverless for GPU inference with predictive optimization, multi-workergroup endpoints, pay-per-second pricing, and enterprise-grade compliance.

Vast.ai Serverless: Automated GPU Scaling for AI Inference - Without the Overhead


All startups rely on infrastructure that can grow with them, but AI startups in particular face an added challenge: supporting compute-heavy workloads and iterating fast without slowing progress or overspending. 

Whether you're bootstrapping or venture-funded, it's always best to maximize the return on every GPU hour and dollar spent. But how do you do that? 

For AI startups, the right GPU infrastructure is absolutely critical. Training large models and running compute-heavy experiments can quickly consume budgets if you're not careful. Traditional cloud providers often charge a premium for the level of compute that AI workloads demand, while locking users into long-term contracts or inflexible pricing models. 

Buying your own dedicated hardware rarely solves the problem. The high upfront cost can be prohibitive for early-stage teams, and delays can stall progress. You also risk being tied to specific hardware that may be inadequate for your startup's next phase of development.

At Vast.ai, we take a different approach. Our cloud GPU rental platform offers high-performance machines with flexible spot pricing, so you can get the compute you need, when you need it, as affordably as possible.

Let's take a closer look at what makes Vast.ai's [spot GPU rental](https://cloud.vast.ai/) an ideal choice for AI startups.

## The Cost Advantage of Spot GPUs for AI Startups

To keep cloud costs under control as an AI startup, it's important to carefully consider which [pricing model](https://docs.vast.ai/documentation/instances/choosing/instance-types) is right for you. Vast.ai offers on-demand GPU instances as well as spot instances – each balancing cost and reliability in different ways.

**On-demand instances** offer guaranteed uptime at a fixed price, so you pay a slightly higher rate to access a GPU without the risk of sudden interruption. This option is well suited for continuous training jobs or production inference, or basically any workload where you need to prioritize stability over cost.

**Spot instances**, also known as interruptible instances on [Vast.ai](https://vast.ai), let you rent unused GPU capacity at lower rates determined by bidding. The highest bid wins the instance. The trade-off is the possibility that your instance may be interrupted or preempted with little to no notice, requiring you to restart the job later (when you regain priority) or move it to another GPU. This option is nevertheless ideal for workloads that can tolerate occasional interruptions in exchange for significant cost savings. 

For AI startups, the main draw of spot instances is simple: they deliver the same high-performance GPUs at a drastic discount. On Vast.ai, interruptible instances can be **up to 80% cheaper** than traditional cloud rates.

These savings can give early-stage AI startups the momentum to scale faster. The next move is scaling smarter.

## How Spot GPUs Help AI Startups Scale Smarter

As a startup, you want to accelerate your AI development thoughtfully and efficiently. Lowering your cloud GPU costs is the first step, and then you can reinvest those savings to accelerate growth.

With spot pricing, access to high-performance GPUs remains cost-effective. As a result, larger-scale training becomes affordable enough for early-stage AI startups to run more jobs and test new ideas efficiently – ultimately iterating faster. 

Additionally, because spot instances are billed on a pay-as-you-go basis, you can run workloads on cutting-edge GPUs that may otherwise be out of reach – like the H200 – for limited periods as needed, and then scale back down quickly.

Lower costs and operational agility translate into a competitive edge that can make all the difference for an early-stage startup. 

From there, how can you make spot instances work best for you? Here are a few quick tips:

**Use spot GPUs for flexible, restartable jobs.** Batch data processing and any experiments or workloads that can resume from checkpoints are excellent options when you're using interruptible instances on Vast.ai. 

**Make sure your workflows are tolerant of interruptions, and if you can, automate recovery.** With spot or interruptible instances, disruptions will almost certainly happen on occasion, and you'll want to be ready for that.

**Switch to on-demand instances for critical workloads.** Live inference and customer-facing applications are good candidates for on-demand. Anytime you want to minimize productivity loss from an interruption (for example, when a member of your team is actively debugging or deploying code), guaranteed uptime may be worth the higher cost.

## The Bottom Line

Interruptible instances on Vast give AI startups a cost-effective way to iterate quickly and scale smarter. By delivering the same high-performance hardware at a fraction of the cost, spot GPUs help extend your runway and free up budget to reinvest in growth. 

To learn more about why startups of all stripes choose Vast.ai, check out our previous post about how our platform is tailored to meet your needs from day one.

Ready to put the Vast.ai advantage to work for you? Explore our [spot GPU options](https://cloud.vast.ai/) and see how much faster – and smarter – your AI startup can scale!


For AI startups working with limited budgets and evolving product requirements, spot (interruptible) GPUs offer a flexible, cost-effective path to building and scaling AI infrastructure without breaking the bank.

Starting Smart: Why Spot GPUs Are Ideal for AI Startups


When you're training a custom AI model, access to the right GPU compute resources is crucial. That can be a tall order – but you don't have to drain your budget or get locked into long-term contracts to get the compute power you need. With **Vast.ai**, there's a better way.

Our cloud GPU rental platform gives you direct access to high-performance GPUs through a transparent, pay-as-you-go model that lets you train on your own terms: anytime, anywhere, and at the right price.

In this post, we'll take a closer look at how Vast.ai helps you train custom AI models faster and more affordably, without compromising performance or scalability.

## 1. Flexible, Budget-Friendly Pricing

Training AI models can get expensive. Vast.ai offers different [pricing options](https://docs.vast.ai/instances/rental-types) to help you keep cloud costs under control:

**Interruptible instances** let you rent GPUs at lower prices through interruptible or bid-based access. While these instances can be interrupted with little notice, they're a great fit for workloads that can handle brief disruptions in exchange for major cost savings – **up to 80% cheaper** than traditional cloud rates. For many users, that tradeoff is worth it for the reduction in training costs.

**On-demand instances** give you uninterrupted GPU access at a fixed rate. This setup is best for continuous training, real-time inference, or workloads where stability is crucial.

Of course, price is just one part of the equation: the next step is choosing the right GPU for the job. Vast.ai gives you that flexibility, as well.

## 2. GPU Options for Every AI Workload

There's no one-size-fits-all solution when you're training custom AI models. That's why our wide range of cloud GPU options is such an advantage. You can choose exactly what your workload needs, whether that's an enterprise-grade accelerator like the **NVIDIA [A100](https://vast.ai/article/H100-vs-A100-Comparing-two-Powerhouse-GPUs)** or **[H200](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights)**, an ultra-high-performance machine like the **[B200](https://vast.ai/article/nvidia-h200-vs-b200-comparing-datacenter-grade-accelerators)**, or a top-tier consumer card like the **[RTX 4090](https://vast.ai/article/nvidia-rtx-4090-vs-a100-two-powerhouses-two-purposes)**, **[5090](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know)**, or **[L40S](https://vast.ai/article/comparing-nvidia-l40-vs-l40s-and-more)**.

Each listing on Vast.ai displays **transparent instance specs**, making it easy to compare GPUs by type, RAM, storage, location, and more. Simply filter and select the best fit for you!

Most importantly, you'll never have to worry about overpaying for capacity you don't need or being held back by hardware limitations. When your workloads grow, scaling up is seamless. Vast.ai lets you spin up multiple GPU instances and customizable, on-demand [GPU/CPU clusters](https://vast.ai/clusters), so you can expand training capacity instantly – and only pay for what you use.

## 3. Supporting Real-World Use Cases

Our platform is ideal for anyone training, fine-tuning, or experimenting with custom AI models. Whether you're an AI researcher or developer, an enterprise team, or a [startup](https://vast.ai/article/7-reason-why-startups-uses-vast-ai), Vast.ai provides the compute power you need to train models for use cases like automating customer inquiries, identifying sales trends and other business insights, and analyzing massive research datasets.

You can build and refine domain-specific models, fine-tune foundation models like Llama and Mistral on proprietary data, and retrain as your datasets evolve. With support for LLMs and diffusion models, our platform is **compatible with all major machine learning (ML) frameworks** – and you can even run custom containers and tooling.

## 4. Quick Start with Pre-Built Templates

You don't have to start from a blank slate. Vast.ai offers pre-built [templates](https://docs.vast.ai/templates) for popular frameworks – **PyTorch, TensorFlow, vLLM, ComfyUI, and more** – that are designed to help you launch a training environment in minutes.

Each template includes the dependencies and configurations you need for model training or fine-tuning, so you can spend less time setting up and more time improving your results. Customize templates with your own datasets, scripts, or containers. You have complete flexibility to tailor them to your project needs!

Finally, Vast.ai also ensures that you can train your AI models in a trusted environment [with control over data exposure](https://vast.ai/article/running-private-ai-models-without-the-risk-of-data-exposure).

## 5. Enterprise-Grade Security

With Vast.ai, your workloads run on a secure, trusted infrastructure designed to [protect your data](https://vast.ai/compliance) – crucial when training custom AI models. Choose either **verified hosts** with strong reliability metrics or **vetted datacenter partners** who maintain third-party compliance certifications and are bound by comprehensive Data Processing Agreements (DPAs) with Vast.ai.

Our **Secure Cloud** offering gives you the option to select GPU infrastructure only from our vetted datacenter partners. Each of these providers is located in a certified facility that's current with ISO 27001 and/or Tier 3/4 standards at minimum.

We've also [completed](https://vast.ai/article/vast-soc2-typeII-certification) our own SOC 2 Type II certification, affirming our adherence to industry-leading compliance frameworks.

## Get Started with Vast.ai

Training custom AI models shouldn't be limited to those with deep pockets or powerful infrastructure. **Vast.ai** levels the playing field.

With high-performance GPUs, flexible pricing, and pre-configured templates to help you get started, you can move from idea to implementation faster and more affordably than ever with Vast.ai. You'll also enjoy peace of mind knowing that your data and models are protected by enterprise-grade security every step of the way.

Ready to train your own model? Spin up your first instance today at Vast.ai!


When you're training a custom AI model, access to the right GPU compute resources is crucial. Vast.ai gives you direct access to high-performance GPUs through a transparent, pay-as-you-go model that lets you train on your own terms.

Why Choose Vast.ai to Train Custom AI Models


We're excited to announce a new deployment guide for MiniMax-M2, the breakthrough 230 billion parameter language model that's making waves in the open-source AI community. If you've been looking for a cost-effective way to run state-of-the-art LLMs, this guide is for you.

## What Makes MiniMax-M2 Special?

MiniMax-M2 achieves the #1 composite score among open-source models while maintaining incredible efficiency. Despite its 230B parameter size, it activates only 10B parameters per inference, delivering fast responses without the typical computational overhead of large models.

Key features include:
- **Interleaved Thinking**: Transparent reasoning via `<think>...</think>` tags
- **131K Token Context**: Handle large documents and complex conversations
- **OpenAI Compatible**: Drop-in replacement for existing applications
- **MIT Licensed**: No restrictions on commercial use

## What's in the New Guide

Our latest documentation example walks you through deploying MiniMax-M2 on Vast.ai from start to finish. The guide includes:

**Complete Deployment Instructions**
- Hardware requirements and instance selection
- Step-by-step provisioning with the Vast.ai CLI
- API key authentication for secure deployments
- Configuration parameters validated through real deployments
- Expected timelines and what to watch for

**API Integration Examples**
- Python code using the OpenAI SDK with authentication
- cURL examples for quick testing
- Streaming response implementations
- Best practices for multi-turn conversations

**Troubleshooting Section**
Based on real deployment testing, we've documented the four critical issues you might encounter and their solutions:
- Docker image version requirements
- Disk space allocation
- GPU memory optimization
- CUDA driver compatibility

**Performance Data**
All metrics in the guide come from actual deployments on Vast.ai infrastructure:
- 30-minute initial deployment time
- ~7 seconds for 100-token responses
- 131K token context window
- Production-ready performance on 4x H100 GPUs

## Why Deploy on Vast.ai?

Vast.ai's GPU marketplace gives you access to enterprise hardware at competitive rates, making it economical to run powerful models like MiniMax-M2 for production workloads. Our guide uses a 4x H100 (80GB) configuration with the vLLM nightly build. You get:

- Enterprise GPU access (H100 recommended for best compatibility)
- Predictable costs with no usage-based pricing
- Full control over your infrastructure
- OpenAI-compatible API for easy integration

The guide shows you how to leverage these advantages for cost-effective LLM inference that rivals cloud API quality.

## Scaling for Production

While the 4x H100 configuration demonstrated in our guide provides an excellent starting point for deploying and testing MiniMax-M2 on Vast.ai, production deployments typically require larger GPU configurations to support longer context lengths and higher concurrent request volumes. For production use cases, consider configurations such as 8x H100, 4x H200, or 8x H200, which provide substantially more GPU memory for handling concurrent requests and extended context windows.

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Developers** building AI-powered applications who want cost control and flexibility
- **Research teams** running experiments that need high-volume inference
- **Startups** prototyping AI features without committing to expensive cloud APIs
- **Anyone** curious about deploying open-source LLMs at scale

## Get Started

The complete guide is now available in our documentation:

**[Read: Running MiniMax-M2 on Vast.ai →](https://docs.vast.ai/examples/text-generation/minimax-m2#minimax-m2)**

Whether you're exploring open-source AI options or ready to deploy your first large language model, this guide provides everything you need to get MiniMax-M2 running on Vast.ai infrastructure.

Ready to try it? [Sign up for Vast.ai](https://cloud.vast.ai) and follow the guide to deploy your first instance.

---

*All performance metrics and cost estimates in this article are based on actual deployment testing on Vast.ai infrastructure. Results may vary based on instance availability and configuration.*


Learn how to deploy MiniMax-M2, the 230B parameter language model with state-of-the-art performance, on Vast.ai infrastructure.

New Guide: Deploy MiniMax-M2 on Vast.ai

	
Choosing an RTX 6000 GPU isn't quite as straightforward as it might seem. The name "RTX 6000" now spans multiple cards across two architectures – the NVIDIA RTX 6000 Ada, as well as three newer Blackwell-based variants built for very different workloads. The four GPUs share some impressive fundamentals, but their performance profiles and a few important design differences set them apart.

Today we're looking at these NVIDIA RTX 6000 GPU options, what they're built for, and how to decide which one best fits your needs. So let's get into it!

### The RTX 6000 Family at a Glance

For starters, here's a brief overview of the four GPU models in the RTX 6000 lineup:

-   [RTX 6000 Ada Generation](https://cloud.vast.ai/?gpuModelNames=rtx6000AdaGeneration&instanceDiskSizeMin=700&instanceDurationMin=604800&instanceType=onDemand&isAvxSupported=false&isHostSecure=false&isMachineIpStatic=false&isOfferAvailable=true&isOfferCompatible=true&isOfferVerified=true&isQueryInverted=false&machineReliabilityMin=0.9&sorts=score-desc) – An enthusiast-class professional graphics card with solid all-around performance.

-   [RTX PRO 6000 Blackwell Workstation Edition](https://cloud.vast.ai/?gpuModelNames=rtxPro6000BlackwellWorkstation&instanceDiskSizeMin=700&instanceDurationMin=604800&instanceType=onDemand&isAvxSupported=false&isHostSecure=false&isMachineIpStatic=false&isOfferAvailable=true&isOfferCompatible=true&isOfferVerified=true&isQueryInverted=false&machineReliabilityMin=0.9&sorts=score-desc) – A full-power 600W desktop card, best used for single-GPU high-throughput workflows.

-   **RTX PRO 6000 Blackwell Max-Q Workstation Edition** – Basically the same hardware as the standard Workstation Edition above, but with a few design differences that make it better suited for multi-GPU setups.

-   [RTX PRO 6000 Blackwell Server Edition](https://cloud.vast.ai/?gpuModelNames=rtxPro6000BlackwellServer&instanceDiskSizeMin=700&instanceDurationMin=604800&instanceType=onDemand&isAvxSupported=false&isHostSecure=false&isMachineIpStatic=false&isOfferAvailable=true&isOfferCompatible=true&isOfferVerified=true&isQueryInverted=false&machineReliabilityMin=0.9&sorts=score-desc) – A passive-cooled variant designed for rackmount systems and high-density environments.

Next, we'll break down each GPU in more detail, then wrap up with some benchmarks and a side-by-side comparison table to help guide your decision.

### RTX 6000 Ada Generation: Solid and Dependable

The RTX 6000 Ada is the previous-generation flagship graphics card, and it still holds its own as a solid, versatile workstation GPU. Built on the Ada Lovelace architecture, it delivers strong performance for enhanced content creation and light-to-moderate AI training.

Overall, it's a well-rounded option. Featuring 48GB of GDDR6 ECC memory, a respectable 18,176 CUDA cores, up to 91.1 TFLOPS of single-precision performance, and 300W TDP, the RTX 6000 Ada can handle complex tasks and multi-application workflows without the more extreme price or power demands of a Blackwell-class GPU. It remains dependable for many professional use cases, especially if your projects don't require cutting-edge AI acceleration.

The RTX 6000 Ada Generation may be the right choice for you if:

Your workloads span creative applications, 3D tools, and mixed GPU tasks.

You don't need 96GB of VRAM but still want professional-grade stability and performance.

You're looking for a more cost-effective alternative to Blackwell for workstation use.

### RTX PRO 6000 Blackwell Workstation: Top-Tier Desktop Performance

The RTX PRO 6000 Blackwell Workstation Edition is the top-performing desktop card in this group. Built on NVIDIA's Blackwell architecture, it boasts 96GB of GDDR7 memory, a wide 512-bit memory bus, and extremely high memory bandwidth at nearly 1.8 TB/s. Plus, it has a staggering 125 TFLOPS of single-precision performance. This combination makes it well suited for the most demanding AI workloads as well as complex visualization, simulation, and neural rendering tasks.

The Workstation Edition can sustain peak performance under a 600W power load, without thermal throttling – but at the same time, it's not recommended for dense multi-GPU configurations due to its higher heat output and flow-through dual-fan design. Instead, it works best as a single GPU in a workstation setup.

The RTX PRO 6000 Blackwell Workstation Edition makes sense for you if:

You want the highest sustained throughput the RTX PRO 6000 series can deliver in a workstation, for the most demanding workloads.

You don't need to run multiple GPUs in the same system.

Your workstation has the power and cooling to support a 600W GPU.

### RTX PRO 6000 Blackwell Max-Q Workstation: Multi-GPU Friendly

RTX PRO 6000 Blackwell Max-Q Workstation Edition is ideal for anyone who wants Blackwell-level performance in a cooler, more efficient card that works well in multi-GPU configurations. For simplicity's sake, let's call it the Max-Q variant as opposed to the standard Workstation Edition.

The Max-Q features the same basic hardware as the full-power standard Workstation Edition – including 96GB of GDDR7 memory, a 512-bit bus, and 1.8 TB/s memory bandwidth – but it runs at a reduced 300W TDP and uses a blower-style cooler. Its enclosed single-fan design pulls air in and then exhausts it out the rear of the chassis. With lower heat output and superior management of airflow, the Max-Q is a much better fit for multi-GPU configurations than the standard Workstation Edition.

You may want to go with the Max-Q Workstation model if:

You're looking for Blackwell-class performance and memory capacity in a cooler, more efficient card.

You can't support a 600W GPU but still need 96GB of VRAM and high bandwidth.

You plan to run 2-4 GPUs and your workloads benefit from multi-GPU scaling.

### RTX PRO 6000 Blackwell Server Edition: Scalable and Server-Ready

The RTX PRO 6000 Blackwell Server Edition is built specifically for datacenter and rackmount environments. It lacks an active cooling fan, instead relying entirely on external airflow with a passive heatsink. It can also be configured up to the full 600W power envelope – as long as your airflow solution can handle the cooling required, of course!

Like the other Blackwell variants, the Server Edition includes 96GB of GDDR7 memory and a 512-bit bus. Its memory bandwidth is slightly lower at 1.6 TB/s, compared to 1.8 TB/s on the workstation cards, but the Server Edition is still more than capable of handling large models, massive datasets, and multi-GPU workloads. What really sets it apart is its ability to deliver stable, scalable performance in high-density environments.

Choose the RTX PRO 6000 Blackwell Server Edition if:

You need stable performance for continuous workloads across large-scale multi-GPU setups.

You want Blackwell-level memory and bandwidth in a form factor designed for dense compute.

You're deploying into rackmount servers where the chassis provides adequate high-pressure airflow.

### RTX 6000 Series: Performance Benchmarks

Now that we've covered design differences and use cases, let's check out how these GPUs actually perform. The benchmarks below show how each RTX 6000 variant handles real workloads, so you can get a clearer picture of what to expect in practice.

![RTX 6000 benchmark comparison](/uploads/vast_blog_6000_comparison_benchmarks.png)

And finally, for reference, here's a side-by-side specs comparison table showing how the four GPUs stack up against each other:

<table class="w-full border-collapse border border-gray-300">
  <thead>
    <tr class="bg-gray-100">
      <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
        Spec
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX 6000 Ada
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX PRO 6000 Blackwell Workstation
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX PRO 6000 Max-Q
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX PRO Blackwell Server Edition
      </th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Architecture
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Ada Lovelace</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Blackwell</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Blackwell</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Blackwell</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        VRAM
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">48GB GDDR6</td>
      <td class="border border-gray-300 px-4 py-2 text-center">96GB GDDR7</td>
      <td class="border border-gray-300 px-4 py-2 text-center">96GB GDDR7</td>
      <td class="border border-gray-300 px-4 py-2 text-center">96 GB GDDR7</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        ECC
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Memory Bandwidth
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">960 GB/s</td>
      <td class="border border-gray-300 px-4 py-2 text-center">1,792 GB/s</td>
      <td class="border border-gray-300 px-4 py-2 text-center">1,792 GB/s</td>
      <td class="border border-gray-300 px-4 py-2 text-center">1,597 GB/s</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Memory Interface
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">384-bit</td>
      <td class="border border-gray-300 px-4 py-2 text-center">512-bit</td>
      <td class="border border-gray-300 px-4 py-2 text-center">512-bit</td>
      <td class="border border-gray-300 px-4 py-2 text-center">512-bit</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        CUDA Cores
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">18,176</td>
      <td class="border border-gray-300 px-4 py-2 text-center">24,064</td>
      <td class="border border-gray-300 px-4 py-2 text-center">24,064</td>
      <td class="border border-gray-300 px-4 py-2 text-center">24,064</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Tensor Cores
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">568 (4th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">752 (5th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">752 (5th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">752 (5th gen)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Ray Tracing Cores
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">142 (3rd gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">188 (4th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">188 (4th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">188 (4th gen)</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Single-Precision Performance
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">91.1 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">125 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">110 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">120 TFLOPS (FP32)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        RT Core Performance
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">210.6 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">380 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">333 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">355 TFLOPS</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Total Board Power
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">300W</td>
      <td class="border border-gray-300 px-4 py-2 text-center">600W</td>
      <td class="border border-gray-300 px-4 py-2 text-center">300W (power-optimized)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">400–600W (configurable)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Form Factor
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">4.4" H x 10.5" L, dual slot</td>
      <td class="border border-gray-300 px-4 py-2 text-center">5.4" H × 12" L, dual-slot, extended height</td>
      <td class="border border-gray-300 px-4 py-2 text-center">4.4" H × 10.5" L, dual-slot</td>
      <td class="border border-gray-300 px-4 py-2 text-center">4.4" H × 10.5" L, dual-slot</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Thermal (Fan Configuration)
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Single-fan active</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Dual-fan active</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Single-fan active</td>
      <td class="border border-gray-300 px-4 py-2 text-center">No-fan passive (server chassis)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Date Released
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Dec. 2022</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Mar. 2025</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Mar. 2025</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Mar. 2025</td>
    </tr>
  </tbody>
</table>

### Final Thoughts

Each RTX 6000 variant brings something different to the table. As we explored above, the right choice ultimately comes down to how you plan to use it.

To sum things up, the Ada model is solid and dependable for a broad range of workloads, while the full-power Blackwell Workstation card offers the strongest single-GPU performance. The Max-Q variant is the more practical choice for lower-power or multi-GPU setups, and the Server Edition is designed for dense deployments and continuous workloads.

At the same time, these GPUs aren't exactly budget-friendly. That's where Vast.ai comes in. Our cloud GPU rental platform allows you to access this level of performance on demand, eliminating the upfront cost of purchasing the hardware yourself.

With Vast, you can easily match your compute to your workload and only pay for what you use. Best of all, you'll save 5-6X on average compared to traditional cloud providers.

Ready to see what Vast.ai can do for you? Spin up an RTX 6000 GPU instance today, or explore other powerful GPU options like the A100, H200, and B200 – available right now!


A practical guide to every RTX 6000 option, from Ada to the newest Blackwell workstation and server GPUs.

Which NVIDIA RTX 6000 Is Right for You?


When you think of a large language model (LLM), you probably picture the big household names like ChatGPT, Claude, Gemini, and Llama. They're dominant players for a reason – delivering cutting-edge performance that sets the standard for general-purpose AI – but they're also overwhelmingly centered around the English language and Western culture.

If you look further, there are plenty of other models gaining traction globally. Fueled by government investment as well as tech giants, startups, and research labs, these regional LLMs are trained on local languages and optimized for diverse needs.

With that in mind, let's take a look at some examples from around the world.

### Middle East: Arabic-First AI

Across the Middle East, government partnerships and large-scale investments are driving the rapid expansion of the region's AI sector. Much of this effort is focused on Arabic-first and bilingual LLMs designed for regional sovereignty and cultural awareness. For instance, the following are some of the leading projects in the region:

*   Jais offers open bilingual models up to 30B parameters, trained on large Arabic-English datasets, with the ability to switch between Arabic dialects. Instruction-tuned "Chat" variants, optimized for conversational applications, have also been released.

*   ALLaM powers Saudi Arabia's popular HUMAIN Chat conversational assistant and emphasizes Arabic fluency and cultural nuance. Its series of models is designed to support the ecosystem of Arabic Language Technologies (ALT).

*   Falcon, Abu Dhabi's flagship open LLM series, ranges from 7B to 180B parameters and combines commercially permissive licensing with strong performance that prioritizes resource efficiency – making it popular for academic, commercial, and government uses.

### Asia-Pacific: Scale, Localization, and Linguistic Diversity

The AI ecosystem in the Asia-Pacific region spans the largest and most linguistically diverse populations in the world. Both government-backed initiatives and major tech firms are creating models that serve dozens of languages and dialects while delivering performance that in many cases stands on par with the most advanced models worldwide.

*   [DeepSeek](https://vast.ai/article/serving-deepseek-models-on-vast-ai-with-vllm-and-langchain), one of the most capable open-source models to emerge from China, uses a Mixture-of-Experts (MoE) architecture to balance accuracy and computational efficiency. It offers more fluent support for Asian languages, with cultural nuance that may be missed by Western-centric models.

*   [Qwen](https://vast.ai/article/alibabas-qwen-an-open-source-ai-model-that-surpasses-deepseek), developed by Alibaba, covers 119 languages and dialects, with specialized multimodal variants and built-in support for Model Context Protocol. It features hybrid "thinking" and "non-thinking" modes for flexible reasoning – and you can even set a "thinking budget" to control how much computational effort it devotes to reasoning versus how quickly it replies.

*   tzusumi is a lightweight Japanese LLM (0.6B and 7B parameters) designed for on-prem use, offering strong Japanese-language processing and multimodal support with minimal hardware requirements.

*   Solar Pro 2 offers frontier performance in a comparatively compact 31B model, with multilingual capabilities and standout fluency in Korean. It's designed for real-world agent-like workflows across domains.

*   SeaLLM is a multilingual model built for Southeast Asia that outperforms most open-source LLMs across domains like science, physics, and economics in the languages of the region. Notably, it supports non-Latin scripts and low-resource languages like Lao and Khmer.

### Europe: Responsible AI and Data Sovereignty

In Europe, AI development tends to emphasize data sovereignty and compliance. The region has produced a number of LLMs that serve multilingual users and align with the EU AI Act and GDPR while competing with U.S. and Asian counterparts in raw performance.

*   Mistral, developed in France, offers a suite of open and proprietary models for reasoning, multimodal, coding, and other applications, including its flagship Mistral Large 2 and the open-source Devstral family – and even a multilingual chatbot called Le Chat that boasts deep research and image editing capabilities.

*   Luminous by Aleph Alpha is a multilingual model family (13B–70B parameters) built for enterprise and sovereign-AI use. It emphasizes compliance and transparency and provides support for English, German, French, Italian, and Spanish. The company also offers Pharia, a newer generation of models designed for concise, length-controlled responses.

*   BLOOM is a 176 billion-parameter open-access model produced through a year-long volunteer research effort to provide a transparent alternative to proprietary AI models. It can generate text in 46 natural languages and 13 programming languages.

### Other Regions: Emerging Innovation

Outside the major AI centers, new regional efforts are gaining momentum. For instance, Latam-GPT is an open initiative by Chile's nonprofit CENIA to develop a collaborative AI model completely within and for Latin America – trained on regional data and languages. The project's goal is to advance technological independence and cultural representation across the region.

In Africa, InkubaLM (named after the IsiZulu term for a dung beetle) addresses the need for lightweight African-language models. Trained on English and French as well as five African languages – Hausa, Swahili, IsiXhosa, IsiZulu, and Yoruba – the compact 0.4B-parameter model delivers efficient natural-language capabilities without heavy compute demands.

These projects reflect how emerging regional players are similarly building language models that prioritize inclusion and linguistic diversity, broadening the reach of global AI innovation.

### How to Choose a Regional LLM

Choosing a regional model depends on your goals and infrastructure. Here are a few considerations to keep in mind:

*   Look for open-weight models if you're prioritizing sovereignty or on-prem use.

*   Pick the right model size for your infrastructure! Many regional LLMs offer compact variants.

*   Consider dialect coverage and code-switching capabilities (e.g., training on colloquialisms) – along with nuanced cultural awareness – for social and chat applications.

*   If your use case requires meeting strict data protection and compliance standards, EU-built models developed with these priorities in mind may be ideal.

Once you've found the right fit, the next step is putting these models to work!

### The Bottom Line

Regional LLMs reflect how AI is evolving across a global spectrum of languages, cultural norms, regulatory priorities, and specialized technical needs.

Ultimately, they represent a more inclusive approach to model development. In turn, this helps bridge the AI gap for underrepresented groups worldwide. For developers and organizations, these LLMs provide context-aware and culturally fluent performance that's both versatile and cost-efficient.

With platforms like [Vast.ai](https://vast.ai), you don't need massive in-house infrastructure to take full advantage of these models. Our affordable cloud GPU rental makes it easy to train, fine-tune, and deploy regional LLMs at scale – helping you find just the right fit for your use case and bring AI closer to the communities and markets you serve.

> **Start experimenting with regional LLMs on [Vast.ai](https://vast.ai) today!**


From Jais to Qwen, Falcon to BLOOM - a look at how regional large language models are reshaping AI across languages, cultures, and continents.

Global Growth of Regional Large Language Models (LLMs)


What if your Integrated Development Environment (IDE) could talk back to you, suggest solutions, and write half your code for you? That's essentially what vibe coding does.

Coined earlier this year, "vibe coding" is an interactive approach where developers partner with AI to generate code. Instead of typing every line manually, they use natural language to collaborate with an AI model – guiding and prompting the model to generate and refine code through an iterative "vibing" process.

So why is vibe coding taking off? What are some of its benefits and challenges? Let's take a closer look at this very new approach to coding.

### Why Vibe Coding Is Growing in Popularity

Vibe coding has lowered the barrier to entry for software development, allowing non-technical individuals and hobbyists to build applications by simply describing what they want. Suddenly just about anyone can create their own custom tools and prototypes.

It's no wonder vibe coding is catching on. The following are some key reasons why:

#### 1\. Context-aware Assistance

In vibe coding, the AI model understands the broader context of the project. It can interpret the developer's intent and suggest code that aligns with the project's purpose, while also tracking variables, dependencies, comments, and recent edits in order to maintain a seamless workflow.

#### 2\. Productivity Gains

Research has [found](https://itrevolution.com/articles/new-research-reveals-ai-coding-assistants-boost-developer-productivity-by-26-what-it-leaders-need-to-know/) that AI coding assistants can boost the productivity of software developers by 26% in real-world business settings. AI models automate mundane and repetitive tasks like writing boilerplate code and generating tests, increasing the speed of iteration and time-to-prototype.

#### 3\. Cognitive Offloading

Instead of focusing on syntax and structure, developers can spend more time thinking about architecture and design. Delegating low-level tasks to the AI model enables them to focus on higher-level decisions like what the software should do, rather than how to make it happen.

### Benefits of Vibe Coding

Beyond speed and efficiency, here are a few benefits of vibe coding:

-   **On-demand collaboration**: The AI model is always available to collaborate as a coding partner. For those new to coding, it can effectively act as a built-in teacher, and for seasoned developers, it helps brainstorm and validate ideas.

-   **Creative workflow**: By handling the mundane, repetitive parts of coding, AI lets developers concentrate on the big picture, which encourages deeper thinking and creative problem-solving.

-   **Democratizing software development**: With vibe coding, non-technical users can build and prototype their own tools. Programming expertise is no longer required to turn ideas into working applications, opening the door to greater creativity and inclusion in tech.

Vibe coding isn't without drawbacks, however.

### Potential Pitfalls of Vibe Coding

Despite its advantages, vibe coding does introduce real challenges that must be thoughtfully dealt with, such as:

-   **Hallucination or incorrect code**: AI models can generate convincing but incorrect code, which could include security flaws. Developers should review and test everything that comes from the AI, essentially treating it as an assistant that requires careful supervision.

-   **Version drift**: Over time, inconsistencies may emerge between human-written and AI-generated sections of code, and the code may increasingly deviate from the intended architectural design. Careful integration pipelines and regular "re-anchoring" to the original goals are crucial to enforce version control.

-   **Governance and compliance issues**: With AI contributing portions of code, organizations must ensure that all of the code they use is auditable and properly licensed while also being safe to deploy. Human oversight at every step is vital.

Overall, vibe coding is a pretty exciting new phase in software development. It just needs to be used carefully, at the right time and in the right place.

### Final Thoughts

Most vibe coding happens on public platforms powered by cloud-hosted AI models. This is perfectly fine for experimentation but may not be ideal for projects involving sensitive code or proprietary data. For individuals and teams that need privacy or operate under strict compliance requirements, a closed and controlled environment may be a better fit.

With **Vast.ai**, it's easy to spin up secure, isolated GPU instances on demand – so you can host open-source models like Code Llama or DeepSeek Coder and get started vibe coding while keeping your data fully contained. Save up to **5–6X** compared to traditional cloud providers and enjoy both privacy and performance without the heavy price tag.

Our platform offers a practical way to experiment with vibe coding on your own terms – securely, affordably, and under your control. Explore Vast.ai today\!


Discover vibe coding — an AI-driven way to code faster and smarter. Collaborate with AI to generate, refine, and prototype code securely on Vast.ai.

Less Coding, More Creating: The Rise of Vibe Coding


One of the most important considerations when you're running private AI models on a cloud platform is data security. You want to protect your training data and the outputs your models generate – and even the models themselves.

At Vast.ai, we recognize that there's always some level of risk when sensitive data moves through cloud infrastructure. That's why we've built safeguards into every layer of our platform.

From containerized environments to our Secure Cloud offering and ongoing compliance efforts, here's how we help ensure you can run private AI models with confidence on Vast.

### Docker Container Isolation

When you rent GPUs on a cloud platform like Vast.ai, your workloads run inside Linux Docker instances. Each of these instances is a container created from a Docker image, where the container packages your processes and dependencies into a self-contained unit. This container runs in its own isolated environment, separate from the host system and from other clients' containers.

Why is this important? On a multi-tenant platform, multiple users may be running workloads on the same physical server. But with the isolation provided by a containerized environment, **each client only has access to their own data.**

Additionally, when you delete your instance, all data stored on it is also deleted. If you stop your instance, your data will still be stored on the disk and able to be copied in and out, but it's crucial to destroy your instances when you no longer need them. This helps protect your data while also avoiding unnecessary disk storage charges.

If you require the extra assurance of enterprise-grade data protection, you're in luck. Our Secure Cloud option may be just what you're looking for.

### Vast.ai's Secure Cloud Offering

Our **Secure Cloud offering** is designed for clients with the strictest data protection requirements – particularly well suited for when you're running your own private AI models or working with proprietary or sensitive datasets. It's our highest security tier, giving you the option to select GPU infrastructure only from our vetted datacenter partners.

The datacenter providers in our Secure Cloud all hold a **minimum of ISO 27001 certification**. Many are also HIPAA, NIST, PCI, and/or SOC 1-3 certified, as well as GDPR compliant. Each datacenter provider signs comprehensive Data Processing Agreements (DPAs) with Vast.ai and undergoes both internal and third-party audits on a regular basis.

Furthermore, we compile documentation on the identity of each provider and ensure they follow industry-recognized best practices for incident response, continuous monitoring systems, and security and compliance training for employees.

Our Secure Cloud offering provides the strongest safeguards available on Vast.ai. However, we also maintain our own ongoing efforts to reinforce data protection across our platform.

### Ongoing Compliance Efforts

At Vast.ai, we understand that compliance is an ongoing responsibility. Our integrated approach to security and trust weaves robust compliance measures throughout our platform to meet the unique needs of every client.

Some of our ongoing efforts include:

-   [**SOC 2 Type I and II and SOC 3 Certification**](https://vast.ai/article/vast-soc2-typeII-certification)s – Independent audits of our internal controls, validating that your sensitive data is secure, available, and privately stored across all of our systems (along with a standard 12-month Type II verification cycle for continuous coverage year round).

-   [**Vulnerability Bounty Program**](https://vast.ai/article/announcing-the-vast-ai-vulnerability-bounty-program) – A structured way to report bugs and responsibly disclose any security issues on our platform, as well as suggest new features.

Our [**Trust Center**](https://trust.vast.ai/) provides a detailed overview of our controls, compliance stance, FAQs and Subprocessors, and details about our other certifications and security practices.

### Securing Your Peace of Mind

Running private AI models on a cloud GPU platform requires careful attention to security. You must be able to protect not only the data you train on, but also the models you've built and the outputs they produce. Addressing these concerns is a top priority for us.

At Vast.ai, our goal is to give you the flexibility to scale workloads while maintaining the privacy and integrity of your data. As a result, you can be confident that your sensitive assets stay secure. As we continue to advance the security infrastructure we've built over 6+ years of trusted service, we remain focused on making powerful GPU compute accessible and secure for everyone.

Explore how **Vast.ai** can support your private AI workloads at scale, today\!


Learn how Vast.ai protects your sensitive data while running private AI models in the cloud. Discover our Secure Cloud offering, Docker container isolation, and ongoing compliance measures that ensure your AI workloads stay private and secure.

Running Private AI Models Without the Risk of Data Exposure


We're back with a new round of updates here at Vast.ai\! These past few months, we've expanded our offerings so you can get the most out of your time and budget – with new hardware options, fresh templates, and continued enhancements to elevate your experience.

To that end, we're pleased to offer **NVIDIA B200 GPUs** for rent\! These incredibly powerful machines deliver exceptional performance for develop-to-deploy pipelines and all of your most demanding enterprise AI tasks.

But that's not all. Here are the details on what's new\!

### NVIDIA Cloud GPU Updates

As mentioned, the NVIDIA B200 joins our growing GPU lineup, bringing you more power and options than ever.

Featuring **192 GB** of HBM3e memory with extremely high bandwidth, the B200 is ideal for high-throughput AI workloads like fine-tuning, Retrieval-Augmented Generation (RAG), and LLM inference. For additional information and pricing details, learn more [here](https://vast.ai/pricing/gpu/B200).

### Fresh New Templates

We've also rolled out several templates that expand what you can build and run on Vast. Jump-start your creative and technical projects with templates such as:

-   [**InvokeAI**](https://cloud.vast.ai/?ref_id=62897&template_id=c309df384010cf9d3dc9abb9e0a5a6fd) – A creative engine providing a professional web interface for AI image generation using Stable Diffusion models.

-   [**Langflow \+ Ollama**](https://docs.vast.ai/langflow-ollama) – A visual AI workflow platform with an intuitive drag-and-drop interface in Langflow and a hosted Ollama API server with models like Llama, Mistral, and more.

-   [**Ostris AI Toolkit**](https://cloud.vast.ai/?ref_id=62897&template_id=81e6e9dd266d9bdbcbb0fd48b9afe357) – A complete environment for training custom Stable Diffusion models, LoRAs, and other AI image generation models using your own datasets.

-   [**ACE Step**](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ACE%20Step%20Music%20Generation) – An AI-powered music generation system that lets you create original music through a simple web interface.

### Other Improvements

We've recently updated our Discord server to make conversations more focused, relevant, and helpful for you. Didn't know we had a Discord? [**Join us**](https://discord.gg/vast) there today for direct access to updates, technical guidance, and our supportive user community\!

We've also made an important update to how API keys are handled. To strengthen security, **API keys are now one-time viewable and not readable anywhere on Vast.** Be sure to store your key off of Vast when it's created, as it won't be visible again afterward. Vast.ai API endpoints will no longer return API keys as part of their response (see details in change log below).

That wraps up the updates for October 2025\!

### Our Commitment

From the new B200 GPUs to convenient templates that help you move from idea to execution, this month's updates continue our mission to make high-performance compute accessible to everyone. We're here to help you build faster and create more, with a platform that grows with your needs.

Need assistance? Reach out to our customer support team at [support@vast.ai](mailto:support@vast.ai) or join our [Discord](https://discord.gg/vast) server anytime for tips, troubleshooting, and community insights.

### Change Log

#### **InvokeAI Template now available\!**

InvokeAI is a leading creative engine for Stable Diffusion models that provides a professional web interface for AI image generation. It's perfect for artists, designers, content creators, or anyone wanting to generate stunning AI artwork. Find the InvokeAI template on Vast.ai [here](https://cloud.vast.ai/?ref_id=62897&template_id=c309df384010cf9d3dc9abb9e0a5a6fd).

#### **Langflow Template & Guide are now live\!**

This template gives you a complete visual AI workflow platform with Langflow and a hosted Ollama API server running in Docker containers. Langflow provides an intuitive drag-and-drop interface for building complex AI applications, while Ollama serves as your private LLM backend with models like Llama, Mistral, and more. Learn how to launch Langflow on Vast.ai [here](https://docs.vast.ai/langflow-ollama).

#### **Ostris AI Toolkit Template now available\!**

This template gives you a complete Stable Diffusion training environment with the Ostris AI Toolkit running in a Docker container. The toolkit provides an intuitive web interface for training custom Stable Diffusion models, LoRAs, and other AI image generation models with your own datasets. Find the Ostris template on Vast.ai [here](https://cloud.vast.ai/?ref_id=62897&template_id=81e6e9dd266d9bdbcbb0fd48b9afe357).

#### **ACE Step Template now available\!**

This template gives you a hosted ACE Step music generation server running in a Docker container. ACE Step is an AI-powered music generation system that lets you create original music through a web interface. It's perfect for musicians, content creators, developers, or anyone who wants to experiment with AI music generation. Find the ACE-Step template on Vast.ai [here](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ACE%20Step%20Music%20Generation).

# **Vast Discord Update: Unlock Your Channels\!**

We’ve streamlined our Discord to make it cleaner, faster, and more relevant to you. Here’s what’s changed.

## **Connect your Vast account to instantly unlock channels designed for your role:**

-   **Hosts** → Access to \#Hosting, \#Hosting-rigs, \#Support-hosting. (More host channels coming soon\!)
-   **Clients** → Unlock support rooms for templates, feature requests, and more.

# **API Key Security Update \+ API Breaking Changes**

**API keys are now one-time viewable and not readable anywhere on Vast.** You will need to store your API key off of Vast at the moment it is created, as it will only be visible this one time. If you need access to your existing API key, and did not store it off of Vast, you will need to delete it and create a new one. **Vast.ai API endpoints will no longer return API keys as part of their response.** However, we've added **key_id** to responses specifically for cases like this. The newly added key_id can be used to verify a user instead of api_key in certain scenarios. The key_id associated with a user's API key is returned from the show api keys endpoint / command.

## **key_id is now available for these API calls:**

-   create_user \- POST /api/v0/users
-   log in (regular and oauth and tfa) \- PUT /api/v0/users/current
-   update_user \- PUT /api/v0/users
-   get_current_user \- GET /api/v0/users/current
-   get_root_user \- GET /api/v0/users/root
-   connect_discord \- GET /api/v0/oauth/connect-discord/
-   disconnect_discord \- DELETE /api/v0/admin/disconnect-discord/
-   get_subaccounts \- GET /api/v0/subaccounts/

## **The following API key endpoints are now deprecated:**

-   GET /api/v0/auth/apikeys/&#123;id&#125;/
-   PUT /api/v0/commands/reset_apikey/

## **And the following will now have null for the key values:**

-   GET /api/v0/auth/apikeys/
-   GET /api/v0/team/apikeys/

## **New NVIDIA B200s added\!**

B200 GPUs are ideal for high-throughput AI workloads like fine-tuning, Retrieval-Augmented Generation, and LLM inference. These GPUs feature 192 GB of HBM3e memory with extremely high bandwidth.

-   [Learn more](https://vast.ai/pricing/gpu/B200)
-   [Rent now](https://cloud.vast.ai/?gpu_option=B200)


Discover what's new at Vast.ai! From the powerful NVIDIA B200 GPUs to fresh AI templates like InvokeAI, Langflow + Ollama, and Ostris, plus security and API updates — explore how we're making high-performance compute faster, safer, and more creative for everyone.

October 2025 Product Update


By now, just about everyone has either heard of large language models (LLMs) or has used them before. Not as many people are familiar with small language models (SLMs), however.

The names offer a clue – large and small – but size isn't the only thing that sets them apart.

So what are the differences between LLMs and SLMs? When do you actually need a large model, and when might a small one be the better choice?

In this post, we'll answer these questions and more. Here's what you need to know.

### Large Language Models vs. Small Language Models

An AI language model is a type of artificial intelligence that is trained on huge text datasets to **understand and generate human language.** These models use probabilistic machine learning to predict which words are most likely to appear next in a phrase sequence – and, in doing so, are able to generate text that resembles how people actually write and speak.

In practice, this ability is what underpins natural language processing (NLP) systems that let machines interact with us in useful ways.

When it comes to LLMs and SLMs specifically, though, let's look at how the two compare.

#### What Are LLMs?

With **massive parameter counts** in the billions and even trillions, most modern LLMs rely on the transformer architecture, which uses a **self-attention mechanism** to make sense of relationships between words across a sequence. This allows them to model incredibly complex patterns and long-range dependencies in language.

Since they're **trained on extensive datasets over a variety of domains**, LLMs are highly versatile. They have broad general knowledge and can generalize well across different tasks – even multitasking effectively across domains.

The downside is that training and running LLMs requires **significant computational power.** It's a resource-intensive process that often involves costly hardware and large-scale distributed computing infrastructure.

#### What Are SLMs?

At the basic level, an SLM is a smaller version of an LLM. It has **fewer parameters** – in the millions to low billions – and its architecture is focused on efficiency.

For instance, some SLMs use a **sliding window attention** mechanism (where the model focuses on a fixed-length "window" that slides across the text), and others rely on techniques like **grouped-query attention** (GQA), **sparse attention patterns**, and **low-rank adaptation** (LoRA) to improve efficiency and lower compute costs.

Unlike LLMs, SLMs are **trained on domain-specific data** or smaller datasets tailored to particular tasks. Although they may lack broad general knowledge, they do well in their specific domains and can even be fine-tuned for niche or regulated industries like finance and healthcare.

Because of their smaller size, SLMs need **less computational power** and can often be trained and deployed on more modest hardware.

### LLMs vs. SLMs: Strengts and Limitations

Both LLMs and SLMs have advantages and drawbacks. Here's a quick side-by-side comparison of their respective pros and cons:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Model Type
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Strengths
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Limitations
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">LLM</td>
            <td class="border border-gray-300 px-4 py-2">
                Broad general knowledge across different domains
                <br />
                Can handle complex, open-ended tasks with more contextual
                understanding
                <br />
                Greater multimodal potential
            </td>
            <td class="border border-gray-300 px-4 py-2">
                Requires extensive compute resources and costly hardware
                <br />
                Slower inference and higher latency
                <br />
                Not optimized for specific tasks, and higher risk of bias due to
                unfiltered training data
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">SLM</td>
            <td class="border border-gray-300 px-4 py-2">
                Easier to fine-tune for specific domains
                <br />
                Uses less compute resources
                <br />
                Faster inference and lower latency
                <br />
                Stronger data control and easier on-prem deployment
            </td>
            <td class="border border-gray-300 px-4 py-2">
                Narrower focus and less general knowledge; may require
                retraining for new tasks
                <br />
                Struggles with highly complex or long-context tasks
                <br />
                Limited multitasking capability
            </td>
        </tr>
    </tbody>
</table>

So how do you decide which type of model is right for your needs?

### Choosing the Right Model

As we've covered, neither model type is universally superior. The value of each one depends on what you're trying to accomplish and the resources you have available. That said, here are some guidelines to keep in mind.

#### An LLM may be the best choice if:

-   You need a model for **open-ended, multi-domain applications** such as general-purpose chatbots or creative content generation.  

-   Your use case involves **long-range context** across different subject areas.  

-   You have access to **powerful compute resources** and can manage the costs involved.

#### An SLM may be right for you if:

-   You prefer a model that is **lightweight and efficient**, suited for **resource-constrained environments.**  

-   Your application is **narrow or domain-specific**, such as FAQ bots or translation from one language to another.  

-   You want to **fine-tune with proprietary data** or maintain stricter control in regulated industries.

At the same time, you don't necessarily even have to choose one or the other; a hybrid approach could be the way to go. An SLM can take on routine tasks while an LLM handles more nuanced issues. For instance, you could use an SLM for standard customer queries and escalate more complex problems to an LLM.

### Final Thoughts

Both LLMs and SLMs offer unique advantages. Getting the most out of them often depends on having scalable, affordable compute – and that's where **Vast.ai** comes in.

Our cloud GPU platform gives you the flexibility to run a lightweight SLM on a single GPU or to train a massive LLM across [distributed clusters](https://vast.ai/products/clusters), all at a fraction of the typical cost. With Vast.ai, you can **save up to 5–6X** compared to traditional cloud providers – and train, fine-tune, and deploy AI models on your own terms.

Ready to get started? Spin up GPUs on demand with [Vast.ai](http://Vast.ai) today\!


Learn the key differences between Large Language Models (LLMs) and Small Language Models (SLMs), their strengths, limitations, and when to use each.

LLMs vs. SLMs: What's the Difference, and Why Does It Matter?


Are you looking for a faster way to turn your AI application ideas into working prototypes?

Langflow may be just the solution you need. Its visual editor simplifies the process, allowing you to build and test a functional AI application workflow quickly and easily. You can develop anything from chatbots and document analysis systems to content generators and agentic applications – and much more.

### What Is Langflow?

Put simply, [Langflow](https://docs.langflow.org/about-langflow) is an **open-source, node-based agent builder** that you can use right from your web browser. In minutes, you can create and serve flows: that is, functional representations of application workflows. All you have to do is connect and configure component nodes using the drag-and-drop visual editor, where each component represents one step in the workflow.

Then you can test your flows in real time using the Playground. This is a dynamic interface that lets you examine and experiment with different inputs, outputs, and memories – and get feedback about flow logic and response generation – before you move forward with more formal app development. Flows can even be embedded into your application code using the Langflow API.

Plus, Langflow is model-agnostic. It supports all major LLMs and vector databases you want to use, and it has built-in support for the Model Context Protocol (MCP), as well. It even has a **convenient Ollama integration** that makes it easy to use with open-weight models as well as custom fine-tunes.

With Vast, you can run Langflow with Ollama on powerful GPU instances at affordable rates, enabling cost-effective AI app development and prototyping. This guide will show you how to set up [Langflow \+ Ollama](https://docs.vast.ai/langflow-ollama) using our pre-built template in just minutes.

### Getting Started with Langflow \+ Ollama

Before you do anything else, set up your Vast account and add credit. If you need help with this step, review our [QuickStart guide](https://docs.vast.ai/quickstart) for a walkthrough of the setup process.

Now it's time to configure your template. Minimal changes are needed here, but you'll probably want to customize the template so that Ollama will download your preferred model automatically.

Follow our [step-by-step guide](https://docs.vast.ai/langflow-ollama) to get you started in our documentation.

### Conclusion

There is so much more to Langflow than it's possible to cover in this short introduction. The endlessly customizable framework lowers the barrier to building and testing AI applications, while Ollama keeps model choice open and flexible. Both beginners and seasoned developers can benefit from its ease of use and support for creating more complex projects.

Running Langflow \+ Ollama on **Vast.ai** gives you the cost-effective GPU infrastructure to go from prototype to production on your own terms.

Turn your ideas into working real-world solutions today\!


Run Langflow with Ollama on Vast.ai GPUs for fast AI prototyping. Step-by-step guide to build chatbots, agents, and custom AI apps with a visual editor.

Build and Test AI Applications with Langflow + Ollama on Vast.ai


[Video generation with AI](https://docs.vast.ai/video-generation) has come a long way. If you saw the infamous AI-generated video of Will Smith eating spaghetti a couple of years ago, you probably remember how unnatural and distorted the clip looked\! But that was just the beginning.

Things look very different today, with more advanced video generation tools producing much higher quality results. Today's models can create photorealistic visuals with smooth, consistent motion as well as stylized and cinematic effects if that's what's called for.

AI video generation has become a practical tool for anyone who needs an easy way to go from text to video, no matter the project. The right templates make that process even simpler.

With that in mind, let's take a look at a few of the best [AI video generation](https://docs.vast.ai/video-generation) templates available today.

### 1\. [ComfyUI \+ LTX Video](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%20%2B%20LTX%20Video)

[**LTX Video**](https://github.com/Lightricks/ComfyUI-LTXVideo) is a group of open-source AI video models built for high-quality diffusion-based video generation. It offers both speed and precision, generating high-resolution videos **up to 60 seconds long** (with some versions of the model limited to **10 seconds**), often in less time than it takes to watch the resulting video.

How is this possible? LTX Video uses a unique approach where it **compresses video data 192 times smaller than the original** – past the limits of what most models can achieve – while still maintaining crisp visuals and smooth, realistic motion. A bonus is that this drastically reduces the amount of memory and compute required. In fact, LTX Video is optimized specifically for consumer-grade GPUs.

While most diffusion models use a two-step pipeline where a second-stage model sharpens details, LTX Video does decoding and cleanup in just one step. With a Video-VAE (variational autoencoder) and a denoising transformer combined into a single integrated system, there's no need to patchify the video stream, and high-quality results are possible even under heavy compression.

This [template](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%20%2B%20LTX%20Video) pairs LTX Video with **ComfyUI**, a modular, node-based interface for Stable Diffusion workflows. Together, they serve as a seamless way to run efficient, high-compression video pipelines with a user-friendly interface.

### 2\. [Mochi](https://vast.ai/article/generating-videos-with-mochi-on-vast.ai)

Designed to excel in quality of motion as well as prompt adherence, [**Mochi**](https://github.com/genmoai/mochi) is a state-of-the-art open-source video generation model that performs competitively with leading closed models.

It demonstrates strong motion quality, generating smooth videos at **30 frames per second for durations of up to 5.4 seconds.** Fluid dynamics, fur and hair simulation, and human movement are all rendered with impressive realism.

Mochi's alignment with text prompts is also exceptional. Generated videos accurately depict the given input, allowing users to control characters, settings, and actions with a high degree of detail. Prompt adherence is benchmarked with a vision-language model and generated videos are evaluated using Gemini to ensure accurate, consistent results.

However, keep in mind that the initial release of Mochi only generates videos at **480p**, and the model is optimized for photorealistic styles, so animated content is not recommended.

### 3\. [Open-Sora](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Open-Sora)

[**Open-Sora**](https://github.com/hpcaitech/Open-Sora) is an open-source video generation model with a streamlined, user-friendly platform that lowers the barrier to efficient, high-quality video content creation. It can produce videos **up to 15 seconds long, at resolutions of up to 720p, with any aspect ratio.**

The model also supports a wide range of visual generation tasks: text-to-image, text-to-video, and image-to-video generation. This flexibility means users can run diverse workflows within a single framework.

Notably, Open-Sora introduces two innovations. The first is the Spatial-Temporal Diffusion Transformer (STDiT), a framework that decouples spatial and temporal attention. The second is a highly compressive 3D autoencoder that not only makes representations more compact but also accelerates training with its own ad hoc training strategy.

These features allow Open-Sora to generate high-quality results more efficiently – and with a ready-to-use template, it's easier than ever to start experimenting.

### Final Thoughts

AI video generation has matured rapidly in just a few short years. Instead of glitchy experiments, we now have reliable tools that deliver polished video content.

At **Vast.ai**, we believe in democratizing [access to these advanced AI tools](https://docs.vast.ai/video-generation), making them easy to use and available to everyone without the overhead of complex setup.

With our pre-built templates for [LTX Video](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%20%2B%20LTX%20Video), [Mochi](https://vast.ai/article/generating-videos-with-mochi-on-vast.ai), and [Open-Sora](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Open-Sora), you can launch cutting-edge video generation models in minutes – backed by affordable, on-demand cloud GPUs that save you from having to purchase and maintain your own expensive hardware. Use only the compute you need, when you need it.

Ready to experiment? Explore the [video generation templates](https://docs.vast.ai/video-generation) on Vast.ai and bring your ideas to life today\!


Generate AI videos with LTX Video, Mochi, and Open-Sora on Vast.ai. Step-by-step templates for high-quality, photorealistic text-to-video workflows with affordable GPU hosting.

Vast.ai: Best Video Generation Templates 2025


NVIDIA is [reportedly](https://www.reuters.com/world/china/nvidia-working-new-ai-chip-china-that-outperforms-h20-sources-say-2025-08-19/) working on a new China-specific AI chip that will be more powerful than the aging HGX H20 model the company is currently authorized to sell in China.

Recent reports [suggest](https://www.tomshardware.com/tech-industry/artificial-intelligence/nvidia-responds-to-reports-that-its-h20-gpu-for-china-is-ending-production-next-gen-b30a-green-light-up-to-the-united-states-government-according-to-ceo-jensen-huang) that the H20 – originally developed as a scaled-down alternative to the A100 and H100 that were banned from export to China – is nearing the end of its production run. Its successor: the tentatively named B30A chip.

Based on the latest Blackwell architecture, the B30A will feature a single-die design delivering roughly half the raw computing power of NVIDIA's flagship B300 accelerator, according to sources. The new chip will include high-bandwidth memory and NVLink support for fast data transfer between processors – both of which are also present in the Hopper-based H20.

NVIDIA aims to send test batches of the B30A chip to Chinese customers as early as sometime this month, sources said.

The move comes following remarks from President Donald Trump suggesting that more advanced NVIDIA chips could be cleared for sale in China. However, regulatory approval remains uncertain. There are ongoing concerns about granting China broader access to U.S. AI technology – which could weaken the United States' competitive edge in the sector.

At the same time, analysts warn that restricting China too heavily risks driving developers toward rivals like Huawei. NVIDIA argues that it's important to retain market share in China – which generated 13% of the company's revenue in the past year – so that its software ecosystem stays relevant in one of the world's largest AI markets.

NVIDIA is also preparing a second Blackwell-based China-specific chip called the RTX6000D, designed primarily for AI inference tasks, which is expected to start shipping this month. With weaker specs than the H20 and bandwidth just under U.S. export limits, it's set to be a compliant, lower-cost alternative within NVIDIA's lineup.

For those outside China, however, these region-specific models highlight the stark difference between scaled-down chips and the unrestricted performance available elsewhere.

### On-Demand Access to Powerful GPUs, Anytime

While NVIDIA adapts its offerings to meet specific market regulations, users elsewhere don't have to work around restricted versions.

On [**Vast.ai**](http://Vast.ai), you can access a wide range of powerful GPUs – from the [RTX 5090](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know) to the [A100](https://vast.ai/article/nvidia-rtx-4090-vs-a100-two-powerhouses-two-purposes), [H200](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights), [B200](https://vast.ai/article/nvidia-h200-vs-b200-comparing-datacenter-grade-accelerators), and more – on demand, at a fraction of the cost of traditional cloud providers. Our cloud GPU rental platform makes it easy. Just choose the hardware you need, spin up an instance in seconds, and pay only for the compute you use.

Get started with Vast today, and enjoy flexible, cost-effective GPU compute that scales with your needs\!


NVIDIA is developing the B30A AI chip for China, a Blackwell-based successor to the H20 with half the power of the B300. Learn what it means for AI exports, competition, and how you can still access unrestricted GPUs like the H200, B200, and A100 on Vast.ai.

Report: NVIDIA Developing New Blackwell-Based AI Chip for China


Remember when ChatGPT couldn't tell you the weather or gather top headlines? Are you still frustrated asking Gemini to create a JIRA ticket from a bug report email? Perhaps you're tired of Cursor hallucinating database column names because it's walled off from your live schema, so you're constantly copy-pasting the same context across your Claude and Cursor apps. Despite their eloquent email-writing powers, out-of-the-box Large Language Models (LLMs) are powerful predictors, not contextually-aware assistants. They operate in silos, forcing you to start with fresh context \-- or none at all \-- for every new task.

This constant context-juggling isn't just annoying, it's fragile. When a better model is released or your data governance rules change, do you have to re-implement your entire workflow? What if you want to easily swap out Gemini's new video generation model for one from Stable Diffusion?

With [Model Context Protocol](https://modelcontextprotocol.io/) (MCP), an open-source framework released by Anthropic for building truly helpful, environment-agnostic AI assistants, you can create a persistent, shareable context layer across models and tools. And with [Vast.ai](http://vast.ai)'s secure and affordable cloud GPU instances, implementing your own MCP has never been easier.

**Model Context Protocol Architecture**

MCP architecture is built on three key components:

-   Clients: The user-facing applications (like Cursor, a custom chatbot, or Claude) that coordinate between the user, the AI model, and the MCP server.
-   Servers: The adapter between your tools and AI models, serving tools, resources, and prompts to your LLM provider and other services.
-   Service Providers: The software tools and data sources (like Slack, Notion, GitHub, or your company's internal database) that have existing APIs.

With MCP, any developer can build a server to connect an API to any MCP-compatible client. You no longer have to wait for official integrations from LLM providers.

An MCP server provides three categories of capabilities to the model – tools, resources, and prompts. In the following guide, you'll focus on creating MCP tools and learn to create useful actions to spawn from your clients via a remote MCP server on [Vast.ai](http://Vast.ai)'s platform.

**Remote MCP Servers**

Initially, MCP architecture only supported locally running servers using stdio transport. Since summer 2025, it supports remote access via streamable-http, enabling scalable, AI-enabled workflows that integrate LLMs across your organization. Now teams can more easily integrate data services and build custom workflows within their workspaces.

​

[Vast.ai](http://vast.ai)'s affordable GPU instance rentals provide flexible and affordable hosting for as many MCP servers as you want to create – scaling up your team's AI workflows quickly and securely.

**Tutorial: Build a Stock Sentiment Analysis MCP**

Today, you'll build an MCP server for stock sentiment analysis correlation hosted on [Vast.ai](http://vast.ai)'s platform. This setup provides a foundation for team-based workflows. For example, a financial analyst company may use this MCP tool to probe further into market conditions all within the Claude interface, or spur discussions by connecting this output to a shared Slack channel, providing up to date and collective information to multiple teams. To begin, let's review what you'll need for this process.

[Vast.ai](http://vast.ai)'s GPU rentals offer space for a vast variety of MCP servers, easily accommodating 70B large language models or image diffusion models on the most up-to-date graphics card offerings. Their platform creates opportunities for creative, custom server integrations like creating a shared, product-specific image prompt for your product teams or pulling in proprietary climate energy data to your coding and marketing workflows.

By the end of this tutorial, you will understand how to build a similar bespoke MCP server for your organization's needs, and have a running blueprint for creating your collection of remote MCP servers on [Vast.ai](http://vast.ai). Let's get started.

**Prerequisites**

-   A [Vast.ai](http://vast.ai) account with an API key
-   A Hugging Face account with Mistral7B access token
-   Python 3.10+ environment

1. **Rent and Configure Your [Vast.ai](http://vast.ai) GPU Instance**  
   Start by selecting a GPU Instance from the Vast.ai console. For this project, you'll use a **Mistral 7B LLM** to summarize and analyze market sentiments, so we recommend using an **RTX 4090 GPU with 24GB of VRAM**. [Vast.ai](http://vast.ai) offers a variety of powerful graphics cards, including A100 and 80GB VRAM for larger LLM and diffusion tasks.  
   Begin by selecting a PyTorch (Vast) template. Modify this template to launch as an interactive shell server without a Jupyter notebook instance and with 24 GB of disk space. For more information on selecting templates, please refer to the [template guide.](https://docs.vast.ai/guides/instances/choosing/templates)<br />
   ![][image3]Now select an RTX 4090 offering at your required price point and select "Rent". Navigate to the "Instances" section in the sidebar. When the server is up and running, you should see "Running" next to "Status". Now you're ready to create your MCP server.  
   ![][image4]
2. **Create Your Environment**  
   In your terminal, install uv and set up your Python environment using the following command:

​​

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Now, restart your terminal to activate the \`uv\` command. Then, initialize a Python project.

```bash
uv init vast_mcp_server
cd vast_mcp_server
```

​​

Finally, create and activate your virtual environment to manage project dependencies.​

```bash
uv venv
source .venv/bin/activate
```

3. **Install Project Dependencies**  
   Create a requirements.txt file to outline project dependencies. You'll use this to write your server code locally and then install dependencies on your [Vast.ai](http://vast.ai) instance later.​

```bash
touch requirements.txt
```

Then add the following dependencies:​

```
fastmcp
pydantic
transformers
torch
requests
uvicorn
accelerate
```

Install the dependencies using the following command.

```bash
pip install -r requirements.txt
```

4. **Create Your MCP Server**  
   Now you'll create your first MCP server. This server will facilitate summarizing and analyzing news headlines using an LLM and correlating them with real-time stock data. This is a real-time task that out-of-the-box LLMs like Claude may struggle with. You'll create three MCP tools for stock analysis retrieval, sentiment analysis, and correlation – all available via your remote server.  
   This example uses [FastMCP](https://gofastmcp.com/getting-started/welcome) to facilitate simple protocol implementation. FastMCP uses simple decorators to handle the low-level implementation details, similar to the FastAPI approach.

Create a 'server.py' and 'test_server.py' file using the command:​

```bash
touch server.py test_server.py
```

First, you'll configure a Mistral 7B model for summarization and sentiment analysis for a given news article. You'll also set up a FastMCP server. Add the following code to the 'server.py' file.​

```python
import random
import logging
import asyncio
import os
import json
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelForCausalLM
from fastmcp import FastMCP
import torch


logging.basicConfig(level=logging.INFO)

# Initialize the MCP server
mcp = FastMCP("Financial Market Analyzer")

# Configure Mistral 7B LLM model
MODEL_NAME = "mistralai/Mistral-7B-Instruct-v0.2"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {DEVICE}")

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    device_map="auto"
)

class NewsItem(BaseModel):
    title: str
    description: str
```

Now, you'll create three tools. The first uses the Mistral 7B LLM to summarize and analyze a provided news article. In a real-world implementation, you may want to update this example to accept a URL as input. Add the following code:

```python
# Analyzes the sentiment and provides a summary of a news headline using a large language model.
@mcp.tool()
def analyze_sentiment(news_item: NewsItem):
    prompt = (
        f"You are a financial news analysis expert. "
        f"Analyze the following news headline and description. "
        f"Provide a brief summary and determine the sentiment as POSITIVE, NEGATIVE, or NEUTRAL. "
        f"Respond only with a JSON object containing two keys: 'summary' and 'sentiment'.\n\n"
        f"Headline: \"{news_item.title}\"\n"
        f"Description: \"{news_item.description}\"\n\n"
        f"JSON Response:"
    )

    inputs = tokenizer(prompt, return_tensors="pt").to(DEVICE)

    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)

    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

    try:
        json_start_index = generated_text.find("{")
        if json_start_index != -1:
            json_str = generated_text[json_start_index:]
            analysis_result = json.loads(json_str)
            summary = analysis_result.get("summary", "No summary found.")
            sentiment_label = analysis_result.get("sentiment", "NEUTRAL").upper()
        else:
            summary = "Could not generate a structured summary."
            sentiment_label = "NEUTRAL"
            logging.warning("LLM response did not contain a valid JSON object.")

    except json.JSONDecodeError as e:
        summary = "Could not parse LLM response."
        sentiment_label = "NEUTRAL"
        logging.error(f"Failed to parse JSON from LLM: {e}")

    logging.info(f"Analyzed '{news_item.title}': Sentiment={sentiment_label}, Summary='{summary}'")
    return {"title": news_item.title, "sentiment": sentiment_label, "summary": summary}
```

Now you'll add a stock retrieval tool. In this example, you'll mock the stock data. However, in an actual implementation, you might pull this data from a stock-related API. Add the following code:

```python
# Fetches mock real-time stock data. In a real-world implementation, you would connect to a secure financial database API.
# For this example, we are using a simple mock to simulate real-time data.
@mcp.tool()
def get_stock_data(symbol: str):
    price = round(random.uniform(100.0, 500.0), 2)
    volume = random.randint(100000, 5000000)
    logging.info(f"Fetched mock data for {symbol}: Price={price}, Volume={volume}")
    return {"symbol": symbol, "price": price, "volume": volume}
```

Finally, you'll add a tool that correlates the stock prices and news sentiment. Add the following code:

```python
@mcp.tool()
async def correlate_data(stock_data: dict, news_analysis: dict):
    sentiment = news_analysis['sentiment']
    price = stock_data['price']

    correlation_message = "No clear correlation found."
    if sentiment == "POSITIVE" and stock_data['volume'] > 2000000:
        correlation_message = f"POSITIVE news coincided with a high trading volume. Possible impact. News Summary: {news_analysis['summary']}"
    elif sentiment == "NEGATIVE" and price < 300:
        correlation_message = f"NEGATIVE news for a low-priced stock. Potential further decline. News Summary: {news_analysis['summary']}"

    logging.info(f"Correlated data: {correlation_message}")
    return {"stock_data": stock_data, "news_analysis": news_analysis, "correlation": correlation_message}
```

Now, you'll create the ability to run your MCP remote server via streamable-http. Add the following code:

```python
# Runs the MCP server remotely using streamable-http transport
if __name__ == "__main__":
    logging.info(f"MCP server started on port {os.getenv('PORT', 8080)}")
    asyncio.run(
        mcp.run_async(
            transport="streamable-http",
            host="0.0.0.0",
            port=os.getenv("PORT", 8080),
        )
    )
```

Great work! Now you have three MCP tools defined on your server. After deploying via [Vast.ai](http://vast.ai), you can access any of these tools remotely by integrating with Claude, Cursor, or other MCP supporting apps. Let's upload it to your [Vast.ai](http://vast.ai) GPU instance.

5. **Upload Your MCP Server to [Vast.ai](http://Vast.ai)**  
   First, retrieve your API key from the Vast.ai Console. Then, install the Vast.ai CLI in your Linux terminal using the following command.

```bash
pip install --upgrade vastai
```

Set your API key in your local Vast CLI configuration.

```bash
vastai set api-key #your-api-key-here
```

Now you'll be able to connect by SSH to your remote GPU instance. Navigate to your [Vast.ai](http://vast.ai) Console and select your GPU instance. Click on "Open Terminal Access". Here, you should see your SSH commands to connect to your GPU instance with the following structure:

```bash
ssh -p XXXXX root@XXX.XXX.XXX.XXX -L XXXX:localhost:XXXX
```

Navigate back to your terminal and use the command to connect to your instance. If your instance is running, you should see your terminal login to the remote [Vast.ai](http://vast.ai) instance with a welcome message.

Since you are using a Hugging Face model, you'll need to login to the Hugging Face CLI using the following:

```bash
huggingface-cli login
```

This command with prompt you for your access token from the Hugging Face website.

Then install the dependencies using:

```bash
pip install -r requirements.txt
```

Finally, you'll securely copy your [server.py](http://server.py) and requirement.txt files to the remote GPU. Navigate to the terminal you used to create your server environment and ensure the virtual environment is activated. Use the following command to move your files.

```bash
scp -P 29946 server.py requirements.txt root@172.81.127.37:/root/
```

Now you're ready to run and test your server!

6. **Run and Test Your Server**  
   In your open terminal connected to your [Vast.ai](http://vast.ai) GPU instance, simply run

```bash
uv run server.py
```

If all is well, you should see the models downloading and the server startup success message:

```
Starting MCP server 'Financial Market Analyzer' with transport 'streamable-http' on http://0.0.0.0:8080/mcp
INFO: Started server process [2135]
INFO: Waiting for application startup.
INFO:mcp.server.streamable_http_manager:StreamableHTTP session manager started
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080
```

As a final step, test the server connection to ensure your tools are available. In your Python environment, create a file called 'test_server.py' and paste the script below.

```python
import asyncio
import json

from fastmcp import Client

# Test the MCP server using streamable-http transport.
async def test_server():
    async with Client("http://localhost:8080/mcp") as client:

        # List available tools
        tools = await client.list_tools()
        for tool in tools:
            print(f"Tool found: {tool.name}")

        # Call the news analysis tool
        print("Calling stock tool for INVC")
        stock_result = await client.call_tool("get_stock_data", {"symbol": "INVC"})
        print(f"Result: {stock_result[0].text}")

        # Call the sentiment analysis and summarization tool
        print("2. Calling 'analyze_sentiment' tool...")
        news_item_payload = {
            "news_item": {
                "title": "Tech Giant Announces Record Profits",
                "description": "Shares surged after the company reported quarterly earnings that exceeded all analyst expectations, signaling strong growth."
            }
        }
        sentiment_result = await client.call_tool("analyze_sentiment", news_item_payload)
        print(f"Result: {sentiment_result[0].text}")

        # Call the correlation tool
        print("3. Calling 'correlate_data' tool...")
        correlation_payload = {
            "stock_data": json.loads(stock_result[0].text),
            "news_analysis": json.loads(sentiment_result[0].text)
        }
        correlation_result = await client.call_tool("correlate_data", correlation_payload)
        print(f"Result: {correlation_result[0].text}")

if __name__ == "__main__":
    asyncio.run(test_server())
```

```bash
uv run test_server.py
```

Navigate back to your Python environment terminal and run the script.

​If your server is working correctly, you should see output showing each of the tools and calling each tool in turn.

Congratulations! You have created your first MCP server on [Vast.ai](http://vast.ai). Now you can integrate this into Claude, Cursor, Windsurf, or any other MCP-supporting client, so you can customize, scale, and create more MCP servers for your organization and AI-enabled workflows.


Learn how to build and host a custom Model Context Protocol (MCP) server on Vast.ai's affordable GPU cloud. Step-by-step guide for creating AI-enabled workflows, integrating stock sentiment analysis, and scaling remote MCP servers with Claude, Cursor, and more.

Building Your First MCP Server on Vast.ai

Deploy OpenAI's GPT-OSS-20B on Vast.ai with Harmony SDK for transparent multi-channel AI reasoning. Step-by-step tutorial to build a weather assistant with function calls and reasoning visibility.

Build a Weather Assistant with OpenAI GPT-OSS and Harmony SDK on Vast.ai


## Introduction

Qwen-Image revolutionizes image generation with its exceptional ability to render complex text directly within images, supporting multiple languages including English and Chinese with perfect typographic detail. This powerful foundation model from Alibaba's Qwen series goes beyond simple image creation, offering advanced editing capabilities, style transfer, and precise control over visual elements.

What sets Qwen-Image apart is its nuanced understanding of text-image relationships. While other models struggle with text rendering, Qwen-Image seamlessly integrates readable text into visuals, making it ideal for creating signage, posters, infographics, and artistic compositions that require precise text placement and formatting.

Vast.ai provides the perfect platform for deploying Qwen-Image, offering access to high-performance GPUs at competitive prices with the flexibility to scale based on your needs.

In this guide, you'll deploy Qwen-Image on Vast.ai and explore its capabilities for text-integrated image generation.

## Setting Up the Environment

First, install the Vast CLI and configure your API key:

```bash
pip install --upgrade vastai
export VAST_API_KEY="<your-api-key>"
vastai set api-key $VAST_API_KEY
```

You can obtain your API key from the [Vast.ai Account Page](https://cloud.vast.ai/account/).

## Choosing the Right Hardware

Qwen-Image requires substantial GPU resources for optimal performance:

-   **GPU Memory**: Minimum 80GB VRAM for the full model
-   **Recommended Hardware**: NVIDIA A100 80GB or H100
-   **Storage**: 200GB disk space for model weights and dependencies

Search for suitable instances with these specifications:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 80 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 150 \
rentable = true"
```

This search filters for:

-   Modern GPUs with compute capability 7.5+
-   At least 80GB VRAM for the model
-   Static IP for stable connections
-   Verified providers for reliability
-   Adequate storage for model and outputs

## Deploying Qwen-Image

Deploy the instance using a pre-configured Docker image optimized for transformer models:

```bash
export INSTANCE_ID=<instance-id>
vastai create instance $INSTANCE_ID \
    --image huggingface/transformers-pytorch-gpu \
    --env '-p 8888:8888' \
    --disk 200 \
    --args bash -c "pip install jupyter && jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root --NotebookApp.token='' --NotebookApp.password=''"
```

**Key parameters explained:**

-   `--image`: HuggingFace's PyTorch GPU image with ML libraries pre-installed
-   `--env '-p 8888:8888'`: Maps Jupyter's default port for external access
-   `--disk 200`: Allocates 200GB for model weights and generated images
-   `--args`: Auto-installs Jupyter and launches a passwordless notebook server

## Connecting to Your Instance

After deployment:

1. Navigate to the [Instances tab](https://cloud.vast.ai/instances/)
2. Find your instance and wait for "Running" status
3. Look for the "Open Ports" section in instance details
4. Click the port mapping link (e.g., `123.123.123.123:12345 -> 8888/tcp`)
5. This opens Jupyter directly in your browser
6. Create a new notebook or upload the example notebook if available

## Installing Dependencies

Install the required packages in your notebook:

```bash
apt install -y git
pip install git+https://github.com/huggingface/diffusers
```

The latest diffusers library from GitHub includes Qwen-Image support.

## Loading the Model

Initialize Qwen-Image with optimal settings for GPU deployment:

```python
from diffusers import DiffusionPipeline
import torch

model_name = "Qwen/Qwen-Image"

# Load the pipeline
if torch.cuda.is_available():
    torch_dtype = torch.bfloat16
    device = "cuda"
else:
    torch_dtype = torch.float32
    device = "cpu"

pipe = DiffusionPipeline.from_pretrained(model_name, torch_dtype=torch_dtype)
pipe = pipe.to(device)
```

The model uses bfloat16 precision on GPU for optimal memory usage and performance.

## Using Qwen-Image

Now let's explore Qwen-Image's text rendering capabilities with examples:

### Example 1: Complex Text Integration with Multiple Languages

```python
import os

positive_magic = {
    "en": "Ultra HD, 4K, cinematic composition.", # for english prompt
}

# Create img folder if it doesn't exist
os.makedirs("img", exist_ok=True)

# Generate image
prompt = '''A coffee shop entrance features a chalkboard sign reading "Qwen Coffee 😊 $2 per cup," with a neon light beside it displaying "Welcome". Next to it hangs a poster showing a beautiful Chinese woman, and beneath the poster is written "π≈3.1415926-53589793-23846264-33832795-02384197". Ultra HD, 4K, cinematic composition'''
negative_prompt = " " # using an empty string if you do not have specific concept to remove

# Set image dimensions (16:9 aspect ratio)
width, height = 1664, 928

image = pipe(
    prompt=prompt + positive_magic["en"],
    negative_prompt=negative_prompt,
    width=width,
    height=height,
    num_inference_steps=50,
    true_cfg_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(42)
).images[0]

# Save image to img folder
image.save("img/example_1.png")

# Display the image
display(image)
```

### Output

![Qwen Coffee Shop with Text](/uploads/example_1.webp)

This example demonstrates Qwen-Image's ability to:

-   Render multiple text elements (chalkboard, neon sign, poster)
-   Handle emojis and special characters
-   Integrate mathematical constants with precise formatting
-   Maintain readability across different text styles

### Example 2: Creative Fantasy Scene

```python
positive_magic = {
    "en": "Ultra HD, 4K, cinematic composition.", # for english prompt
}

# Create img folder if it doesn't exist
os.makedirs("img", exist_ok=True)

# Generate image
prompt = '''Rare photo: UFO hovering over a yeti riding the Loch Ness monster.'''
negative_prompt = " " # using an empty string if you do not have specific concept to remove

# Set image dimensions (16:9 aspect ratio)
width, height = 1664, 928

image = pipe(
    prompt=prompt + positive_magic["en"],
    negative_prompt=negative_prompt,
    width=width,
    height=height,
    num_inference_steps=50,
    true_cfg_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(42)
).images[0]

# Save image to img folder
image.save("img/example_2.png")

# Display the image
display(image)
```

### Output

![Fantasy Scene with UFO and Mythical Creatures](/uploads/example_2.webp)

This showcases Qwen-Image's creative capabilities in composing imaginative scenarios.

### Example 3: Futuristic Space Scene

```python
positive_magic = {
    "en": "Ultra HD, 4K, cinematic composition.", # for english prompt
}

# Create img folder if it doesn't exist
os.makedirs("img", exist_ok=True)

# Generate image
prompt = '''Now I want you to show a spaceships approaching Andromeda Galaxy!, it should be futuristic spaceships out on exploring universe'''
negative_prompt = " " # using an empty string if you do not have specific concept to remove

# Set image dimensions (16:9 aspect ratio)
width, height = 1664, 928

image = pipe(
    prompt=prompt + positive_magic["en"],
    negative_prompt=negative_prompt,
    width=width,
    height=height,
    num_inference_steps=50,
    true_cfg_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(42)
).images[0]

# Save image to img folder
image.save("img/example_3.png")

# Display the image
display(image)
```

### Output

![Futuristic Spaceships Approaching Andromeda](/uploads/example_3.webp)

This demonstrates Qwen-Image's ability to create detailed sci-fi scenes.

## Advanced Features and Capabilities

### Flexible Aspect Ratios

Qwen-Image supports various aspect ratios for different use cases:

```python
aspect_ratios = {
    "1:1": (1328, 1328),    # Social media posts
    "16:9": (1664, 928),    # Widescreen displays
    "9:16": (928, 1664),    # Mobile/portrait content
    "4:3": (1472, 1140),    # Traditional displays
    "3:4": (1140, 1472),    # Portrait photos
    "3:2": (1584, 1056),    # Standard photos
    "2:3": (1056, 1584),    # Portrait standard
}
```

### Generation Parameters

Fine-tune your outputs with these key parameters:

-   `num_inference_steps`: Higher values (50-100) for better quality
-   `true_cfg_scale`: Controls prompt adherence (3.0-7.0 range)
-   `generator`: Set seed for reproducible results
-   `negative_prompt`: Specify elements to avoid

## Practical Applications

Qwen-Image's unique text rendering capabilities enable numerous applications:

-   **Marketing Materials**: Create posters, banners, and advertisements with perfectly rendered text
-   **Educational Content**: Generate infographics and diagrams with clear labels and annotations
-   **Social Media**: Design posts with integrated text overlays and captions
-   **Multilingual Content**: Produce visuals with text in multiple languages
-   **Creative Projects**: Combine artistic imagery with typographic elements
-   **UI/UX Mockups**: Generate interface designs with readable text elements

## Performance Optimization

To maximize performance on Vast.ai:

1. **Use bfloat16 precision**: Reduces memory usage while maintaining quality
2. **Batch generation**: Process multiple prompts together when possible
3. **Cache the model**: Keep it loaded between generations to avoid reload overhead
4. **Optimize resolution**: Start with lower resolutions for testing, then increase for final outputs

## Conclusion

Qwen-Image represents a significant advancement in image generation technology, particularly in its ability to seamlessly integrate readable text into visual content. Its support for multiple languages, mathematical notation, and various text styles makes it uniquely suited for projects requiring precise text-image integration.

By deploying on Vast.ai, you gain access to powerful GPUs capable of running this advanced model efficiently. Whether you're creating marketing materials, educational content, or artistic compositions, Qwen-Image provides the tools to generate professional-quality visuals with integrated text.

### Next Steps

Try these to explore further:

-   **Experiment with multilingual prompts**: Test the model's capabilities with Chinese, Japanese, or mixed-language text
-   **Create branded content**: Generate marketing materials with your company's text and logos
-   **Build an API service**: Wrap the model in a REST API for integration with your applications
-   **Explore style transfer**: Use the model's editing capabilities to transform existing images

[Get started on Vast.ai](https://cloud.vast.ai/) and experience Qwen-Image today!


Deploy Qwen-Image on Vast.ai to create stunning visuals with perfectly rendered text, multilingual support, and advanced editing features.

Deploying Qwen-Image for Advanced Text-Integrated Image Generation on Vast.ai


If you're working at the cutting edge of AI and high-performance computing (HPC), NVIDIA's H200 and B200 GPUs are likely already very much on your radar. They're each designed to handle the massive compute demands of datacenter-class workloads – such as training trillion-parameter models and running high-speed inference at scale.

While both GPUs deliver incredible power, it's crucial to understand the differences between the two in order to make the right choice for your needs.

Today, we'll explore how the **H200** and **B200** measure up against each other and how each fits into real-world deployments. Which one is right for you? Let's take a look.

### NVIDIA H200: Hopper-Based Heavyweight

Building on the foundation laid by the H100, the NVIDIA H200 is based on the same Hopper architecture while bringing substantial improvements in memory capacity and bandwidth. This means fewer bottlenecks, an advantage that becomes increasingly important as you move further into enterprise scale.

The H200 nearly doubles the memory of its predecessor, boasting **141 GB of HBM3e memory** and **4.8 TB/s of bandwidth** – delivering about **1.4X faster** data access. MLPerf benchmarks using Llama 2 70B demonstrated the H200's power: it reached over 31,000 tokens per second, about **45% faster** than the H100.

With the same **fourth-gen NVLink**, **PCIe Gen5**, **MIG support**, and full Tensor Core and Transformer Engine capabilities, the H200 is a solid next step for existing Hopper-based deployments. You don't have to rethink your entire infrastructure; you just get more horsepower.

To sum up, here are some of the most notable features of the H200:

-   **141 GB of HBM3e memory at 4.8 TB/s** – almost double the memory of the H100 with 1.4X the bandwidth.

-   **Up to 1.9X improved LLM inference performance**, with more tokens per second and higher throughput for gen AI workloads compared to its predecessor.

-   **Lower energy use and total cost of ownership** (TCO) over the H100 thanks to improved energy efficiency and thermal management.

-   **An upgrade path exists on qualified HGX/MGX platforms**, so that little to no reconfiguration is required.

In short, the H200 represents the very best of the Hopper-based lineup. When it was introduced in late 2024, it set a new standard for memory-intensive and high-throughput workloads in datacenter GPUs – and it's still one of the most powerful GPUs on the market today.

The B200, on the other hand, is an entire generational leap forward.

### NVIDIA B200: The New Generation

Built on NVIDIA's advanced Blackwell architecture, the B200 features **192 GB of ultra-fast HBM3e memory** with **8.0 TB/s of memory bandwidth.** Those are some staggering numbers, even compared to the H200's respectable 141 GB at 4.8 TB/s.

The Blackwell architecture also introduces **fifth-gen Tensor Cores** and **dual Transformer Engines**, which are particularly useful for AI workloads with long context windows and heavy token parallelism. The B200 scales much more efficiently than the H200 for the latest and most advanced model classes.

In fact, scaling is a major area where the B200 stands out. With **fifth-gen** **NVLink**, its inter-GPU bandwidth reaches **1.8 TB/s** within a node – compared to Hopper's 900 GB/s – with cross-node scaling supported via the NVLink Switch System.

Of course, this may not matter to you very much if you're only using a single card. At cluster scale, however, or in multi-GPU systems like the DGX B200 (which connects eight B200 GPUs together), the improvements in distributed training efficiency really make a difference.

The B200 truly sits at the frontier of performance. Its raw capacity and speed can support next-gen inference pipelines and AI workloads at hyperscale. According to NVIDIA, it delivers **3X the training performance** and **15X the inference performance** of the previous generation.

These gains do come at a cost in power draw. The B200 has a max thermal design power (TDP) of **1000 W** compared to the H200 at 700 W – which means it also requires far more robust cooling solutions. But the extra power consumption may just be worth it for the boost in performance.

To recap, a few advantages of the B200 include:

-   **192 GB of HBM3e memory at 8.0 TB/s** – ultra-high capacity that's well suited for trillion-parameter models and allows training of AI models _entirely in GPU memory_.

-   **Fifth-gen Tensor Cores and second-gen Transformer Engines** for higher efficiency in FP8, FP16, and mixed precision performance, especially in long-context and multi-modal workloads.

-   **NVLink 5 interconnect** with up to **1.8 TB/s** GPU-to-GPU bandwidth for multi-node scaling.

-   **Fine-grain micro-tensor scaling** that optimizes performance and accuracy, enabling 4-bit floating point (**FP4**) AI.

The B200 was built to shatter performance records. As you can imagine, it has an eyewatering price tag to match. Nonetheless, it can be helpful to compare the B200 and H200 based on specs alone.

### H200 vs. B200: Key Specs Compared

Here's a side-by-side look at some key features of the H200 and B200:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Feature
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                H200
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                B200
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2">Hopper</td>
            <td class="border border-gray-300 px-4 py-2">Blackwell</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory
            </td>
            <td class="border border-gray-300 px-4 py-2">141 GB HBM3e</td>
            <td class="border border-gray-300 px-4 py-2">192 GB HBM3e</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Bandwidth
            </td>
            <td class="border border-gray-300 px-4 py-2">4.8 TB/s</td>
            <td class="border border-gray-300 px-4 py-2">8 TB/s</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                CUDA Cores
            </td>
            <td class="border border-gray-300 px-4 py-2">16,896</td>
            <td class="border border-gray-300 px-4 py-2">16,896 x2</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Cores
            </td>
            <td class="border border-gray-300 px-4 py-2">528 (4th Gen)</td>
            <td class="border border-gray-300 px-4 py-2">528 x2 (5th Gen)</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Boost Clock
            </td>
            <td class="border border-gray-300 px-4 py-2">1.98 GHz</td>
            <td class="border border-gray-300 px-4 py-2">1.98 GHz</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Transformer Engine
            </td>
            <td class="border border-gray-300 px-4 py-2">Single</td>
            <td class="border border-gray-300 px-4 py-2">Single (2nd gen)</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Form Factor
            </td>
            <td class="border border-gray-300 px-4 py-2">SXM</td>
            <td class="border border-gray-300 px-4 py-2">SXM</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Interconnect
            </td>
            <td class="border border-gray-300 px-4 py-2">
                NVLink 4 (900 GB/s) per GPU
            </td>
            <td class="border border-gray-300 px-4 py-2">
                NVLink 5 (1.8 TB/s) per GPU
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Precision Formats
            </td>
            <td class="border border-gray-300 px-4 py-2">
                FP8, FP16/BF16, TF32, FP32, FP64, INT8
            </td>
            <td class="border border-gray-300 px-4 py-2">
                FP4, FP6, FP8, FP16/BF16, TF32, FP32, FP64, INT8
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Multi-Instance GPUs
            </td>
            <td class="border border-gray-300 px-4 py-2">Up to 7</td>
            <td class="border border-gray-300 px-4 py-2">Up to 7</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Max Thermal Design Power (TDP)
            </td>
            <td class="border border-gray-300 px-4 py-2">700 W</td>
            <td class="border border-gray-300 px-4 py-2">1000 W</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Recommended Power Supply
            </td>
            <td class="border border-gray-300 px-4 py-2">1100 W</td>
            <td class="border border-gray-300 px-4 py-2">1400 W</td>
        </tr>
    </tbody>
</table>

Choosing the right GPU often comes down to more than just specs, however.

### Use Cases: Which GPU Should You Choose?

While both the H200 and B200 deliver game-changing performance capabilities, they each have distinct advantages depending on your needs.

#### You should consider the H200 if you prefer:

-   **A high-performance GPU** ideal for fine-tuning LLMs and high-throughput inference at the enterprise level without getting into exascale territory.

-   **A proven workhorse** that balances strong performance with lower TDP and cost.

-   **Being able to seamlessly upgrade** from existing Hopper deployments without overhauling infrastructure.

#### Or the B200 might be the right choice if you need:

-   **To push model boundaries** and handle long-context and/or multi-modal foundation model training at scale.

-   **The absolute maximum throughput and efficiency** for multi-GPU or multi-node clusters.

-   **A future-proof investment** for infrastructure built to support the next generation of cutting-edge AI and AGI.

### Conclusion

At **Vast.ai**, we know that the choice between GPUs like the H200 and B200 can depend on a lot more than just performance. Access and cost are important factors, as well.

The H200 and B200 represent the very best of NVIDIA's current lineup, and they're among the most powerful and expensive GPUs in the world today. Buying them outright is undoubtedly out of reach for most teams.

That's what we're here for. With Vast, you can rent high-end GPUs like the [A100](https://vast.ai/article/H100-vs-A100-Comparing-two-Powerhouse-GPUs), [H100](https://vast.ai/article/nvidia-h100-vs-l40s-power-meets-versatility), [H200](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights), and soon the B200, with on-demand pricing and spot instances – and pay only for what you use.

On average, **Vast users save 5–6X on GPU compute** compared to traditional cloud providers. You'll find a variety of options on our platform to fit your needs, so you can focus on what really matters: your work, not your budget.

Explore Vast today and enjoy enterprise-grade compute without enterprise-level costs.


The H200 and B200 GPUs represent the top end of NVIDIA's current lineup for AI and high-performance computing (HPC). Both are designed for serious, large-scale workloads – from training foundation models to running complex scientific simulations.

NVIDIA H200 vs. B200: Comparing Datacenter-Grade Accelerators

Deploy DeepSeek R1 on Vast.ai with vLLM and integrate Crew AI for scalable multi-agent workflows. Create cost-effective, OpenAI-compatible AI endpoints.

Running DeepSeek R1 + Crew AI on Vast.ai


## Introduction

OpenAI has just released the GPT-OSS family of models, marking their return to open-weight model releases. The family includes GPT-OSS-120B (120 billion parameters) and GPT-OSS-20B (20 billion parameters), offering developers access to state-of-the-art language capabilities previously locked behind proprietary APIs.

GPT-OSS offers configurable reasoning capabilities with reasoning levels - from "low" for quick responses to "high" for complex problem-solving. The model uses OpenAI's harmony encoding system, which structures conversations and reasoning in a sophisticated format that enables fine-grained control over the model's thinking process.

This breakthrough means you can now:

-   Deploy OpenAI-quality models on your own infrastructure
-   Scale inference according to your exact needs

Vast.ai provides the perfect platform for running GPT-OSS models. With its marketplace of GPUs, you can choose the right hardware for your needs - whether running the efficient 20B model or the powerful 120B variant.

In this guide, we'll show you how to deploy both GPT-OSS models on Vast.ai using vLLM for optimized inference, with a focus on the 120B model. You'll learn how to interact with these models using the harmony encoding system for different reasoning levels.

## Setting Up the Environment

Before we can deploy our model, we need to set up our Vast.ai environment. First, install the Vast SDK:

```bash
# Install required packages
pip install --upgrade vastai
pip install --upgrade openai
pip install --upgrade openai-harmony
```

Set up your Vast API key (available from your [Account Page](https://cloud.vast.ai/account/)):

```bash
# Set your Vast.ai API key
export VAST_API_KEY="" # Your key here
vastai set api-key $VAST_API_KEY
```

## Choosing the Right Hardware

The GPT-OSS models have different hardware requirements:

-   **GPT-OSS-120B**: Requires an H100 GPU with 80GB VRAM
-   **GPT-OSS-20B**: Requires 16GB+ VRAM

For this guide, we'll demonstrate with the 120B model on H100. Let's search for suitable instances:

```bash
# Search for suitable GPU instances
vastai search offers " \
gpu_name = H100_SXM \
geolocation=US \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 120 \
rentable = true"
```

## Deploying GPT-OSS with vLLM

Now let's deploy our GPT-OSS instance using vLLM's optimized inference server. We'll use the latest vLLM image with special GPT-OSS support. The pip install commands are from the model's Hugging Face pages ([120B](https://huggingface.co/openai/gpt-oss-120b), [20B](https://huggingface.co/openai/gpt-oss-20b)).

### Deploying GPT-OSS-120B (H100 recommended)

```bash
# Deploy vLLM instance for 120B model
export INSTANCE_ID= # Insert instance ID from search results

vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 160  --onstart-cmd 'uv pip install --system --upgrade transformers kernels torch openai; uv pip install --system --pre vllm==0.10.1+gptoss --extra-index-url https://wheels.vllm.ai/gpt-oss/ --extra-index-url https://download.pytorch.org/whl/nightly/cu128; vllm serve openai/gpt-oss-120b --max-model-len 80000'
```

### Alternative: Deploying GPT-OSS-20B on A100

Here's how to deploy the 20B model on an A100 GPU:

```bash
# Deploy vLLM instance for 20B model
export INSTANCE_ID= # Insert instance ID from search results

vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 80  --onstart-cmd 'uv pip install --system --upgrade transformers kernels torch openai; uv pip install --system --pre vllm==0.10.1+gptoss --extra-index-url https://wheels.vllm.ai/gpt-oss/ --extra-index-url https://download.pytorch.org/whl/nightly/cu128; vllm serve openai/gpt-oss-20b'
```

After deployment, wait for the model to download and start serving. You can monitor the logs to see when it's ready. Once running, find your instance's IP address and port from the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console.

## Interacting with GPT-OSS Using Harmony Encoding

GPT-OSS uses OpenAI's harmony encoding system, which provides structured conversation formatting and enables different reasoning levels. Let's set up our client:

```python
from openai import OpenAI
from openai_harmony import (
    load_harmony_encoding,
    HarmonyEncodingName,
    Role,
    Message,
    Conversation,
    SystemContent,
    DeveloperContent,
)

# Your server details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize client
client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

# Load harmony encoding
enc = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)

## Simple Chat Function
def chat_gpt_oss(prompt, reasoning="medium"):
    """
    Simple chat with GPT-OSS
    reasoning: "low", "medium", or "high"
    """
    # Create conversation with harmony format
    conversation = Conversation.from_messages([
        Message.from_role_and_content(Role.SYSTEM, SystemContent.new()),
        Message.from_role_and_content(
            Role.DEVELOPER,
            DeveloperContent.new().with_instructions(f"Reasoning: {reasoning}")
        ),
        Message.from_role_and_content(Role.USER, prompt),
    ])

    # Get formatted prompt
    formatted_prompt = enc.render_conversation_for_completion(conversation, Role.ASSISTANT)

    # Make API call
    response = client.completions.create(
        model="openai/gpt-oss-120b",
        prompt=formatted_prompt,
        max_tokens=2000,
        temperature=0.7,
    )

    return response.choices[0].text


prompt = "Explain quantum computing to a 4th grader. Keep in concise."
```

## Comparing Reasoning Levels

One of GPT-OSS's most interesting features is its configurable reasoning levels. Let's see how the model responds differently to the same prompt with low and high reasoning:

### Low Reasoning Response

```python
result = chat_gpt_oss(prompt, "low")
print("----LOW REASONING----")
print(result)
```

Output:

```
----LOW REASONING----
analysisWe need to explain quantum computing to a 4th grader, concise. Use simple analogies, like bits as light switches, qubits as coins that can be heads, tails, or both. Keep short.assistantfinal**Quantum Computing in a Nutshell (for a 4th‑grader)**

1. **Regular computers = light switches**
   - A normal computer works with bits that are either **0** (off) or **1** (on), just like a light switch that's either down or up.

2. **Quantum computers = magic coins**
   - A quantum computer uses **qubits**.
   - Think of a qubit like a coin that can be **heads (0)**, **tails (1)**, **or both at the same time** while it's spinning in the air. That "both at once" is called **superposition**.

3. **Working together**
   - When you have many regular bits, you have to try every possible combination one after another.
   - With many qubits, they can be in many combinations **all at once** because they can be linked together in a special way called **entanglement**. It's like a group of magic coins that always know what the others are doing, even if they're far apart.

4. **Why it matters**
   - Because of superposition and entanglement, a quantum computer can solve some puzzles (like cracking codes or simulating molecules) **much faster** than a regular computer.

**Bottom line:** A regular computer is like a row of on/off switches; a quantum computer is like a bunch of spinning magic coins that can be many things at once, letting it try many solutions at the same time.
```

### High Reasoning Response

```python
result = chat_gpt_oss(prompt, "high")
print("----HIGH REASONING----")
print(result)
```

Output:

```
----HIGH REASONING----
analysisWe need to explain quantum computing to a 4th grader, concise. Use simple analogies, avoid jargon, but keep accurate. Should be short. Provide analogies like coins, dice, superposition, entanglement. Keep concise.assistantfinal**What is a quantum computer?**

Imagine you have a super‑smart robot that can solve puzzles much faster than a regular computer. A quantum computer is that robot, but instead of using normal "on/off" switches (bits) like a regular computer, it uses tiny, magical switches called **qubits**.

### Bits vs. Qubits
- **Bit (regular computer):** Like a tiny light that can be **off (0)** or **on (1)**.
- **Qubit (quantum computer):** Like a tiny spinning top that can be **0**, **1**, **or both at the same time** (this is called *superposition*).

### Why is that cool?
If you have 10 bits, they can show only one of 2¹⁰ = 1,024 possible patterns at a time.
If you have 10 qubits, they can be in all 2¹⁰ = 1,024 patterns **at once**! That lets a quantum computer try many solutions together.

### A Simple Analogy
Think of a maze puzzle:

- **Regular computer:** Sends a single mouse down the maze, step by step, until it finds the exit.
- **Quantum computer:** Sends a whole swarm of mice that can explore every possible path at the same time and tell you the exit almost instantly.

### Entanglement (Teamwork)
Sometimes two qubits become "best friends" and link together, no matter how far apart they are. Changing one instantly changes the other. This is called **entanglement**, and it lets quantum computers coordinate their work in a super‑efficient way.

### Bottom Line
A quantum computer uses qubits that can be 0, 1, or both at once, and they can be entangled with each other. This lets the machine look at many possibilities at the same time, solving certain problems way faster than ordinary computers.

So, it's like a super‑fast, super‑clever puzzle‑solver that can try lots of answers all at once!
```

## Comparing Low vs High Reasoning Outputs

The quantum computing explanations demonstrate the distinct capabilities of GPT-OSS-120B's reasoning levels. With low reasoning, the model produced a concise, structured explanation using simple analogies like "magic coins" and "light switches" - delivering clear concepts in a compact format suitable for quick understanding.

The high reasoning response revealed significantly more sophisticated thinking. The model provided detailed analogies like the maze puzzle with mice, explained mathematical concepts (2¹⁰ patterns), and structured the explanation with clear sections and formatting. Most notably, it offered multiple perspectives on the same concepts - from the spinning top analogy to the teamwork explanation of entanglement.

This difference in reasoning depth represents a significant advancement in model control. Rather than simply adjusting temperature or other parameters, GPT-OSS allows direct specification of how thoroughly the model should think through problems. The low reasoning approach prioritized efficiency while maintaining accuracy, while high reasoning prioritized comprehensive understanding with rich examples and detailed explanations.

## Conclusion

OpenAI's first open-source model thus introduces not just accessibility to advanced language capabilities, but a new paradigm for controlling model reasoning depth - enabling applications to dynamically adjust between quick responses and thorough analysis based on context and requirements.


Learn how to deploy and run OpenAI's GPT-OSS models on Vast.ai using vLLM, with configurable reasoning levels and harmony encoding system.

Running OpenAI's GPT-OSS on Vast.ai


At **Vast.ai**, we believe privacy and security are inextricably woven into our mission to equip customers worldwide with accessible, scalable GPUs. In honor of this commitment, we are pleased to announce that we have achieved **SOC 2 Type II certification** – an independent assurance that with us, your data is secure, reliably available, and continuously protected.

Vast.ai’s Enduring Security Commitment

Earlier this year, our [SOC 2 Type I audit](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) confirmed we have the right internal controls in place to meet trust services criteria for data security, integrity, and privacy. The Type II audit put those controls to the test, evaluating their operational effectiveness over an extended period.

Achieving **SOC 2 Type II certification** is independent validation that the security infrastructure we've built over 6+ years of trusted service continues to meet rigorous [compliance standards](https://vast.ai/article/security-and-compliance-at-vast-ai) in day-to-day execution. This achievement strengthens our ability to support the evolving security needs of enterprises, research institutions, and organizations building on our infrastructure.

Here's a quick look at SOC 2 Type II and what it entails.

### What Is SOC 2 Type II?

System and Organization Controls 2 (SOC 2\) audits – developed by the American Institute of Certified Public Accountants (AICPA) – protect customer data by rigorously evaluating an organization’s internal controls across security, confidentiality, privacy, and processing integrity benchmarks.

- **Type I** reports provide a control design snapshot at a single point in time.

- **Type II** reports go further, assessing how those controls perform over months of continuous operation – confirming they're both well-designed and consistently implemented.

Achieving both Type I and Type II certifications means your sensitive data is secure, available, and privately stored across all our systems. This establishes a foundation of enduring excellence aligned with rigorous industry standards for all customers, which we can tailor to your unique data security and regulatory compliance needs to scale infrastructure and further reduce risks.

After completing our initial 3-month Type II audit, we’ve launched a standard 12-month cycle to ensure we have no gaps in coverage. This two-phase strategy lets us move fast on improvements, then settle into an annual rhythm of verification. We are now in the 12-month cycle with continuous coverage year round.

### Vast.ai's Secure Cloud Offering

For customers with stricter security and compliance requirements, our [**Secure Cloud**](https://cloud.vast.ai/?secure_cloud=true) offering provides high-security GPU access through our certified datacenter partners. This environment, already backed by stringent operational and security controls, now benefits from the added assurance of a SOC 2 Type II audit across our platform infrastructure.

Our **[Trust Center](https://trust.vast.ai/?utm_source=blog)** offers a deeper look into our controls, compliance stance, FAQs and Subprocessors, and our other compliance certifications provide additional safeguards.

Why This Matters

Earning our **SOC 2 Type II certification** is another step forward in strengthening our foundation of trust and security that our customers rely on every day. It reflects both our adherence to proven industry practices and our ongoing commitment to transparency.

This achievement isn't the end of our compliance journey – it's part of an ongoing initiative. Vast.ai will undergo SOC 2 Type II audits every 12 months to ensure continual coverage.

Additionally, we've requested an [**SOC 3 report**](https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-3), which will provide a publicly available summary of our security posture for those who want to review our controls without accessing the full SOC 2 report.

### Our Ongoing Commitment

Achieving the **SOC 2 Type II certification** is a crucial step in our compliance roadmap as we continue to build trustworthy and dependable GPU infrastructure for our users. Alongside other security and [compliance measures](https://vast.ai/compliance) across cloud and AI services, such as our [SOC 2 Type I certification](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) and [Vulnerability Bounty Program](https://vast.ai/article/announcing-the-vast-ai-vulnerability-bounty-program), we are weaving security and trust throughout our platform for every member of our community.

Through this integrated approach, we’re dedicated to protecting the privacy and integrity of the data entrusted to us by developers, researchers, and enterprise teams around the world. We take this responsibility seriously as a distributed peer-to-peer platform powering global compute, and we look forward to sharing future updates as we continue to advance our security and compliance roadmap.

For more information on Vast.ai's comprehensive security and compliance stance, or to request a copy of our SOC 2 reports, please contact us at [compliance@vast.ai](mailto:compliance@vast.ai).

Thank you to our team at Vast.ai and our audit partners for helping us achieve this milestone.

We’re committed to protecting your data and upholding the trust you place in us, and we’ll keep you updated on our ongoing progress in our mission to provide powerful, dependable GPUs.


Vast.ai has achieved SOC 2 Type II certification, demonstrating our commitment to enterprise-grade security, data protection, and compliance. Learn about our comprehensive security controls, Secure Cloud offering, and ongoing compliance roadmap that ensures your data remains secure and protected.

Vast.ai Achieves SOC 2 Type II Certification


The landscape of AI inference is rapidly evolving, with organizations seeking cost-effective solutions that don't compromise on performance or control. While cloud providers offer convenient managed services, they often come with premium pricing and limited customization options. On the other hand, on-premises deployments provide control but require significant hardware investments.

A hybrid approach offers the best of both worlds: maintaining local control over your inference pipeline while leveraging cost-effective remote GPU resources. This blog post demonstrates how to implement such a system using LiteLLM as a local proxy and Vast.ai for remote GPU hosting.

## The Hybrid Architecture Advantage

This architecture combines local control with cost-effective cloud GPU access through two key components:

**LiteLLM** is a Python SDK and proxy server that provides a unified OpenAI-compatible interface for 100+ LLM APIs from different providers. It includes features like cost tracking, rate limiting, and logging capabilities when used as a proxy server.

**Vast.ai** is a GPU marketplace that offers access to over 20,000 GPUs from secure datacenters. According to their website, users can save up to 80% compared to traditional cloud services with their pay-as-you-go pricing model.

## What You'll Learn

In this blog post, we'll deploy a complete hybrid inference pipeline:

1. Deploy a vLLM server with the DeepSeek-R1 model on Vast.ai
2. Configure LiteLLM locally to proxy requests to the remote server
3. Test the complete pipeline using OpenAI client libraries
4. Demonstrate advanced reasoning capabilities

The result is a flexible, cost-effective inference setup that you control locally while leveraging remote GPU resources.

## Deploy LiteLLM + vLLM Pipeline on Vast.ai

Let's start by installing and configuring the Vast AI API client.

### Install Vast AI Client

Get your API key from the [Account Page](https://cloud.vast.ai/account/) in the Vast Console.

```bash
pip install --upgrade vastai
```

```bash
# Here we will set our api key
export VAST_API_KEY="" #Your key here
vastai set api-key $VAST_API_KEY
```

### Hardware Requirements

For our LiteLLM + vLLM pipeline, we need a GPU with 24GB+ RAM for the DeepSeek-R1 model weights and KV cache, plus a static IP address with direct ports for stable connections.

Let's search for suitable instances:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 24 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

## Deploy vLLM on Vast.ai

Choose an instance ID from the search results above and deploy using the Docker command below.

This deployment uses the vllm/vllm-openai:latest image to serve the deepseek-ai/DeepSeek-R1-0528-Qwen3-8B model with reasoning capabilities on port 8000:

```bash
export INSTANCE_ID="" #insert instance ID

vastai create instance $INSTANCE_ID \
  --image vllm/vllm-openai:latest \
  --env '-p 8000:8000' \
  --disk 60 \
  --args \
    --model deepseek-ai/DeepSeek-R1-0528-Qwen3-8B \
    --served-model-name deepseek \
    --max-model-len 4096 \
    --reasoning-parser qwen3 \
```

## Get Vast IP and Port

After deployment, go to the [Instances Tab](https://cloud.vast.ai/instances/) in the Vast AI Console and find your instance.

Click the IP address button at the top of the instance. A panel will show the IP address and forwarded ports:

```
Open Ports
XX.XX.XXX.XX:YYYY -> 8000/tcp
```

Note down VAST_IP (XX.XX.XXX.XX) and VAST_PORT (YYYY) for the next steps.

## Installing LiteLLM and Dependencies

Now let's install LiteLLM locally to proxy requests to our Vast.ai server:

```bash
pip install litellm
pip install 'litellm[proxy]'
```

### Configure LiteLLM

Set your Vast.ai instance details from the previous step:

```python
# Configure your Vast.ai instance details
VAST_IP = ""
VAST_PORT = ""
MODEL_NAME = "deepseek"

# Configure LiteLLM settings
LITELLM_PORT = 4000  # Change this if port 4000 is in use

# Create the API base URL
API_BASE = f"http://{VAST_IP}:{VAST_PORT}/v1"
print(f"Vast.ai API Base URL: {API_BASE}")
print(f"LiteLLM will run on port: {LITELLM_PORT}")
```

```python
# Write config file with variables
config_content = f"""model_list:
  - model_name: {MODEL_NAME}
    litellm_params:
      model: openai/{MODEL_NAME}
      api_base: {API_BASE}
      api_key: fake-key
general_settings:
  master_key: fake-key
"""

with open('litellm_config.yaml', 'w') as f:
    f.write(config_content)
    
print("Config file created with:")
print(f"- Model: {MODEL_NAME}")
print(f"- API Base: {API_BASE}")
```

```python
import subprocess
import time

# Kill any existing LiteLLM processes
subprocess.run("pkill -f litellm || true", shell=True)
time.sleep(1)

# Start LiteLLM
cmd = f"nohup litellm --config litellm_config.yaml --port {LITELLM_PORT} --host 0.0.0.0 > litellm.log 2>&1 &"
subprocess.run(cmd, shell=True)
time.sleep(5)

print(f"✅ LiteLLM running on port: {LITELLM_PORT}")
print(f"URL: http://localhost:{LITELLM_PORT}/v1")

# Save port for other cells
with open('.litellm_port', 'w') as f:
    f.write(str(LITELLM_PORT))
```

Output:
```
✅ LiteLLM running on port: 4000
URL: http://localhost:4000/v1
```

## Install OpenAI Python Client

Now install the OpenAI client to test our setup:

```bash
pip install openai
```

## Test the Pipeline

Let's test our complete LiteLLM + Vast.ai setup with a basic API call:

```python
from openai import OpenAI

client = OpenAI(
    api_key="fake-key",
    base_url=f"http://localhost:{LITELLM_PORT}/v1"
)

# Basic test
try:
    response = client.chat.completions.create(
        model="deepseek",
        messages=[
            {"role": "user", "content": "Hello! How are you?"}
        ],
        max_tokens=1000
    )
    print("Response:", response.choices[0].message.content)
    
except Exception as e:
    print(f"❌ Error: {e}")
```

Output:
```
Response: 
Hello! I'm just a friendly AI here, so I don't get tired or have feelings—but I'm fully ready and happy to help you! How are you doing today? Let me know if there's anything you'd like to chat about or need assistance with.
```

Perfect! Now let's test the reasoning capabilities of the DeepSeek model:

```python
try:
    response_1 = client.chat.completions.create(
        model="deepseek",
        messages=[
            {"role": "user", "content": "what is 8 * 7?, Give me your answer in step by step reasoning."}
        ],
        max_tokens=3000
    )
    print("Response:", response_1.choices[0].message.content)
    
except Exception as e:
    print(f"❌ Error: {e}")
```

Output:
```
Response: 
### Step-by-Step Reasoning for 8 * 7

Multiplication is a basic arithmetic operation that can be thought of as repeated addition. To find the product of 8 and 7, you can add the number 8 to itself 7 times (or equivalently, add the number 7 to itself 8 times). I'll use the repeated addition method with the number 8 to make it clear.

Start with the first 8:
- Add the second 8: 8 + 8 = 16
- Add the third 8: 16 + 8 = 24
- Add the fourth 8: 24 + 8 = 32
- Add the fifth 8: 32 + 8 = 40
- Add the sixth 8: 40 + 8 = 48
- Add the seventh 8: 48 + 8 = 56

As shown, after adding 8 seven times, the result is 56.

Alternatively, you can verify this with another method, such as multiplying 7 eight times or subtracting 8 from 8 * 8 (since 8 * 7 = (8 * 8) - 8 = 64 - 8 = 56). Both methods confirm the result.

**Final Answer:** 56
```

The pipeline is working correctly. You now have a complete LiteLLM + Vast.ai setup with local proxy control and remote GPU inference.

**Note**: For production deployments, consider using Docker containers for LiteLLM rather than the command-line approach shown here.

## Conclusion

This hybrid inference architecture demonstrates how modern AI deployment can balance cost, control, and performance. By combining LiteLLM's local proxy capabilities with Vast.ai's GPU marketplace, we've created a system that offers several key advantages:

**Cost Considerations**: Vast.ai's marketplace model and pay-as-you-go pricing can provide cost savings compared to traditional cloud providers, while running LiteLLM locally eliminates additional hosting costs for the proxy layer.

**Local Control**: Running LiteLLM locally provides control over request routing, logging, and configuration. The proxy server includes built-in features for cost tracking, rate limiting, and observability integrations.

**API Compatibility**: LiteLLM's OpenAI-compatible interface allows existing applications to integrate with minimal changes, while supporting multiple LLM providers through a unified API.

**Flexible Architecture**: This setup allows you to experiment with different models and providers while maintaining a consistent interface for your applications.

Whether you're building AI-powered applications, conducting research, or optimizing existing ML pipelines, this hybrid approach provides a practical path to cost-effective, controllable AI inference that doesn't sacrifice capability for economics.

Deploy hybrid AI inference using LiteLLM + Vast.ai. Run local OpenAI-compatible proxies with remote GPU models for scalable, cost-effective LLM performance.

Hybrid AI Inference: Local LiteLLM Proxy with Remote Vast.ai GPU

Deploy Llama 4, Qwen3, and Kimi K2 on Vast.ai in minutes. Use optimized templates like Oobabooga, vLLM, HuggingFace TGI, and Open WebUI—no setup hassle.

Modular MAX vs vLLM Performance Comparison on Vast.ai


Today, we’re announcing the launch of our Vast.ai Vulnerability Bounty Program. This is a new initiative we’ve created to help us improve, expand, and innovate our platform while making it more secure. Vast.ai’s mission is to provide the best GPUs for AI compute at accessible and affordable prices for everyone who needs it. With this bounty program, we’re inviting AI developers, researchers, and enthusiasts to collaborate with us. The launch of this program is in direct response to our community’s feedback, and we appreciate all the collaboration happening already.

### **Responsible Vulnerability Disclosure**

### We take security seriously and encourage responsible disclosure of any [vulnerabilities](https://vast.ai/vulnerability-disclosure-policy).

**What is the [Vast.ai](http://Vast.ai) Vulnerability Bounty program?**

The program provides a structured way to report bugs, suggest new features, or responsibly uncover vulnerabilities. Whether you’re a security researcher or a community builder, there’s an opportunity for you to do the right thing and be rewarded.

### **What Is In Scope?**

-   Vast.ai web console, REST API & billing flows
-   Provider Daemon code (host agent)
-   Match‑making & pricing engine
-   Default Docker & KVM isolation on reference images
-   GPU memory‑isolation / tenant breakout flaws

What is out of Scope?

-   User workloads & third‑party container images
-   Social‑engineering, physical security, or denial‑of‑service (DoS) tests
-   Brute‑force attacks against customer passwords or MFA
-   Any activity that violates applicable law, exports regulations, or provider Terms of Service

### If you discover a security issue, we ask that you:

-   Report it directly to our team at [security@vast.ai](mailto:security@vast.ai?subject=Vulnerability%20Bounty%20Program%20Inquiry).

-   Allow us a reasonable time to address the issue before any public disclosure.

### In return, we commit to:

-   Acknowledging your report promptly and keeping you informed of progress.

-   Recognizing your efforts with a form of bounty rewards, platform credit, or joining our hall of fame.

### **How to Get Started?**

1. Join the [Vast.ai Discord](https://discord.gg/NCgfu5BrdQ) to connect with our team and community.

2. Submit your bounty contributions and start hunting.

Please review [the policy](https://vast.ai/vulnerability-disclosure-policy) for the Rules of Engagement and specifics on scope, which will be updated periodically.


Help secure Vast.ai and earn rewards. Join our Vulnerability Bounty Program to report bugs, suggest features, and collaborate with the AI infrastructure community.

Announcing the Vast.ai Vulnerability Bounty Program


In Part 1 of this series, we created a quantized version of the Qwen3-8B model (Qwen3-8B-W8A8) using LLM-Compressor with 8-bit weight and activation quantization. In this notebook, we'll demonstrate how to deploy and compare this quantized model against its full precision counterpart (Qwen3-8B) using Vast.ai. We'll examine both deployment processes and compare the outputs to assess any quality differences between the models while highlighting the efficiency gains from quantization.

## Key benefits of quantized models:

- Reduced memory footprint (approximately 1.7x smaller in our case)
- Lower inference latency
- Decreased computational requirements
- More affordable deployment options

Let's start by exploring how to deploy our quantized model on Vast.ai.

## Deploying our Model on Vast.ai

First, we will install the Vast.ai SDK and input our API key. This allows us to interact with the Vast.ai platform programmatically and manage our GPU instances.

```bash
#In an environment of your choice
pip install --upgrade vastai
```

```bash
# Here we will set our api key
export VAST_API_KEY="VAST_API_KEY" #Your key here
vastai set api-key $VAST_API_KEY
```

Next, we'll search for an appropriate GPU instance to serve our quantized model. Since this is a W8A8 quantized version of Qwen3-8B, we can use a machine with less VRAM than would be required for the full model. We're looking for machines with at least 24GB of VRAM, which is sufficient for our 8-bit quantized model and its context window:

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 24 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 20 \
rentable = true"
```

Once we've identified a suitable instance from the search results, we'll deploy our quantized model to that instance. We're using the vLLM Docker image with an OpenAI-compatible API, which makes it easy to serve and interact with our model.

Note that we're deploying the W8A8 quantized version of Qwen3-8B that was created and uploaded to Hugging Face in Part 1. Please replace `your-hf-username` with your actual Hugging Face username to use your compressed model:

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 20 --args --model your-hf-username/Qwen3-8B-W8A8
```

## Calling Our Model Using the OpenAI SDK

Now that our model is deployed and running on Vast.ai, we'll set up a client to interact with it. The vLLM server exposes an OpenAI-compatible API, allowing us to use the OpenAI SDK to send requests to our model.

First, we need to install the OpenAI SDK if you haven't already:

```bash
pip install openai
```

Then, we use the OpenAI SDK and the `VAST_IP_ADDRESS` and `VAST_PORT` from our instance to call our model.

Also add your hugginface username `HF_USERNAME` to call the model we saved to hugginface.

```python
from openai import OpenAI

VAST_IP_ADDRESS = "VAST_IP_ADDRESS"
VAST_PORT = "VAST_PORT"

HF_USERNAME="HF_USERNAME"

# Create a client instance pointing to the vLLM server
client = OpenAI(
    api_key="dummy-key",  # vLLM doesn't require a real API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"  # Point to your vLLM server
)

# Call the model
response = client.chat.completions.create(
    model=f"{HF_USERNAME}/Qwen3-8B-W8A8",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What are the major applications of deep learning?"}
    ],
    temperature=0.7,
    max_tokens=1000
)

# Print the response
print(response.choices[0].message.content)
```

Output:

```
<think>
Okay, the user is asking about the major applications of deep learning. I need to provide a comprehensive yet clear answer. Let me start by recalling the different areas where deep learning is commonly used.

First, computer vision comes to mind. That includes image recognition, object detection, and maybe even image generation. Then there's natural language processing, like chatbots, translation, and sentiment analysis.

I should also mention speech recognition, which is used in virtual assistants. Autonomous vehicles rely on deep learning for perception tasks. Healthcare is another big area, such as medical imaging analysis and drug discovery.

Finance uses deep learning for fraud detection and algorithmic trading. Recommender systems in e-commerce and streaming services are also important. Maybe touch on generative models like GANs and diffusion models for creating art or text.

Wait, I should categorize these applications to make it organized. Let me list them under different headings. Also, I need to ensure I don't miss any major areas. Oh, maybe mention robotics and control systems too.

I should check if there are any recent advancements or emerging fields. For example, deep learning in gaming, like reinforcement learning for game AI. Also, maybe something about anomaly detection in various industries.

Need to keep the explanation straightforward without too much jargon. Make sure each application is explained briefly but clearly. Avoid going into too much technical detail unless necessary.

Let me structure the answer with clear sections for each major application. Start with an introduction, then list each area with a brief description. Conclude by summarizing the versatility of deep learning.

Wait, the user might be looking for both current and potential future applications. Should I include that? Maybe a short note at the end about ongoing research and future possibilities.

Also, check for any overlaps or if some applications are better categorized under specific fields. For example, NLP and speech recognition are both related to language processing.

I think that's a solid structure. Now, I'll draft the answer accordingly, making sure it's informative and covers all key areas without being too verbose.
</think>

Deep learning, a subset of machine learning, has revolutionized numerous fields due to its ability to model complex patterns in data. Here are its major applications across various domains:

---

### **1. Computer Vision**
- **Image Recognition**: Identifying objects, scenes, and patterns in images (e.g., facial recognition, medical imaging analysis).
- **Object Detection**: Locating and classifying multiple objects in a single image (e.g., autonomous vehicles, surveillance systems).
- **Image Generation**: Creating new images using Generative Adversarial Networks (GANs) or diffusion models (e.g., AI art tools).
- **Video Analysis**: Action recognition, video summarization, and scene understanding.

---

### **2. Natural Language Processing (NLP)**
- **Text Generation**: Writing articles, stories, or code using models like GPT or BERT.
- **Translation**: Real-time language translation (e.g., Google Translate).
- **Sentiment Analysis**: Determining emotions in text (e.g., social media monitoring).
- **Chatbots & Virtual Assistants**: Interactive AI assistants (e.g., Siri, Alexa, customer service bots).

---

### **3. Speech Recognition & Synthesis**
- **Voice Assistants**: Converting speech to text (e.g., Alexa, Google Assistant).
- **Speech-to-Text**: Transcribing audio for accessibility or transcription services.
- **Text-to-Speech**: Synthesizing natural-sounding speech (e.g., audiobooks, virtual characters).

---

### **4. Autonomous Systems**
- **Self-Driving Cars**: Perceiving environments, navigating, and making real-time decisions.
- **Robotics**: Controlling robotic arms, navigation, and object manipulation.
- **Industrial Automation**: Predictive maintenance and quality control in manufacturing.

---

### **5. Healthcare**
- **Medical Imaging**: Diagnosing diseases (e.g., tumors in X-rays, MRIs) using CNNs.
- **Drug Discovery**: Accelerating the development of new medications through molecular modeling.
- **Personalized Medicine**: Predicting patient outcomes based on genetic data and medical history.

---

### **6. Finance**
- **Fraud Detection**: Identifying unusual transaction patterns.
- **Algorithmic Trading**: Predicting market trends and executing trades.
- **Risk Management**: Assessing credit risk and financial stability.

---

### **7. Recommender Systems**
- **E-commerce**: Personalized product recommendations (e.g., Amazon, Netflix).
- **Streaming Services**: Content suggestions based on user preferences.
- **Social Media**: Curating news feeds and trending topics.

---

### **8. Gaming & Entertainment**
- **Game AI**: Creating intelligent opponents or NPCs (e.g., AlphaGo, Dota 2 bots).
- **Content Creation**: Generating music, art, or video using deep learning models.

---
```

## Compare with full `Qwen/Qwen3-8B` Model

Now that we've seen the output from our quantized model, let's deploy and test the full precision version of the same model for comparison. This will help us evaluate any potential quality differences and understand the tradeoffs of quantization.

First, we need to find a more powerful machine with significantly more GPU memory. The full precision model requires approximately 3-4x more VRAM than its quantized counterpart. Let's search for instances with at least 80GB of VRAM:

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 80 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 40 \
rentable = true"
```

Now we'll deploy the full precision model. Notice the differences in our configuration compared to the quantized model:

1. We're requiring more disk space (40GB vs 20GB)
2. We're using a much more powerful GPU with significantly more VRAM
3. We're deploying the original model from Qwen directly (Qwen/Qwen3-8B)

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 40 --args --model  Qwen/Qwen3-8B
```

With the full precision model deployed, let's query it with the same prompt we used for the quantized model to perform a direct comparison of their outputs:

```python

VAST_IP_ADDRESS = "VAST_IP_ADDRESS"
VAST_PORT = "VAST_PORT"

# Create a client instance pointing to the vLLM server
client = OpenAI(
    api_key="dummy-key",  # vLLM doesn't require a real API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"  # Point to your vLLM server
)

# Call the model
response = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What are the major applications of deep learning?"}
    ],
    temperature=0.7,
    max_tokens=1000
)

# Print the response
print(response.choices[0].message.content)
```

```
<think>
Okay, the user is asking about the major applications of deep learning. Let me start by recalling what I know. Deep learning is a subset of machine learning that uses neural networks with many layers. It's been really popular in various fields. I need to list the main areas where it's applied.

First, computer vision comes to mind. Image recognition, object detection, maybe even things like facial recognition. Then there's natural language processing, like chatbots, translation services, text generation. Speech recognition is another big one, like virtual assistants.

What else? Autonomous vehicles use deep learning for perception tasks, like identifying objects on the road. Healthcare is another area, maybe medical imaging analysis, predicting diseases. Then there's recommendation systems used by companies like Netflix or Amazon.

Wait, what about robotics? Deep reinforcement learning is used in robotics for control systems. Also, maybe gaming, like in AlphaGo. Financial services for fraud detection or algorithmic trading.

I should also mention some more specialized areas like generative models for creating images or music, and maybe something about NLP like sentiment analysis. Oh, and maybe mention that these applications are driven by the ability of deep learning to handle large amounts of data and complex patterns.

Wait, the user might be looking for a comprehensive list, so I should structure it into categories. Let me check if I missed any major areas. Maybe cybersecurity? Or maybe something like anomaly detection in various industries. Also, maybe mention the difference between deep learning and traditional machine learning in terms of application.

I need to make sure each application is clearly explained with examples. Let me organize them into sections. Computer Vision, NLP, Speech Recognition, Autonomous Vehicles, Healthcare, Recommendation Systems, Robotics, Gaming, Financial Services, Cybersecurity, Generative Models, and maybe others.

I should also note that deep learning is used in both supervised and unsupervised tasks. Oh, and maybe mention some specific models like CNNs for images, RNNs or Transformers for NLP. But the user might not need the technical details, just the applications.

Wait, the user might be a student or someone new to the field, so keeping it clear and concise is important. Avoid jargon where possible. Also, make sure to highlight the impact of each application.

Let me make sure I cover all major areas. Maybe add something about drug discovery in healthcare. Or maybe mention that deep learning is used in weather prediction or climate modeling.

Hmm, I think that's a good list. Let me structure it with bullet points and explanations for each application. Also, maybe conclude with a note on the versatility and impact of deep learning.
</think>

Deep learning, a subset of machine learning, has revolutionized numerous fields due to its ability to model complex patterns and process large datasets. Here are the **major applications** of deep learning:

---

### **1. Computer Vision**
- **Image Recognition**: Identifying objects, people, and scenes in images (e.g., Google Photos, medical imaging).
- **Object Detection**: Locating and classifying objects in real-time (e.g., autonomous vehicles, security cameras).
- **Facial Recognition**: Used in biometric authentication (e.g., smartphones, border control).
- **Image Generation**: Creating realistic images (e.g., GANs for art, style transfer).

---

### **2. Natural Language Processing (NLP)**
- **Text Generation**: Writing articles, stories, or code (e.g., chatbots, AI assistants like Siri).
- **Machine Translation**: Languages like Google Translate use neural networks for real-time translation.
- **Sentiment Analysis**: Analyzing emotions in text (e.g., social media monitoring, customer feedback).
- **Speech Recognition**: Converting speech to text (e.g., voice assistants, transcription services).

---

### **3. Autonomous Systems**
- **Self-Driving Cars**: Detecting pedestrians, traffic signs, and obstacles (e.g., Tesla Autopilot, Waymo).
- **Robotics**: Controlling robotic arms and navigation systems (e.g., industrial robots, drones).
- **Reinforcement Learning**: Training agents for tasks like game playing (e.g., AlphaGo, Dota 2).

---

### **4. Healthcare**
- **Medical Imaging**: Diagnosing diseases from X-rays, MRIs, or CT scans (e.g., cancer detection).
- **Drug Discovery**: Accelerating the development of new drugs by predicting molecular interactions.
- **Personalized Medicine**: Tailoring treatments based on patient data (e.g., genomics analysis).

---

### **5. Recommendation Systems**
- **E-commerce**: Product recommendations (e.g., Amazon, Netflix).
- **Streaming Services**: Content suggestions (e.g., Spotify, YouTube).
- **Social Media**: Curating news feeds (e.g., Facebook, Instagram).

---

### **6. Finance**
- **Fraud Detection**: Identifying unusual
```

## Conclusion: Comparing Model Outputs and Size Tradeoffs

After deploying and testing both the quantized model (W8A8) and the full precision model, we can make several key observations:

### Model Size and Resource Requirements

- **Full Precision Model (Qwen3-8B)**:

  - Required ~80GB+ VRAM GPU
  - Needed 40GB disk space
  - Model size: ~16GB (FP16)

- **Quantized Model (Qwen3-8B-W8A8)**:
  - Required only 24GB VRAM GPU (~3.3x less memory)
  - Needed only 20GB disk space
  - Model size: ~9.5GB (8-bit weights and activations)

### Output Quality Comparison

Looking at the outputs from both models, we can observe:

1. **Structural Similarity**: Both models organized their responses with similar categories (Computer Vision, NLP, Healthcare, etc.) and maintained good formatting with markdown.

2. **Content Quality**: Both models produced high-quality, relevant content with appropriate examples for each application domain. The specific details varied slightly, but the core information was consistent.

3. **Reasoning Process**: Both models showed similar thinking processes in their `<think>` sections, demonstrating that the quantized model retained the structured reasoning capabilities of the original model.

4. **Minimal Quality Degradation**: The quantized model showed negligible degradation in output quality compared to the full precision model, despite being significantly smaller.

### Key Takeaways

1. **Deployment Accessibility**: The W8A8 quantized model makes deployment much more accessible, requiring far less expensive hardware while maintaining output quality.

2. **Cost Efficiency**: Using the quantized model can significantly reduce cloud GPU costs by allowing deployment on less expensive hardware.

3. **Production Readiness**: 8-bit quantization appears to be production-ready for this type of model, offering an excellent balance between efficiency and performance.

4. **Use Case Fit**: For most general text generation and understanding tasks, the quantized model appears to be a suitable replacement for the full precision model.

This notebook demonstrates that weight and activation quantization (W8A8) using LLM Compressor is an effective approach for deploying large language models in resource-constrained environments without significant quality compromises.

Vast.ai provides an excellent platform for this workflow, allowing us to use high-end GPUs for model compression and more cost-effective GPUs for efficient inference.


Compare quantized vs. full Qwen3-8B models deployed on Vast.ai. See quality vs. efficiency tradeoffs using W8A8 compression with LLM-Compressor and vLLM.

Using LLM-Compressor to Quantize Qwen3-8B on Vast.ai (Part 2)


As AI models grow increasingly powerful, they're also becoming increasingly expensive to deploy. The latest language models require massive amounts of GPU memory and computational resources, putting them out of reach for many teams and use cases. Model compression has emerged as a critical technique for making these models more accessible while maintaining their performance.

Today, we'll explore how to compress large language models using LLM-Compressor and deploy them cost-effectively on Vast.ai. We'll walk through the complete process of taking a 16GB model and reducing it to approximately 9.5GB while preserving quality—making deployment significantly more affordable and accessible.

## The Tools: LLM-Compressor and Vast.ai

### LLM-Compressor: Making Models Leaner

The vLLM team has developed [LLM-Compressor](https://github.com/vllm-project/llm-compressor) as a comprehensive solution for reducing model size without sacrificing performance. Unlike other compression tools that focus on a single technique, LLM-Compressor offers a complete toolkit for optimization.

What makes it particularly powerful:

- **Comprehensive approach**: Combines quantization, pruning, and distillation techniques that can be mixed and matched
- **Production-ready**: Built specifically for deployment scenarios with seamless vLLM integration
- **Developer-friendly**: The one-shot API handles the complex compression pipeline automatically
- **Proven techniques**: Implements industry-standard methods like GPTQ, AWQ, and SmoothQuant

### Vast.ai: Affordable GPU Power

Traditional cloud providers can make GPU-intensive tasks prohibitively expensive. Vast.ai changes this equation by creating a marketplace where you can access high-end GPUs at competitive rates—often significantly cheaper than major cloud platforms.

For model compression workflows, Vast.ai offers several advantages:

- **Flexible pricing**: Pay-per-hour model means you only pay for compression time, not idle resources
- **Hardware variety**: Choose from different GPU types based on your specific memory and compute requirements
- **Quick deployment**: Get instances running in minutes without complex setup procedures
- **Community-driven**: Access to a wider range of hardware options through the marketplace model

## In this Tutorial

This tutorial demonstrates how to:

1. Compress the `Qwen3-8B` model using LLM-Compressor with 8-bit weight and activation quantization (W8A8)
2. Upload the compressed `Qwen3-8B-W8A8` model to Hugging Face for easy sharing and deployment

In the second part of this series, we'll show how to deploy the compressed `Qwen3-8B-W8A8` model on Vast.ai for efficient inference and compare its performance with the full-precision model.

## GPU Requirements

Model compression requires significant computational resources, especially for larger models. For compressing the Qwen3-8B model, we recommend the following specifications:

### Minimum Requirements:

- **GPU Memory**: 30GB+ VRAM (e.g., A5000, A6000, RTX 4090, V100-32GB)
- **GPU Compute**: CUDA-compatible GPU with compute capability 7.0+
- **System RAM**: 64GB recommended (minimum 32GB)
- **Storage**: 80GB free disk space for model files and datasets
- **CUDA Version**: 11.8 or newer

### Why these requirements?

- The full Qwen3-8B model requires approximately 16GB just to load in FP16 format
- During compression, we need memory for both the original model and compression overhead
- Calibration datasets and intermediate files require additional storage space
- Model compilers and quantization tools have their own memory overhead

For larger models (12B+), you would need even more VRAM (40GB+) or techniques like model sharding.

## Renting an Instance on Vast.ai

1. Ensure that you have a Vast.ai account. If not, sign up at [vast.ai](https://vast.ai)
2. Go to the Vast Templates in the Console https://cloud.vast.ai/templates/
3. Select the `PyTorch (CuDNN Runtime)` Template - this provides a pre-configured environment with PyTorch and CUDA
4. Filter for an instance with the following specifications for optimal performance:
   - 1 GPU with high VRAM (A5000, A6000, RTX 4090, or similar)
   - 30GB+ of VRAM (required for full model loading plus quantization overhead)
   - 80GB+ of storage (for model weights, checkpoints, and datasets)
   - 32GB+ of system RAM (64GB recommended for smoother operation)
5. Select an instance and click rent - hourly pricing will vary based on the selected hardware
6. Install the Vast TLS certificate in your browser to access the notebook server securely https://docs.vast.ai/instances/jupyter#1SmCz
7. Go to your Instances https://cloud.vast.ai/instances/ and click "Open" to access the Jupyter server on your instance
8. Upload this notebook to the server through the Jupyter interface

## Setup and Installation

First we will install and import our required packages. The cell below installs:

- Transformers: For loading and saving the model in Hugging Face format
- PyTorch: The deep learning framework that powers our model
- Hugging Face Hub: For uploading our compressed model
- LLM-Compressor: The toolkit we'll use for quantization

```bash
# Install required packages
# Using specific versions to ensure compatibility
pip install transformers==4.51.3  # Recent Transformers version with good Qwen3 support
pip install torch==2.7.0          # PyTorch version compatible with current CUDA drivers
pip install huggingface_hub       # Core Hugging Face Hub library
pip install huggingface_hub[hf_xet]  # Extension for handling large file transfers
pip install huggingface_hub[cli]     # Command-line interface for HF Hub
pip install llmcompressor            # The model compression toolkit
```

Next, import the necessary libraries:

```python
# Import necessary libraries
import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from llmcompressor.modifiers.quantization import GPTQModifier  # For weight quantization
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier  # For activation quantization preprocessing
from llmcompressor import oneshot  # Simplified API for end-to-end compression
```

## Define Model and Output Path

Next, we'll define our model variables and prepare for downloading the `Qwen/Qwen3-8B` model.

- `MODEL_ID`: The Hugging Face model ID for the original model
- `OUTPUT_MODEL_ID`: A name for our compressed model version
- `OUTPUT_DIR`: Local directory where the compressed model will be saved

```python
# Define model path and output directory
MODEL_ID = "Qwen/Qwen3-8B"  # Original model on Hugging Face
OUTPUT_MODEL_ID = "Qwen3-8B-W8A8"  # Our compressed model name
OUTPUT_DIR = "./Qwen3-8B-W8A8"  # Local directory for saving
os.makedirs(OUTPUT_DIR, exist_ok=True)  # Create output directory if it doesn't exist
```

Download the model files using the snapshot_download function:

```python
from huggingface_hub import snapshot_download

# Download the model files to local directory
# This is a more reliable way to download large models compared to the standard AutoModel approach
# This will download all model files (~16GB for Qwen3-8B) and may take some time depending on connection speed
snapshot_download(
    repo_id="Qwen/Qwen3-8B",
    local_dir_use_symlinks=False  # Set to True to use symlinks instead of copying files (saves space but may cause issues)
)
```

## Configure Quantization

Now we'll define our compression recipe, which consists of two steps:

1. **SmoothQuant**: A technique that redistributes the quantization difficulty between weights and activations, making activations easier to quantize with minimal accuracy loss

2. **GPTQ**: A weight quantization method specifically designed for large language models that uses a reconstruction approach to minimize the impact of quantization on model quality

We're using W8A8 quantization, which means:

- W8: Weights stored in 8-bit precision (instead of FP16/FP32)
- A8: Activations computed in 8-bit precision at inference time

This will reduce the model size by approximately 1.7x and potentially increase inference speed.

```python
# Create quantization recipe
# A recipe is a sequence of compression techniques applied in order
recipe = [
    # First apply SmoothQuant to make activations easier to quantize
    # The smoothing_strength parameter controls how aggressively to shift quantization difficulty
    # from activations to weights (higher = more shifting, 0.8 is a good balance)
    SmoothQuantModifier(smoothing_strength=0.8),

    # Then apply GPTQ for weight and activation quantization
    GPTQModifier(
        targets="Linear",  # Apply to all linear layers in the model
        scheme="W8A8",    # 8-bit weights and activations (other options: W4A16, W8A16, etc.)
        ignore=["lm_head"] # Don't quantize the language modeling head (important for output quality)
    )
]
```

## Apply Quantization with oneshot API

Now we'll apply our compression recipe to the model using the oneshot API.

The oneshot API handles the entire compression workflow:

1. Loading the model from Hugging Face or local path
2. Creating a calibration dataset (or using an existing one)
3. Running calibration to determine optimal quantization parameters
4. Applying the compression recipe
5. Saving the compressed model

We're using the `open_platypus` dataset for calibration, which consists of diverse open-source technical questions and answers. This helps ensure the model maintains accuracy across a range of technical topics.

**Note**: This cell will take a significant amount of time (30-60 minutes) and GPU memory to run.

```python
print(f"Starting compression of {MODEL_ID}...")

# Apply quantization using oneshot API
oneshot(
    model=MODEL_ID,               # HF model ID or local path to load the model from
    dataset="open_platypus",      # Calibration dataset - preset dataset with technical Q&A
    recipe=recipe,                # Our compression recipe defined above
    output_dir=OUTPUT_DIR,        # Where to save the compressed model
    max_seq_length=2048,          # Maximum sequence length for calibration samples
    num_calibration_samples=512,  # Number of samples to use for calibration (more is better but slower)
)
```

## Test the Compressed Model

After compression, it's important to verify that the model still performs reasonably well. This cell loads the compressed model and runs a simple inference test.

We'll evaluate the model on a prompt about AI technology to see how well it generates coherent and relevant text. This helps us verify that the quantization didn't significantly degrade model quality.

Note that a proper evaluation would involve more systematic testing across multiple prompts and metrics.

```python
# Load the compressed model from our output directory
tokenizer = AutoTokenizer.from_pretrained(OUTPUT_DIR)
model = AutoModelForCausalLM.from_pretrained(
    OUTPUT_DIR,
    device_map="auto",      # Automatically place model on available devices (CPU/GPU)
    torch_dtype="auto"      # Use the model's native precision (will be INT8 for quantized weights)
)

# Simple test to verify the model still works
prompt = "AI technology is transforming industries by"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate text with some randomness for creativity
with torch.no_grad():  # Disable gradient calculation for inference
    outputs = model.generate(
        inputs["input_ids"],
        max_new_tokens=100,     # Generate up to 100 new tokens
        do_sample=True,         # Use sampling instead of greedy decoding
        temperature=0.7         # Control randomness (lower = more deterministic)
    )

# Decode the generated text and print it
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Prompt: {prompt}")
print(f"Generated text: {generated_text}")
```

## Upload the Model to Hugging Face

Now that we have successfully compressed the model, we'll upload it to Hugging Face Hub for easy sharing and deployment.

First, we'll set up the necessary credentials and repository information:

```python
from huggingface_hub import HfApi, login
import os

# Define variables for Hugging Face upload
# You'll need to replace these with your actual Hugging Face information
HF_USERNAME = "HF_USERNAME" # Add your Hugging Face username here
HF_TOKEN = "HF_TOKEN"       # Replace with your actual Hugging Face token (from settings page)
HF_MODEL_ID = f"{HF_USERNAME}/{OUTPUT_MODEL_ID}"  # Full repository path on Hugging Face
```

Next, we'll create a proper model card (README.md) with information about our compressed model. This helps others understand what the model is, how it was compressed, and how to use it:

```python
# Create a detailed model card describing the compressed model
model_card_content = f"""---
language:
- en
- zh
library_name: transformers
license: other
datasets:
- open_platypus
---

# Qwen3-8B-W8A8

This is a compressed version of [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) using llm-compressor with the following optimizations:
- 8-bit weight quantization using GPTQ
- 8-bit activation quantization
- SmoothQuant pre-processing

## Model Details

- **Original Model**: Qwen/Qwen3-8B
- **Quantization Method**: GPTQ + SmoothQuant (W8A8)
- **Compression Libraries**: [llm-compressor](https://github.com/vllm-project/llm-compressor)
- **Calibration Dataset**: open_platypus (512 samples)
- **Optimized For**: Inference with vLLM
"""

# Write README.md to model directory
readme_path = os.path.join(OUTPUT_DIR, "README.md")
with open(readme_path, "w") as f:
    f.write(model_card_content)
```

Finally, we'll upload the compressed model to Hugging Face Hub. This process may take some time depending on your internet connection speed, as the model files are large (approximately 9.5GB even after compression):

```python
# Login to Hugging Face with your token
login(token=HF_TOKEN)

# Initialize the Hugging Face API
api = HfApi()

# Create the repository (if it doesn't exist)
api.create_repo(
    repo_id=HF_MODEL_ID,
    exist_ok=True,        # Don't error if repo already exists
    private=False,        # Set to True if you want a private repository
)

# Upload the model files
print(f"Starting upload of {OUTPUT_DIR} to {HF_MODEL_ID}...")
api.upload_folder(
    folder_path=OUTPUT_DIR,
    repo_id=HF_MODEL_ID,
    repo_type="model",
    ignore_patterns=["*.tmp", "*.log", "__pycache__"],  # Files to ignore during upload
    commit_message="Upload compressed Qwen3-8B with W8A8 quantization",
)

print(f"Model successfully uploaded to: https://huggingface.co/{HF_MODEL_ID}")
```

## Wrapping Up: From 16GB to 9.5GB Without Breaking the Bank

We've just walked through a transformation that would have seemed impossible a few years ago—taking a sophisticated language model and cutting its resource requirements by approximately 40% while preserving its capabilities.

### What We Accomplished

Our compression workflow delivered results across multiple dimensions:

**Technical achievements:**

- Reduced model footprint from 16GB to 9.5GB using W8A8 quantization
- Maintained output quality through careful calibration with technical datasets
- Created a production-ready model compatible with modern inference engines

**Economic impact:**

- Enabled deployment on more affordable GPUs with less VRAM requirements
- Reduced cloud inference costs through lower resource requirements
- Made advanced language models accessible to teams with limited budgets

**Operational benefits:**

- Faster model loading and reduced memory pressure during inference
- Better resource utilization allowing for higher throughput or multi-model serving
- Simplified deployment pipeline through automated compression workflows

### The Bigger Picture

Model compression offers an alternative approach to AI deployment. Rather than accepting ever-increasing resource requirements, we can actively optimize models for real-world constraints. The combination of LLM-Compressor's sophisticated algorithms and Vast.ai's accessible infrastructure makes this optimization practical for any team.

### Coming Up Next

In Part 2, we'll put our compressed model to the test. We'll deploy both our optimized Qwen3-8B-W8A8 model and the original full-precision version on Vast.ai, then conduct detailed comparisons of their performance, cost, and output quality. You'll see exactly how much you can save without compromising on results—and learn when compression makes the most sense for your specific use case.


Compress LLMs like Qwen3-8B with LLM-Compressor and deploy on Vast.ai. Cut model size ~40% while preserving quality—save cost, speed up inference, and scale smartly.

Model Compression with LLM-Compressor and Deployment on Vast.ai (Part 1)


You wouldn't buy a car without taking it for a test drive, so why deploy a large language model (LLM) without benchmarking it first?

When you're running LLMs at scale, you need more than just theoretical performance data. You're going to want to know how they actually perform under real conditions – throughput, latency, and hardware efficiency all matter. Skipping this step can result in higher costs and wasted resources.

Benchmarking early helps you avoid inefficiencies down the line, and it shouldn't take hours to get started. That's where **vLLM** comes in.

### A Smarter Way to Serve and Benchmark LLMs

vLLM is an open-source library that optimizes the inference and serving of LLMs, and it [simplifies](https://blog.vllm.ai/2023/06/20/vllm.html) the benchmarking process with an efficient architecture built around the attention algorithm PagedAttention. This mechanism reduces memory waste and improves throughput by storing attention data non-contiguously.

When fast and efficient LLM inference is crucial, understanding your model's performance starts with benchmarking. With vLLM, you can spin up a high-performance inference server and start collecting those benchmarks in minutes.

To make things even easier, you don't need to spin up your own infrastructure from scratch. Vast.ai provides an ideal platform to leverage vLLM, offering high-performance GPUs at a fraction of the cost of traditional cloud providers. With Vast, it's easy to find the right hardware for your model, and with simple Docker integration, you can get up and running with vLLM quickly.

This guide demonstrates how to benchmark any LLM using vLLM on Vast.ai, in just a few minutes.

### Getting Started: Benchmarking LLMs with vLLM

This guide walks you through how to **serve a large language model using vLLM** and run a benchmark test to evaluate performance. It’s using a **H100** **GPU server from [Vast.ai](https://vast.ai/)** with the **"PyTorch" template**. In this example, we’ll use the model `meta-llama/Llama-3.1-8B-Instruct`, but you can use **any Hugging Face-supported model** compatible with vLLM.

**1\. Install required Python packages**

First, install the **vLLM** library along with **Pandas** and Hugging Face’s **datasets** library. For example, you can use pip in your Python environment:

```shell
pip install vllm pandas datasets
```

This installs vLLM and the other libraries for data handling.

**2\. Log in to Hugging Face**

To access the Meta Llama 3.1 model, you need a Hugging Face API token. Then run:

```shell
huggingface-cli login
```

This will prompt you to paste your Hugging Face **access token** (from your Hugging Face account settings). After entering the token, the CLI saves it so vLLM and other tools can download the model.

## **3\. Start the vLLM server**

Now start the vLLM server hosting the Llama 3.1 8B Instruct model. vLLM’s OpenAI-compatible server listens on localhost port 8000 by default. Use the `vllm serve` command with **nohup** to run it in the background and redirect output to a log file. For example:

```shell
nohup vllm serve meta-llama/Llama-3.1-8B-Instruct > vllm.log 2>&1 &
```

This command launches the server with the specified model and writes all output (including any Uvicorn logs) to `vllm.log`. The server will run in the background. (The vLLM docs give a similar example: `vllm serve <model-name>`) By default, this serves the model on [`http://localhost:8000`](http://localhost:8000).

## **4\. Test the server with a curl request**

Once the server is running, you can test it by sending a chat completion request to the API. For example, use `curl` to call the chat endpoint (`/v1/chat/completions`) with a simple prompt:

```shell
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
          "model": "meta-llama/Llama-3.1-8B-Instruct",
          "messages": [
              {"role": "system", "content": "You are a helpful assistant."},
              {"role": "user", "content": "What is the capital of France?"}
          ]
        }'
```

This sends a JSON payload to the server. The `model` field names our loaded model, and `messages` contains a chat-style conversation. The server will respond with a JSON completion. (The official vLLM docs show a similar curl example for chat completions)

You should see a JSON response in the terminal with the model’s answer.

## **5\. Clone the vLLM repository**

To access the benchmark script, clone the vLLM source repository:

```shell
git clone https://github.com/vllm-project/vllm.git
cd vllm/benchmarks
```

This pulls the latest vLLM code. Now you have the `benchmark_serving.py` script and related files in the `benchmarks` folder.

## **6\. Run the benchmark script**

Finally, run the built-in benchmark script to measure serving performance. The `benchmark_serving.py` tool sends many requests to the running server and reports throughput/latency. For example, you might run:

```shell
  python3 vllm/benchmarks/benchmark_serving.py \
    --backend vllm \
    --base-url http://127.0.0.1:8000 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --tokenizer meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name random \
    --random-input-len 128 \
    --random-output-len 128 \
    --num-prompts 80 \
    --max-concurrency 4 \
    --temperature 0.7

```

-   `--backend vllm`: Use the vLLM backend to send benchmark requests.

-   `--base-url http://127.0.0.1:8000`: Address of your running vLLM server.

-   `--model meta-llama/Llama-3.1-8B-Instruct`: Name of the model being benchmarked.

-   `--tokenizer meta-llama/Llama-3.1-8B-Instruct`: Tokenizer used for encoding/decoding; usually the same as the model.

-   `--dataset-name random`: Use synthetic/randomly generated prompts instead of a real dataset.

-   `--random-input-len 128`: Length (in tokens) of each generated input prompt.

-   `--random-output-len 128`: Desired length (in tokens) of the model's output.

-   `--num-prompts 80`: Total number of prompts to send during the test.

-   `--max-concurrency 4`: Number of prompts sent in parallel (simulates batch size).

-   `--temperature 0.7`: Controls randomness in generation; higher \= more creative.

This example command (modeled on published vLLM benchmarks) will run 80 chat requests against your server with concurrency 4\. After it finishes, the script prints stats like throughput and latency.

By following these steps and citing vLLM’s documentation and examples, you can serve `Llama-3.1-8B-Instruct` with vLLM and evaluate its performance. Adjust parameters (batch size, number of tokens, etc.) in the commands above as needed for your environment.

### Conclusion

And that's it\! A fast, easy-to-use LLM benchmarking workflow powered by vLLM. In just ten minutes, you can gather meaningful performance data that will help you make smarter decisions about which models (and hardware) are right for your specific project and workload.

Need powerful GPUs to run your tests? Vast.ai offers on-demand access to H100s, RTX 5090s, and other high-performance options at a fraction of the typical price – so you can save 5-6X on GPU compute. Get started today and benchmark on your terms\!


Benchmark LLMs in minutes using vLLM on Vast.ai. Serve models like Llama 3.1-8B, test performance, and optimize cost with fast, efficient, real-world insights.

How to Benchmark An LLM with vLLM in 10 Minutes




Are you looking to run powerful open-source LLMs like **Llama 4**, **Kimi K2**, or **Qwen3** without the hassle of managing complex infrastructure? Vast.ai makes it easy to train and deploy these models with a curated set of templates built for speed, flexibility, and scale. 

Our [templates](https://docs.vast.ai/templates) cover everything from browser-based interfaces like Oobabooga and Open WebUI to optimized backends like HuggingFace TGI and vLLM. 

Below, we break down the top recommended templates on **Vast.ai** and how you can use them to get your LLM project off the ground quickly and easily. 

### 1\. [Oobabooga (LLM WebUI)](https://docs.vast.ai/oobabooga-llm-webui)

[Oobabooga](https://github.com/oobabooga/text-generation-webui) provides a user-friendly web interface for interacting with open-source LLMs. Its UI  resembles the original ChatGPT style – ideal for those who prefer a graphical interface over command-line operations. 

With support for **multiple text generation backends in one UI/API**, you can switch between different models easily without restarting and maintain fine control over settings. Models like **Falcon**, **Llama**, and **Vicuna** can be loaded and explored with just a few clicks.

Our [Oobabooga template](https://docs.vast.ai/oobabooga-llm-webui) facilitates quick deployment, making it suitable for both beginners and experienced users.

### 2\. [HuggingFace TGI with Llama 3](https://docs.vast.ai/huggingface-tgi-with-llama3)

Ideal for serving **Llama 3**, this template is optimized for **high-performance text generation tasks** using HuggingFace's [Text Generation Inference (TGI) server](https://github.com/huggingface/text-generation-inference). It supports other popular open-source LLMs from HuggingFace, including **Falcon**, **StarCoder**, **BLOOM**, and **GPT-NeoX**, and it's particularly beneficial for applications requiring low-latency responses and scalability, such as chatbots or content generation tools.

For this [template](https://docs.vast.ai/huggingface-tgi-with-llama3), you'll need your own HuggingFace access token – and you'll also need to apply for permission to use Llama 3 on HuggingFace, since the model is hosted in a gated repository.

### 3\. [Open WebUI (Ollama)](https://docs.vast.ai/ollama-webui)

[Open WebUI](https://github.com/open-webui/open-webui) is an extensible, user-friendly platform for self-hosted AI deployments. It operates entirely offline and supports a range of open-source LLM backends like **Llama 4**, **Kimi K2**, **DeepSeek R1**, and **OpenAI-compatible APIs**. 

With **built-in support for retrieval-augmented generation** (RAG), Open WebUI seamlessly integrates document interactions and web search into the chat experience. You can even incorporate image generation capabilities as well as engage with multiple LLMs simultaneously in parallel.

Our [Ollama \+ WebUI template](https://docs.vast.ai/ollama-webui) will automatically set up Open WebUI as a web-based interface and expose a port for the Ollama API, making it easy to run and interact with LLMs directly from your instance.

### 4\. [vLLM](https://docs.vast.ai/vllm-llm-inference-and-serving)

Optimized for [serving open-source LLMs](https://vast.ai/article/serving-online-inference-with-vllm-api-on-vast) with high-throughput inference, [vLLM](https://github.com/vllm-project/vllm) uses a novel architecture that drastically reduces memory overhead, making it easier to serve large models like **Llama**, **Mistral**, and **Falcon** efficiently. Notably, it supports continuous batching for faster and more efficient multi-user inference at scale.

vLLM provides an OpenAI-compatible API server for easy integration into existing workflows, and it's particularly well suited for developers building **commercial or research applications that require fast and stable model responses.** The vLLM framework seamlessly supports most open-source models on HuggingFace, including the ones named above as well as **Mixtral**, **DeepSeek**, **LLaVA**, **BLOOM**, **GPT-NeoX**, **Qwen**, and more.

Our [vLLM template](https://docs.vast.ai/vllm-llm-inference-and-serving) contains everything needed for you to get started – all you have to do is specify the model you want to serve and the corresponding vLLM configuration.

### From Setup to Serving in Minutes

With Vast.ai's market-based cloud GPU rental platform, you can avoid the usual infrastructure and budget roadblocks as you spin up high-performance instances tailored to your workloads. 

Our [library](https://cloud.vast.ai/templates) of **ready-to-use templates** – covering everything from web-based interfaces to optimized inference engines – make it easy to train and deploy LLMs at scale. We take care of the heavy lifting, so you can spend less time configuring and more time building.

Why wait? **Get started with open-source LLMs on Vast.ai today\!**

Run open-source LLMs like Llama 4, Qwen3, and Kimi K2 with ease. Vast.ai templates let you deploy fast using Oobabooga, vLLM, HuggingFace TGI, or Open WebUI.

Open-Source LLMs You Can Train and Deploy on Vast.ai Right Now


The DeepSeek R1-0528 release introduces a significant improvement to the reasoning capabilities of language models by making it easier to access "thinking" mode without requiring complex prompt engineering or pre-pending thinking tokens. This enhanced model provides transparent, step-by-step reasoning that's particularly valuable for educational applications, complex problem solving, and scenarios where transparency in AI decision-making is crucial.

In this post, we'll explore how to deploy the **DeepSeek-R1-0528-Qwen3-8B** model using **vLLM** on Vast.ai's cloud GPU platform, leveraging the new **qwen3 reasoning parser** that simplifies access to the model's internal thinking process.

---

## The Open Source AI Revolution

DeepSeek R1-0528 represents more than just a technical advancement—it's a significant step toward democratizing advanced AI capabilities. As an **open source model**, it makes these reasoning and advanced AI capabilities more accessible.

Unlike closed-source models that require vendor dependence, usage costs, and data privacy concerns, DeepSeek R1-0528 enables **complete control** over your AI infrastructure. Organizations can now deploy advanced reasoning capabilities locally, maintaining data sovereignty while avoiding variable per-token pricing.

This democratization means organizations can access cutting-edge AI without massive infrastructure investments. Advanced reasoning capabilities are no longer exclusive to large tech companies—anyone with GPU access can deploy their own reasoning-capable AI system.

---

## What Makes DeepSeek R1-0528 Special?

The DeepSeek R1-0528 release addresses a key limitation in previous DeepSeek models: the complexity of accessing the model's internal reasoning process. Previously, users needed to manually prepend "thinking" tokens to model outputs or use complex prompt engineering to see step-by-step reasoning.

---

## Deploying DeepSeek R1-0528 on Vast.ai

### Step 1: Set Up Vast.ai Environment

First, install the Vast CLI and configure your API key:

```bash
pip install --upgrade vastai

export VAST_API_KEY="your_vast_api_key"
vastai set api-key $VAST_API_KEY
```

### Step 2: Find Suitable Hardware

The DeepSeek-R1-0528-Qwen3-8B model requires specific hardware capabilities:

-   **Minimum 24GB GPU RAM** for model weights and KV cache
-   **Single GPU configuration** (sufficient for the 8B parameter model)
-   **Static IP address** for stable API endpoint hosting
-   **Direct port access** for the vLLM OpenAI-compatible server
-   **60GB+ disk space** for model storage and dependencies

Search for suitable instances:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 24 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 60 \
rentable = true"
```

### Step 3: Deploy the Model

Deploy using the vLLM OpenAI-compatible server with the qwen3 reasoning parser:

```bash
export INSTANCE_ID="your_instance_id"

vastai create instance $INSTANCE_ID \
--image vllm/vllm-openai:latest \
--env '-p 8000:8000' \
--disk 60 \
--args --model deepseek-ai/DeepSeek-R1-0528-Qwen3-8B \
--max-model-len 4096 \
--reasoning-parser qwen3
```

**Key deployment parameters:**

-   `--reasoning-parser qwen3`: Enables the simplified reasoning parser
-   `--max-model-len 4096`: Accommodates longer reasoning sequences
-   `--tensor-parallel-size 1`: Single GPU configuration
-   `--gpu-memory-utilization 0.90`: Optimized memory usage

### Step 4: Get Instance Details and Test Connection

Navigate to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console and locate your instance. Click the IP address button to view the forwarded ports.

Test the connection with a simple curl command:

```bash
export VAST_IP_ADDRESS="your_ip_address"
export VAST_PORT="your_port"

curl -X POST http://$VAST_IP_ADDRESS:$VAST_PORT/v1/completions \
-H "Content-Type: application/json" \
-d '{"model": "deepseek-ai/DeepSeek-R1-0528-Qwen3-8B", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

---

## Testing the Model's Reasoning Capabilities

### Setup Python Client

```python
import time
from openai import OpenAI

# Replace with your actual Vast.ai instance details
VAST_IP_ADDRESS = "your_ip_address"
VAST_PORT = "your_port"

client = OpenAI(
    api_key="DUMMY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)
```

### Basic Functionality Test

```python
start = time.time()
test_resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{"role": "user", "content": "Hello"}]
)
elapsed = time.time() - start

print("Response:", test_resp.choices[0].message.content)
print(f"Elapsed time: {elapsed:.4f} s")
```

### Reasoning Capability Test

This is where DeepSeek R1-0528 truly shines. The model naturally provides step-by-step reasoning without complex prompt engineering:

```python
reasoning_prompt = """
Write the History of Paris in a paragraph, show your thinking
step by step first before paragraph and then write the history
in a paragraph
"""

start = time.time()
resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{"role": "user", "content": reasoning_prompt}]
)
elapsed = time.time() - start

print("Model output:\n", resp.choices[0].message.content)
print(f"Elapsed time: {elapsed:.3f} s")
```

**Expected output structure:**

1. **Thinking process**: The model shows its step-by-step reasoning
2. **Final answer**: A well-structured paragraph about Paris's history

### Mathematical Reasoning Test

```python
math_prompt = "If 8 × 7 = ?, show your work step by step."

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{"role": "user", "content": math_prompt}]
)

print("Mathematical reasoning:\n", resp.choices[0].message.content)
```

The model will naturally break down the problem, show intermediate steps, and arrive at the solution with clear reasoning traces.

---

## Key Benefits of DeepSeek R1-0528

### 1. **Simplified Reasoning Access**

No need to prepend thinking tokens or use complex prompt engineering. The `qwen3` reasoning parser automatically handles the extraction and presentation of reasoning steps.

### 2. **Educational Value**

Perfect for educational applications where students need to understand the problem-solving process, not just the final answer.

### 3. **Transparency and Trust**

Users can see exactly how the model arrives at its conclusions, building trust in AI-powered decision-making systems.

### 4. **Cost-Effective Deployment**

Running on Vast.ai provides significant cost savings compared to major cloud providers while maintaining high performance.

### 5. **Easy Integration**

OpenAI-compatible API ensures seamless integration with existing applications and workflows.

---

## Use Cases

### Educational Applications

-   Step-by-step problem solving for mathematics, science, and logic
-   Teaching critical thinking and reasoning skills
-   Creating interactive learning experiences

### Business Intelligence

-   Transparent decision-making in automated systems
-   Audit trails for AI-powered recommendations
-   Complex analysis with clear reasoning paths

### Research and Development

-   Understanding model behavior and capabilities
-   Testing reasoning patterns across different domains
-   Developing more interpretable AI systems

---

## Conclusion

The DeepSeek R1-0528 release represents a significant step forward in making AI reasoning more accessible and transparent. By combining this powerful model with Vast.ai's cost-effective GPU infrastructure and the simplified qwen3 reasoning parser, developers can now easily deploy transparent, reasoning-capable AI systems.

Whether you're building educational tools, business intelligence systems, or research applications, DeepSeek R1-0528 on Vast.ai provides an excellent foundation for applications that require clear, step-by-step reasoning capabilities.

The elimination of complex prompt engineering for accessing reasoning modes makes this model particularly valuable for production applications where transparency and interpretability are key requirements.

---

## Useful Resources

-   [DeepSeek R1-0528 Model on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B)
-   [vLLM Documentation](https://docs.vllm.ai/)
-   [Vast.ai GPU Marketplace](https://vast.ai)
-   [OpenAI Python SDK](https://github.com/openai/openai-python)

---

_Ready to deploy transparent, reasoning-capable AI? Get started with DeepSeek R1-0528 on Vast.ai today! 🚀_


Deploy DeepSeek R1-0528 on Vast.ai to access transparent, step-by-step AI reasoning without prompt tricks—ideal for education, research, and business use.

DeepSeek R1-0528: Enhanced Reasoning with Simplified Thinking Mode on Vast.ai


We're back with a new update from the team at **Vast.ai**\! This month, we've rolled out fresh hardware options, new video-generation tools, and continued improvements behind the scenes.

The big news is that our GPU lineup now includes the **NVIDIA DGX B200** as well as the **RTX PRO 6000 WS**\! These powerful machines open the door to even more demanding workloads, bringing you the compute power required for everything from multi-modal inference to precision-driven simulations.

We also recently reached an exciting milestone: we now have **over one thousand RTX 5090 GPUs available on Vast** – 1,180 and counting.

Here are the details about what's new this month\!

### NVIDIA Cloud GPU Updates

Whether you're training foundation models, rendering high-resolution video, or running inference, our latest hardware additions give you extra flexibility:

-   **NVIDIA DGX B200** – A universal AI supercomputer built on the Blackwell architecture, equipped with **eight GPUs** and **1.4 TB** of GPU memory. With massive bandwidth and compute capacity, it's well suited to handle virtually any enterprise AI workload.

-   **RTX PRO 6000 WS** – A high-end Blackwell-based workstation GPU featuring **96 GB** of GDDR7 memory and up to **4000 TOPS** of AI compute. It can easily handle massive datasets and multi-billion-parameter models while supporting seamless multi-app workflows.

### New Guide and Templates

We've focused on video generation this month, adding a couple of new templates and a guide to help you go from idea to clip as seamlessly as possible:

-   [**Guide to Video Generation**](https://docs.vast.ai/video-generation) – A walkthrough for setting up and using ComfyUI on Vast.ai, so you can create advanced stable diffusion pipelines (ideal for video generation workflows) with this powerful node-based interface.

-   **LTX Video (ComfyUI)** – A ready-to-use template that pairs the LTX Video model with ComfyUI, optimized for consumer-grade GPUs.

-   **Open-Sora** – A template for this open-source video generation model with a streamlined, user-friendly platform that lowers the barrier to efficient, high-quality video content creation.

And that's a wrap for June 2025\!

### Our Commitment

Whether you're looking to launch a B200 instance, experiment with video generation using our latest templates, or scale up with our expanding lineup of RTX 5090 GPUs, we're here to support your work at every stage. Our goal at **Vast.ai** is to make powerful compute accessible, flexible, and easy to use – so you can focus on building, training, and creating.

Need help? Reach out to our customer support team anytime at [support@vast.ai](mailto:support@vast.ai), or connect with the community on our Discord server for tips, troubleshooting, and more.

### Change Log

#### New Templates & Guides:

-   [Guide to Video Generation](https://docs.vast.ai/video-generation)
-   LTX Video (Comfy UI) – Video-Gen model optimized for consumer-grade GPUs.
-   Open-Sora – Democratizing efficient video production for everyone.

#### New GPUs:

-   DGX B200
-   RTX PRO 6000 WS

Plus we now have over one thousand 5090 GPUs available.


New DGX B200 and RTX PRO 6000 WS GPUs, over 1000 RTX 5090s available, and new video generation templates.

June 2025 Product Update


Ten years ago today, our CEO and founder, Jacob Cannell, published ["The Brain as a Universal Learning Machine"](https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine) on LessWrong.com.

Drawing from neuroscience, machine learning, and systems theory, he proposed that **the human brain is not a patchwork of specialized, genetically programmed modules, but a general-purpose learning system** – or, as he put it, "a biological implementation of a _Universal Learning Machine_."

In hindsight, the timing was notable. Cannell published the piece just as deep learning was moving from niche research to broader relevance in artificial intelligence (AI). In the decade since then, rapid progress in scalable, trainable, data-driven models has validated the idea that learning, not pre-engineered structure, is the key to building intelligent systems.

So on the tenth anniversary of its publication, we'd like to revisit the key ideas of the article, reflect on what's aged well, and look ahead at what's still unfolding.

We’d encourage you to read the full article on [Less Wrong](https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine).


A look at the brain as a universal learning machine, and how it can be used to learn about the world.

Blog

How to run Mixtral 8X7B

Threadripper: Balancing CPU and GPU for Optimized Performance in Python

November product updates

Accessing the GPU marketplace with Vast.ai and dstack

Quieting The Chip Crunch: Understanding the GPU Supply Shortage

PyTorch vs TensorFlow: Which One Is Right For You