
Qwen 3.5 is Alibaba's latest model family, featuring a hybrid architecture that combines Gated DeltaNet (linear attention) with standard attention in a 3:1 ratio. This design enables dramatically faster inference — up to 8.6x at 32K context and 19x at 256K — while maintaining strong reasoning capabilities. The medium models released February 24, 2026 include:

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Total Params</th>
      <th>Active Params</th>
      <th>Architecture</th>
      <th>VRAM (BF16)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Qwen3.5-122B-A10B</td>
      <td>122B</td>
      <td>10B</td>
      <td>MoE</td>
      <td>~244 GB</td>
    </tr>
    <tr>
      <td>Qwen3.5-35B-A3B</td>
      <td>35B</td>
      <td>3B</td>
      <td>MoE</td>
      <td>~66 GB</td>
    </tr>
    <tr>
      <td>Qwen3.5-27B</td>
      <td>27B</td>
      <td>27B</td>
      <td>Dense</td>
      <td>~54 GB</td>
    </tr>
  </tbody>
</table>

The MoE models route each token through 8 of 256 experts plus 1 shared expert, so only a fraction of the total parameters are active per token. The 35B-A3B model activates just 3B parameters per token while having 35B total — giving you large-model quality at small-model inference cost. All three models are Apache 2.0 licensed, so no HuggingFace token is needed.

In this guide, we'll deploy Qwen3.5-35B-A3B on Vast.ai using SGLang, which has day-0 Qwen3.5 support as of v0.5.9, on a single A100 80GB GPU.

## Deploying on Vast.ai

Install the Vast.ai CLI and set your API key:

```bash
pip install --upgrade vastai
export VAST_API_KEY="YOUR_KEY_HERE"
vastai set api-key $VAST_API_KEY
```

Search for a suitable GPU. The model needs ~66 GB VRAM in BF16, so an 80 GB card works:

```bash
vastai search offers "gpu_ram>=80 num_gpus=1 direct_port_count>=1 rentable=true disk_space>=200 cuda_vers>=12.2" -o dph
```

Deploy the model using SGLang's Docker image. Replace `YOUR_INSTANCE_ID` with an offer ID from the search results:

```bash
vastai create instance YOUR_INSTANCE_ID \
    --image lmsysorg/sglang:latest \
    --env '-p 8000:8000' \
    --disk 200 \
    --onstart-cmd "python3 -m sglang.launch_server \
        --model-path Qwen/Qwen3.5-35B-A3B \
        --host 0.0.0.0 --port 8000 \
        --tp-size 1 \
        --context-length 32768 \
        --reasoning-parser qwen3 \
        --mem-fraction-static 0.85"
```

Key flags:
- `--reasoning-parser qwen3` enables the model's built-in thinking mode, separating reasoning from the final answer
- `--mem-fraction-static 0.85` allocates 85% of GPU memory for KV cache
- `--context-length 32768` sets a 32K context window (the model supports up to 262K natively)

The model will download (~70 GB) and load automatically. Check `vastai logs YOUR_INSTANCE_ID` — you'll see "The server is fired up and ready to roll!" when it's serving.

## Calling the Model

Find your instance's IP and port in the [Instances tab](https://cloud.vast.ai/instances/) of the Vast.ai console. Click the IP address button to see the port mapping for 8000/tcp.

Install the OpenAI SDK:

```bash
pip install --upgrade openai
```

Then call the model:

```python
from openai import OpenAI

VAST_IP_ADDRESS = "YOUR_IP"
VAST_PORT = "YOUR_PORT"

client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B",
    messages=[
        {"role": "user", "content": "What are three benefits of mixture-of-experts models? Be concise."}
    ],
    max_tokens=1024,
    temperature=0.7
)

print(response.choices[0].message.content)
```

### Response

```
1. **Computational Efficiency:** Only a subset of parameters is activated per input,
   reducing inference costs while maintaining a massive total parameter count.
2. **Enhanced Performance:** The larger effective model capacity allows for better
   accuracy on complex and diverse tasks compared to dense models of similar compute.
3. **Specialization:** Individual experts can learn to handle specific input patterns
   or domains, improving the model's robustness and adaptability.
```

With `--reasoning-parser qwen3`, the model's thinking process is automatically separated from the final answer. The thinking tokens appear in `reasoning_content` in the API response, while the clean answer appears in `content`.

## Other Models

You can swap `Qwen/Qwen3.5-35B-A3B` for the other medium models:

- **Qwen3.5-122B-A10B**: Larger MoE model, 122B total / 10B active. At BF16 the full weights need ~244 GB, so you'll need multiple GPUs (e.g., 4× A100 80GB with `--tp-size 4`). Alternatively, GGUF quantizations are available — the Q4_K_M variant is ~70 GB and fits on a single 80GB card.
- **Qwen3.5-27B**: Dense model, all 27B parameters active. Needs ~54 GB VRAM. Fits on a single A100 or H100.

## Conclusion

Qwen3.5-35B-A3B packs 35B parameters into a model that only activates 3B per token, and its hybrid DeltaNet architecture delivers substantially faster inference than standard transformers. With SGLang and a single A100 on Vast.ai, you can have it serving in under 10 minutes.

For production use, consider increasing `--context-length` — the model supports up to 262K natively, with 128K+ recommended for best reasoning performance. GGUF quantizations (Q4_K_M at ~22 GB) are also available for running on consumer GPUs like the RTX 4090.

## Useful Resources

- [Qwen3.5-35B-A3B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)
- [Qwen3.5-122B-A10B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)
- [Qwen3.5-27B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-27B)
- [SGLang Documentation](https://docs.sglang.ai/)
- [Vast.ai Documentation](https://docs.vast.ai/)


Deploy Qwen 3.5 medium models on Vast.ai using SGLang, including setup, launch commands, and API usage.

Running Qwen 3.5 Medium Models on Vast.ai


Liquid AI just released [LFM2-24B-A2B](https://huggingface.co/LiquidAI/LFM2-24B-A2B), a 24 billion parameter foundation model that only activates 2.3 billion parameters per token. That ratio — less than 10% of total parameters active at inference time — makes it one of the most aggressively sparse mixture-of-experts models available today.

What makes LFM2 different from other MoE models isn't just the sparsity. The architecture is a hybrid design that replaces most of the attention layers with gated short convolution blocks. Out of 40 total layers, only 10 use grouped query attention — the other 30 are convolutional. This is a fundamentally different approach from transformer-only models, and it's what allows Liquid AI to push the active parameter count so low while maintaining competitive quality across benchmarks like MMLU-Pro, GPQA Diamond, IFEval, and MATH-500.

Liquid AI, an MIT spinoff, has scaled this architecture from 350M to 24B parameters following log-linear scaling laws. LFM2 is the first model in the family large enough to be broadly useful — and with day-one support for vLLM, SGLang, and llama.cpp, it's ready to deploy today.

> **Note**: LFM2-24B-A2B is an early checkpoint trained on 17 trillion tokens without reinforcement learning or extensive post-training. A post-trained version (LFM2.5-24B-A2B) is coming.

## Model Overview

<table>
  <thead>
    <tr>
      <th>Property</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Developer</td>
      <td><a href="https://www.liquid.ai">Liquid AI</a></td>
    </tr>
    <tr>
      <td>Model</td>
      <td>LFM2-24B-A2B</td>
    </tr>
    <tr>
      <td>Architecture</td>
      <td>Hybrid MoE: 30 convolutional + 10 GQA attention layers, 64 experts, top-4 routing</td>
    </tr>
    <tr>
      <td>Total Parameters</td>
      <td>24B</td>
    </tr>
    <tr>
      <td>Active Parameters</td>
      <td>2.3B per token</td>
    </tr>
    <tr>
      <td>Context Length</td>
      <td>32,768 tokens</td>
    </tr>
    <tr>
      <td>Training Data</td>
      <td>17T tokens (mixed BF16/FP8)</td>
    </tr>
    <tr>
      <td>License</td>
      <td><a href="https://www.liquid.ai/lfm-open-license-v1-0">LFM Open License v1.0</a></td>
    </tr>
    <tr>
      <td>HuggingFace</td>
      <td><a href="https://huggingface.co/LiquidAI/LFM2-24B-A2B">LiquidAI/LFM2-24B-A2B</a></td>
    </tr>
  </tbody>
</table>

## Deploy LFM2-24B-A2B on Vast.ai with vLLM

At BF16, the model weights are ~48 GB — it fits on a single A100-80GB with room for a 32K context KV cache. No quantization or multi-GPU setup needed.

### Prerequisites

- A [Vast.ai](https://cloud.vast.ai) account with credits
- The Vast CLI installed and configured:

```bash
pip install --upgrade vastai  # make sure you're on v0.5.0+
vastai set api-key <YOUR_API_KEY>
```

### Find a GPU

Search for a single GPU with at least 80 GB VRAM and direct port access:

```bash
vastai search offers "gpu_ram>=80 num_gpus=1 direct_port_count>=1 cuda_vers>=12.4 cuda_vers<13" -o dph
```

> **Important**: Use a machine with **CUDA 12.x** (not 13.x). The vLLM nightly image is built against CUDA 12 and will fail with a driver mismatch on CUDA 13 machines.

### Launch the Instance

Pick an instance ID from the search results and deploy with vLLM. LFM2 requires a newer version of the `transformers` library than what ships in the vLLM image, so we upgrade it at startup:

```bash
vastai create instance <INSTANCE_ID> \
  --image vllm/vllm-openai:nightly \
  --env "-p 8000:8000" \
  --disk 100 \
  --onstart-cmd "pip install 'transformers>=5.0.0' && \
    vllm serve LiquidAI/LFM2-24B-A2B \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 32768 --dtype auto \
    --trust-remote-code"
```

This upgrades the tokenizer library, pulls the model from HuggingFace, and starts an OpenAI-compatible API server on port 8000. Initial startup takes several minutes while the model downloads (~48 GB for the BF16 weights).

Once the instance is running, find your IP and port in the [Vast console](https://cloud.vast.ai/instances/) — click the IP button on your instance to see the port mapping.

> **Why `nightly` and `transformers>=5`?** LFM2's tokenizer uses the `TokenizersBackend` class introduced in `transformers 5.0`. The stable vLLM image pins `transformers<5`, so the tokenizer fails to load. Using `nightly` with a transformers upgrade resolves this. This was tested with vLLM nightly build `v0.16.0rc2`. Once a stable vLLM release ships with `transformers>=5.0.0` support, you can switch to `vllm/vllm-openai:latest` and drop the `pip install` step.

## Call the API

```bash
curl http://<VAST_IP>:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "LiquidAI/LFM2-24B-A2B",
    "messages": [{"role": "user", "content": "What makes mixture-of-experts models efficient?"}],
    "max_tokens": 300
  }'
```

## Cleanup

When you're done, destroy the instance to stop charges:

```bash
vastai destroy instance <INSTANCE_ID>
```

## Resources

- [LFM2-24B-A2B on HuggingFace](https://huggingface.co/LiquidAI/LFM2-24B-A2B)
- [LFM2-24B-A2B GGUF Quantizations](https://huggingface.co/LiquidAI/LFM2-24B-A2B-GGUF)
- [Liquid AI Blog: LFM2-24B-A2B](https://www.liquid.ai/blog/lfm2-24b-a2b)
- [Liquid AI Models](https://www.liquid.ai/models)
- [Vast.ai Documentation](https://docs.vast.ai/)
- [vLLM Documentation](https://docs.vllm.ai/)


Liquid AI released LFM2-24B-A2B, a sparse 24B MoE model with 2.3B active parameters per token.

Liquid AI's LFM2 Just Dropped — Here's How to Run It on Vast.ai


Claude Code is one of the most capable agentic AI tools for software development – but it comes with a practical constraint: usage limits and spend. It's easy to run into daily or weekly quotas, and API costs can add up quickly.

Fortunately, there's a workaround.

### Ralph Loop: Running an Open-Source Coding Agent on Vast.ai

Instead of relying on API-limited services or local hardware, you can run open-source coding agents on Vast.ai for about **$1.50 per hour** – even continuously overnight! One example of this is Ralph, an agentic coding loop designed to implement software projects from a Product Requirements Document (PRD).

Ralph picks a user story, writes the code, runs tests, fixes any failures, and moves on to the next story. This process repeats until the project is complete and everything passes.

To jump straight in, follow our [Overnight Ralph Loop guide](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop) to get started. Otherwise, let's explore some more about what you can accomplish with Ralph and why.

### Why Run the Ralph Loop on Vast.ai?

There are several practical reasons to run the Ralph loop on a self-hosted model using our dedicated GPU infrastructure.

First, cost becomes more manageable. On Vast.ai, you pay for GPU time directly, at cost-effective rates that make overnight runs both predictable and affordable. Ralph is built for sustained workflows, so you can run it continuously without the usage caps of API-based tools like Claude Code.

Running Ralph on Vast.ai also gives you direct control over your development environment and supports stricter security and [compliance](https://vast.ai/compliance) needs. At the same time, you can match model size and GPU compute to the complexity of the task, ensuring your resources scale up only when the workload requires it.

### The Ralph Loop in Practice: Model & Use Cases

On Vast.ai, you can run the Ralph loop on Qwen3-Coder-Next, a large Mixture-of-Experts (MoE) model trained specifically for agentic coding tools like Claude Code, Aider, and Cline. It boasts a 256K context length and strong performance across multi-step coding tasks involving large codebases.

Once deployed, Ralph operates continuously and autonomously on real software projects, making it well suited for overnight development workflows. Common use cases include:

- Building a full CLI application
- Implementing REST APIs with authentication and validation
- Generating complete test suites for existing codebases
- Creating a web scraper with multiple site adapters, rate limiting, and data export

You simply define the requirements and allow the agent to carry out the work unattended.

### Getting Started with the Overnight Ralph Loop

With Vast.ai, a typical setup using Qwen3-Coder-Next on a 4x RTX 4090 instance costs approximately $1.50 per hour, with an overnight run often coming in **under $18** .

Our [Overnight Ralph Loop guide](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop) walks you through deploying the model, connecting the agent, and running the Ralph loop on Vast.ai. Start your first overnight run today!


Run autonomous coding agents on Vast.ai for about $1.50/hour with the Ralph Loop workflow.

Affordable Claude Code Alternative: Run Autonomous Coding Agents on Vast.ai


Welcome back to another round of updates at Vast.ai. We've kicked off the new year with improvements focused on expanding orchestration support, refining billing and serverless workflows, and introducing new templates and guides to help you deploy faster and more securely.

Another exciting update: we recently launched the [Vast.ai Startup Program](https://vast.ai/startup), designed to support early-stage teams building on our platform.

Startups can now get up to **$2,500 in free GPU credits** when they move their workloads to Vast.ai. With 24/7 priority support and access to Vast's full range of GPUs, the program delivers high-performance compute to power the next wave of AI innovation without long-term commitments. For more info, see our blog post [here](https://vast.ai/article/vast-ai-startup-program).

Not to mention we have a comprehensive new [Model Library](https://vast.ai/model-library), giving you a faster path from model selection to deployment.

## NVIDIA Cloud GPU Updates

### SkyPilot Enhancements

For teams using [SkyPilot](https://vast.ai/article/vast-ai-gpus-can-now-be-rentend-through-skypilot) to manage deployments, we've made updates that expand configuration options on Vast.ai. SkyPilot users can now filter capacity by Secure Cloud to ensure that workloads run only on verified datacenter-hosted GPUs.

We've also added full support for instance creation parameters via SkyPilot, enabling precise control over images, environment variables, disk size, pricing, startup commands, and other instance-level settings at launch. See [SkyPilot docs](https://docs.skypilot.co/en/latest/reference/config.html#vast) for full details.

### Fresh New Templates and Guides

Our [Model Library](https://vast.ai/model-library) introduces a model-first approach to deploying AI. Templates are organized by modality and use case and include recommended hardware presets and clear dedicated GPU pricing.

Here are some of our newest templates:

- [Kimi K2.5](https://vast.ai/model/kimi-k2.5) - An open-source, native multimodal agentic model that combines vision and language understanding with agentic capabilities.
- [LTX-2](https://vast.ai/model/ltx-2) - A DiT-based audio-video generation model that produces synchronized video and sound.
- [SGLang](https://cloud.vast.ai/template/readme/1fce7860e11071cae9c357fb1c8b7560) - A high-performance LLM inference framework with a hosted API server template.

And our latest guides:

- [dstack + vLLM](https://docs.vast.ai/examples/serving-infrastructure/dstack-vllm): Deploy language models using vLLM with dstack for automated provisioning and cost control.
- [DR-Tulu](https://docs.vast.ai/examples/mcp/dr-tulu): Deploy a research agent trained with an MCP server and web tools.
- [GLiNER2](https://docs.vast.ai/examples/ner/gliner2): Run GLiNER2 for named entity recognition with custom entity types.
- [BrowseSafe](https://docs.vast.ai/examples/ai-agents/browsesafe): Deploy BrowseSafe to detect prompt injection attacks in HTML.
- [Overnight Ralph Loop](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop): Budget-friendly alternative workflow to Claude Code.

### Other Improvements

If you missed it late last year, we also launched [Vast.ai Serverless](https://vast.ai/article/vast-ai-serverless-automated-gpu-scaling). It enables automatic, pay-per-use scaling for AI workloads through a fully serverless API on Vast's globally distributed GPU cloud.

## Our Commitment

With new SkyPilot enhancements, startup opportunities, a model-first deployment experience, and improved inference workflows, we're continuing to build a platform that gives you more control with less friction.

Need help? Contact our customer support team anytime at [support@vast.ai](mailto:support@vast.ai) or join our [Discord](https://discord.gg/vast) server for tips, real-time updates, and ongoing community discussion.

## Change Log

### SkyPilot Enhancements

- Advanced configuration options for SkyPilot on Vast.ai.
- Support for filtering by Secure Cloud (`datacenter_only`).
- Full support for instance creation parameters via SkyPilot.
- See [SkyPilot docs](https://docs.skypilot.co/en/latest/reference/config.html#vast).

### New Templates

- [Kimi K2.5](https://vast.ai/model/kimi-k2.5): Multimodal agentic model.
- [LTX-2](https://vast.ai/model/ltx-2): Video and audio generation model.
- [SGLang](https://cloud.vast.ai/template/readme/1fce7860e11071cae9c357fb1c8b7560): High-performance LLM inference engine.
- Find them in the [Model Library](https://vast.ai/model-library).

### Template Updates

- Base Image and PyTorch now support **CUDA 13.1** and updated NVML libraries.
- Ostris AI Toolkit updates.
- ComfyUI updates.
- Llama.cpp (ssl) updates.
- SGLang onstart script fix.
- Pinokio v6.x updates.

### New Guides

- [dstack](https://docs.vast.ai/examples/serving-infrastructure/dstack-vllm): Serving infrastructure.
- [DR-Tulu](https://docs.vast.ai/examples/mcp/dr-tulu): MCP.
- [GLiNER2](https://docs.vast.ai/examples/ner/gliner2): Named Entity Recognition.
- [BrowseSafe](https://docs.vast.ai/examples/ai-agents/browsesafe): AI agents and security.
- [Overnight Ralph Loop](https://docs.vast.ai/examples/ai-agents/overnight-ralph-loop): Budget-friendly alternative to Claude Code.

### Platform Improvements

- Fixed serverless GPU filter UI.
- Fixed serverless template selection UI.
- Billing Transaction History Invoice PDF Export now excludes failed and pending payments and other non-payment activity.
- Billing Transaction History now supports easy individual invoice download.
- Security improvements.
- Local volume deletion issue resolved.
- Copy progress bar issue resolved.
- Disk quota issues resolved.
- Machines missing geolocation resolved.
- Host CLI schedule maintenance command issue resolved.


Vast.ai's January and February 2026 updates include new SkyPilot controls, Startup Program credits, Model Library additions, fresh guides, and major platform fixes.

January & February 2026 Product Updates


### Introducing the Vast.ai Startup Program: AI Infrastructure Without the Burn Rate

Startups can now get [$2,500 in free GPU credits](https://vast.ai/startup) when they move workloads to Vast.ai.

To apply, get in touch with our team to share a bit about your project and compute needs. From there, we'll finalize your credit offer, help you configure the right setup, and get you up and running quickly. You'll also have access to **24/7 priority support** throughout the credit period.

Compute credits apply across Vast.ai's global GPU marketplace, with access to everything from consumer-grade cards to high-end accelerators like A100s and H200s.

The Vast.ai Startup Program gives startups an on-ramp to the same platform that teams rely on well beyond the early stage. That's because Vast.ai is designed to scale alongside you - without long-term commitments or runaway costs.

### Built for Real AI Workloads

From day one, Vast.ai supports the tools and workflows that teams already use. Launch preconfigured environments with Jupyter, Docker, RAPIDS, and Hugging Face to train deep learning models, run large-scale experiments, and process massive datasets on demand.

We offer a growing selection of ready-to-run AI models in our [Model Library](https://vast.ai/model-library), plus thousands of [templates](https://cloud.vast.ai/templates) to get started immediately - or deploy your own custom model anytime. Your team can even collaborate across departments and campuses, avoiding duplicated infrastructure and fragmented workflows.

Most importantly, with Vast.ai, your team can cut costs without cutting capability. Reduce infrastructure expenses by **up to 81%** compared to major cloud GPU providers, while paying only for what you need, when you need it.

Plus, most startups can migrate existing workloads with minimal changes and begin running jobs shortly after approval.

### Secure, Compliant, and Production-Ready

Vast.ai is **SOC 2 Type II** certified and designed to support all of your [security and compliance](https://vast.ai/compliance) needs from the start.

Run workloads in isolated instances with end-to-end encryption and full control over compute environments to keep your sensitive data protected. Our [Secure Cloud](https://cloud.vast.ai/?secure_cloud=true) option lets you filter for GPUs from our vetted datacenter partners for even higher security needs.

Notably, these safeguards support reproducible research via consistent environments, as well as compliance-friendly configurations aligned with GDPR and HIPAA requirements.

With [Vast.ai Serverless](https://vast.ai/article/vast-ai-serverless-automated-gpu-scaling), moving from development to production is even easier. Inference workloads run through a fully serverless API that automatically routes to the most cost-effective GPUs available, according to your performance requirements. This eliminates manual instance management and capacity planning - so you can leverage our distributed GPU fleet with minimal overhead.

### Get Started with Vast.ai Today

For early-stage teams, access to affordable GPU compute is crucial. But there's more to it than that. You also want infrastructure that scales as you grow and supports evolving security and compliance needs - all without locking you into long-term contracts.

The Vast.ai Startup Program helps you get there faster.

[Apply today](https://vast.ai/startup) to get up to 2,500 free compute hours and start building on Vast.ai on your own terms.


Startups can get up to $2,500 in free GPU credits when moving workloads to Vast.ai, with priority support and production-ready infrastructure.

Introducing the Vast.ai Startup Program


If you've ever tried generating AI video and thought, *This looks cool, but it's not what I asked for*, you're not alone!

**Wan 2.2** was designed to address scaling and efficiency challenges in video generation. It is among the first open-source video generation models to leverage a Mixture-of-Experts (MoE) architecture, enabling more efficient training. Created by Alibaba Tongyi Lab, the model gives creators more control over how their ideas turn into motion – following prompts more closely, with smoother movement and cinematic-quality results – without drastically increasing compute requirements.

On Vast.ai, Wan 2.2 is available in two variants in our [Model Library](https://vast.ai/article/vast-ai-model-library-choosing-best-ai-models):

- **[Text-to-video (T2V)](https://vast.ai/model/wan-2.2-t2v-a14b)** – Generating short videos directly from written prompts.
- **[Image-to-video (I2V)](https://vast.ai/model/wan-2.2-i2v-a14b)** – Transforming a static image into a dynamic video sequence.

Both models share the same underlying architecture, which is where things get interesting...

## What Makes Wan 2.2 Different

Most diffusion-based video models use a single neural network to handle the entire denoising process. Wan 2.2's MoE design, on the other hand, uses two specialized "experts" that strategically separate the video generation workflow:

- A **high-noise expert** that handles the early denoising stages, focusing on layout, composition, and overall scene structure and motion patterns.
- A **low-noise expert** that takes over later, refining video details and enhancing cinematic elements like lighting, contrast, and color tone.

This separation helps reduce common issues with video generation like unstable camera motion and inconsistent frames. It also makes the process more efficient. Even though each model has approximately **27 billion parameters** in total, only about **14 billion are active** per inference step.

The models automatically switch between experts via signal-to-noise ratio (SNR) thresholds – a measure of how much meaningful visual information remains versus how much noise is still present at each stage of generation. By activating the right expert at the right moment, Wan 2.2 avoids unnecessary computation. This keeps performance much more efficient, comparable to smaller single-expert models.

In short, you get more natural motion dynamics, superior visual fidelity, reduced artifacts, and less wasted compute.

## One Architecture, Two Workflows

While their underlying architecture is the same, the two Wan 2.2 variants have different creative entry points.

### Wan2.2 T2V A14B (FP8)

The text-to-video (T2V) model uses text prompts to generate **5-second clips at 480P or 720P** resolution, with granular control over lighting, composition, contrast, and color tone.

A few other capabilities of the T2V variant include:

- Superior performance on Wan-Bench 2.0 evaluations compared to leading commercial video generation systems
- Integration with Qwen models or DashScope API for enhanced prompt elaboration
- Support for parameter-type conversion, leading to improved inference speed

Overall, the text-to-video model is well suited for storyboarding, concept visualization, marketing clips, and rapid prototyping of video ideas from plain language.

### Wan2.2 I2V A14B (FP8)

The image-to-video (I2V) model begins with a static image input and turns it into a short video sequence. It offers both **480P and 720P output**, and can optionally support text prompts to guide content and motion in detail.

Some strengths of the I2V variant include:

- Handling a wide range of aesthetic preferences, from photorealistic to stylized
- Processing at **24 frames per second** (FPS) for smooth, high-definition output
- Automatic prompt derivation that makes it possible to generate videos from an image alone *without* text

The image-to-video model is especially useful for turning concept art, product photos, and illustrations into animated content.

## From Idea to Output: Designed for Real-World Use

Whether you start from a text prompt or from an image, Wan 2.2 is built for real-world creative and production workflows.

You don't even need enterprise-grade hardware to use the model. Wan 2.2's MoE architecture keeps things so efficient, you can achieve high-quality results on consumer GPUs like the RTX 4090. For larger workloads, multi-GPU inference is also an option. The model supports FSDP and DeepSpeed Ulysses for accelerated generation.

Wan 2.2 pairs particularly well with Vast.ai's flexible GPU infrastructure. On Vast, you can spin up exactly the hardware you need and run the model on a single GPU, or scale across multiple GPUs for faster generation as your video projects grow.

Both the [T2V](https://vast.ai/model/wan-2.2-t2v-a14b) and [I2V](https://vast.ai/model/wan-2.2-i2v-a14b) models are compatible with [ComfyUI](https://docs.vast.ai/documentation/serverless/comfyui-wan-2.2) workflows, making them easy to integrate into tools you may already be using – which are also available directly on Vast.ai.

Ready to start creating with Wan 2.2? Launch it on [Vast.ai](https://vast.ai/model-library) in just a few clicks and start generating cinematic video today!


Wan 2.2 uses a Mixture-of-Experts architecture to deliver efficient, high-quality AI video generation. Learn how it works and why it matters.

Wan 2.2 Explained: A New Approach to AI Video Generation


Wouldn't it be great if you could just think of a scene and instantly turn it into a video the way it appears in your head? Technology isn't quite there yet, but today’s most advanced AI video generation models are getting us closer.

Today we’re taking a look at **WAN 2.2** and **LTX-2**, two open-source/open-weights models that transform text and images into short-form video.

## What WAN 2.2 and LTX-2 Are — and How They Differ

From the outside, WAN 2.2 and LTX-2 are pretty similar tools. They’re both open-source/open-weights diffusion-based video generation models designed to turn images or text prompts into short video clips. Their underlying architecture, however, is very different.

### WAN 2.2: Prompt Fidelity and Cinematic Control

Developed by Alibaba Tongyi Lab, WAN 2.2 is built around a Mixture-of-Experts (MoE) architecture. Instead of using a single neural network to manage the entire denoising process, it employs two specialized “experts”: a **high-noise expert** for overall structure and layout, and a **low-noise expert** for refining textures and details like lighting and color tone.

Switching between these two experts means the model can allocate compute depending on what it needs to do at any given moment—focusing on broader structure first and finer details later. It also boosts efficiency by avoiding unnecessary computation.

WAN 2.2 comes in three main variants, each designed for different workflows:

- **Text-to-Video (T2V):** Generates 5-second video clips at 480P to 720P from a text prompt written in plain language. This is a flexible option for scenes where everything needs to be synthesized from scratch.
- **Image-to-Video (I2V):** Begins with a single image and turns it into a short video. It uses automatic prompt derivation to generate video from an image without text input at all, but can also support text prompts for more directed results.
- **Hybrid:** A compact model with 5 billion parameters that handles both text-to-video and image-to-video generation. It delivers high-definition results at up to 720P and 24 FPS, but is designed for users with lower VRAM.

The base WAN 2.2 models generate video only, without native audio output. However, there is a specialized speech-to-video version (WAN 2.2 S2V) that transforms static images and audio inputs into synchronized videos.

### LTX-2: Native Audio-Video Generation

Created by Lightricks, LTX-2 is a DiT-based (Diffusion Transformer) audio-video generation model. It produces audio and visuals together in one pass, keeping dialogue, lip movements, and ambient sound aligned coherently.

Its architecture is based on latent diffusion, which means the model works in a compressed version of the video first, before converting it into full resolution. This makes it more memory efficient and enables faster iteration, translating to quicker experimentation and lower hardware overhead.

LTX-2 can generate up to **~20 seconds of synchronized audio and video**, with support for high resolutions and high frame rates depending on configuration and available compute. The model offers fine-grained control options—such as LoRA-based customization and multimodal inputs including text, image, video, and audio—for precise creative direction.

This makes LTX-2 a highly flexible model. In short, it supports text-to-video, image-to-video, and native audio-visual generation, along with cross-modal workflows like audio-to-video, text-to-audio, and video-to-audio—all within a single model.

## Choosing the Right Model for Your Workflow

How the two models are designed directly affects what you experience as a user.

For instance, WAN 2.2's MoE design prioritizes structured generation and motion consistency. It boasts strong prompt adherence with high-fidelity output and is more likely to preserve scene intent across frames, sticking closely to what you asked for—albeit at the cost of slightly longer generation times.

LTX-2's latent diffusion approach emphasizes speed and accessibility. It’s faster to iterate with, easier to experiment on, and even offers native audio-video sync. However, it may require more prompt tuning to get exactly what you want.

### Choose **WAN 2.2** if you want:

- Cinematic or narrative-style clips where composition and camera motion are critical
- Strong prompt fidelity for complex scenes with multiple elements
- More deliberate, production-oriented outputs and professional video content

### Choose **LTX-2** if you prefer:

- Rapid prototyping of video concepts and creative exploration for lengthier scenes
- Visual storytelling or character-driven video with synchronized dialogue or sound
- A lighter and more iterative workflow where speed matters more than precision

Both models also integrate with ComfyUI, so you can jump right into testing them out with an intuitive node-based visual workflow.

## Final Thoughts

Neither WAN 2.2 nor LTX-2 is objectively superior to the other. The two open-source/open-weights models are designed for different kinds of workflows and creative goals. The best way to get a feel for them is to actually try them out.

The good news is that both models run well on high-end consumer GPUs, making them far more accessible than many people might expect. With Vast.ai, it’s even easier: you can spin up the right hardware on demand and experiment on your own terms, paying only for the compute you need—and save up to 80% over traditional clouds.

Try [WAN 2.2 T2V](https://vast.ai/model/wan-2.2-t2v-a14b) and [WAN 2.2 I2V](https://vast.ai/model/wan-2.2-i2v-a14b), or [LTX-2](https://vast.ai/model/ltx-2) (or both!) in our [Model Library](https://vast.ai/model-library), and build your own creative pipeline on Vast.ai today.


A practical comparison of WAN 2.2 and LTX-2—two open-source/open-weights AI video models—covering architecture, strengths, and use cases.

WAN 2.2 vs. LTX-2: Which AI Video Model Should You Use?


dstack is an open-source GPU orchestration platform that automates instance provisioning and lifecycle management across cloud providers. This guide walks through using dstack with Vast.ai as the backend, combining declarative infrastructure with competitive GPU marketplace pricing.

## What Makes dstack Special?

Key features include:
- **Infrastructure as Code**: Define GPU requirements, pricing limits, and workloads in YAML
- **Automatic Provisioning**: dstack finds and provisions the best available instances
- **Cost Controls**: Set `max_price` to cap hourly costs automatically
- **Built-in Proxy**: Access services through dstack's authenticated endpoint

## What's in the New Guide

Our latest documentation walks you through deploying language models with dstack and vLLM on Vast.ai.

**Complete Setup Instructions**
- Installing and configuring dstack with your Vast.ai API key
- Starting the dstack server and CLI
- Creating service configurations for vLLM deployments

**Working Service Configurations**
- Ready-to-use YAML for deploying Qwen3-30B-A3B
- GPU memory and pricing parameters
- Real deployment outputs showing what to expect

**API Integration Examples**
- Python code using the OpenAI SDK
- cURL examples for testing
- Streaming response implementations

## Why dstack + Vast.ai?

The combination gives you orchestration on top of a GPU marketplace:

- **Simplified Workflow**: No manual instance hunting or environment setup
- **Cost Optimization**: dstack finds the cheapest instance meeting your requirements
- **Flexible Pricing**: Access Vast.ai's competitive rates with automatic cost caps
- **Production-Ready API**: vLLM provides OpenAI-compatible endpoints

The guide demonstrates deploying Qwen3-30B-A3B on an H100 80GB—provisioned with a single command.

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Teams** wanting reproducible, version-controlled GPU deployments
- **Developers** building LLM applications who want simple infrastructure management
- **Anyone** tired of manually provisioning and configuring GPU instances

## Get Started

The complete guide is now available in our documentation:

**[Read: Deploy LLMs with dstack and vLLM on Vast.ai →](https://docs.vast.ai/examples/serving-infrastructure/dstack-vllm)**

Whether you're new to GPU orchestration or looking for a better way to manage LLM deployments, this guide provides everything you need to get started with dstack on Vast.ai.

Ready to try it? [Sign up for Vast.ai](https://vast.ai) and follow the guide to deploy your first model.


Learn how to deploy language models with dstack and vLLM on Vast.ai, combining declarative infrastructure with competitive GPU marketplace pricing.

Deploy LLMs with dstack on Vast.ai


DR-Tulu is AI2's open-source research agent—an 8B model that autonomously plans research strategies, searches the web, reads pages, and synthesizes comprehensive answers with citations. Unlike typical LLMs with tools bolted on, DR-Tulu was trained end-to-end with its MCP tools, making it a compelling open alternative to proprietary research APIs. This guide walks through deploying DR-Tulu on Vast.ai, where you can run the full agent stack on affordable GPU instances and integrate it into your own agentic applications.

## What Makes DR-Tulu Special?

Key features include:
- **Native MCP Integration**: Trained with web search and page reading tools from the start
- **Autonomous Research**: Plans strategies, gathers sources, and synthesizes cited answers
- **Efficient Architecture**: 8B parameters with strong research capabilities

## What's in the New Guide

Our latest documentation walks you through deploying DR-Tulu on Vast.ai with a split architecture: vLLM for GPU inference on Vast.ai, MCP backend running locally for tool execution.

**Complete Deployment Instructions**
- Instance selection with the Vast.ai CLI
- vLLM configuration with the correct version (critical for compatibility)
- MCP backend setup with required API keys
- Workflow configuration to connect the components

**Three Ways to Use DR-Tulu**
- Interactive chat for exploratory research
- Batch evaluation against built-in datasets
- Python API for programmatic integration

## Why This Architecture?

The split deployment keeps GPU-intensive inference on Vast.ai while running tool orchestration locally. This means:

- Your API keys (Serper, Jina) stay on your machine (or wherever you deploy the MCP backend)
- Model inference doesn't require sending data to third-party APIs
- You can modify the MCP backend without redeploying the model
- Lower Vast.ai costs since you only need GPU for inference

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Researchers** who need cited, source-backed answers at scale
- **Developers** building applications that require web research capabilities
- **Teams** evaluating open alternatives to proprietary research APIs

## Get Started

The complete guide is now available in our documentation:

**[Read: Running DR-Tulu on Vast.ai →](https://docs.vast.ai/examples/mcp/dr-tulu)**

Whether you're exploring agentic AI architectures or ready to deploy your own research assistant, this guide provides everything you need to get DR-Tulu running on Vast.ai infrastructure.

Ready to try it? [Sign up for Vast.ai](https://vast.ai) and follow the guide to deploy your first instance.


Learn how to deploy DR-Tulu, AI2's open-source research agent, on Vast.ai for autonomous web research with citations.

Deploy Deep Search with DR-Tulu on Vast.ai


As 2025 comes to a close, we're looking back at the milestones and updates that shaped the past year for us here at Vast.ai.

What a year it's been! From new hardware to major security achievements and product launches, here are our top moments of 2025.

## Expanding Our Global GPU Fleet

The past year saw some of the biggest hardware expansions in Vast.ai's history.

In 2025, we added NVIDIA's latest RTX 5000-series GPUs – including the RTX 5090 – followed by high-performance machines like the RTX PRO 6000 WS and the H200 and B200. These additions supported higher-throughput workloads and more efficient AI inference across a [wide range](https://cloud.vast.ai/) of hardware profiles.

Another highlight: our RTX 5090 inventory surpassed 1,000 machines this year!

## Security and Compliance Milestones

This year, Vast.ai achieved both **SOC 2 Type I** and **Type II** certification. These independent audits validated that the security infrastructure we've built over 6+ years of trusted service continues to meet rigorous compliance standards.

We take this responsibility seriously. Data privacy and integrity remain core priorities in everything we build. As we continue to advance our [security and compliance](https://vast.ai/compliance) roadmap, we're committed to upholding the trust you place in us.

## New Templates and Tools

We introduced a number of ready-to-use templates this year, to help you move from idea to execution quickly and seamlessly. Some notable ones included:

- [Langflow + Ollama](https://docs.vast.ai/langflow-ollama) for visual AI workflow design
- [Ostris AI Toolkit](https://cloud.vast.ai/?ref_id=62897&template_id=81e6e9dd266d9bdbcbb0fd48b9afe357) for custom Stable Diffusion and LoRA training
- [Open-Sora](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Open-Sora) and [LTX Video](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%2520%252B%2520LTX%2520Video) (ComfyUI) for video generation workflows
- [ComfyUI ACE Step](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ACE%2520Step%2520Music%2520Generation) for AI-assisted music creation
- [InvokeAI](https://cloud.vast.ai/?ref_id=62897&template_id=c309df384010cf9d3dc9abb9e0a5a6fd) for image generation in a professional web interface
- [Dia 1.6b TTS](https://cloud.vast.ai/?ref_id=62897&template_id=6ba071acb5f590913c5eaf6c2dc3d0db) for lifelike text-to-speech

But the biggest update of the year was still ahead...

## Introducing Vast.ai Serverless

The headline story of 2025 was the [recent launch](https://vast.ai/article/vast-ai-serverless-automated-gpu-scaling) of **Vast.ai Serverless** – a fully automated GPU compute layer designed for AI inference at scale.

With our Serverless offering, you can easily run workloads without managing instances on your own or overpaying for capacity. You simply send inference requests and the platform automatically selects the right hardware, routes workloads across regions as needed, and predictively scales up and down based on usage.

In short, Vast.ai Serverless gives you access to a diverse, globally distributed GPU fleet with a serverless model that applies predictive optimization and flexible scaling – with none of the overhead. We're thrilled to finally make truly serverless AI inference a reality on Vast.ai!

## Looking Ahead

To all of our users and the wider Vast.ai community, thank you for being part of our journey this year. It's been an incredible year of growth, and we're grateful to everyone building and innovating with us.

Here's to an even stronger year ahead!


As 2025 comes to a close, we're looking back at the milestones and updates that shaped the past year for us here at Vast.ai.

Vast.ai's 2025 Year in Review


With so many AI model libraries to choose from, making the right decision requires research and testing. Luckily, [Vast.ai](http://vast.ai) partners with some of the leading innovators across a variety of generative and assistive models. Along with our GPU rentals, we empower businesses and individuals to produce their best work without latency issues or breaking the bank. Not only are these AI models reliable, they are also very affordable. You can scale production and workload with ease. Whether looking for an image generator, video production, audio, text or computer vision research, these are some of the best AI models available.

Model Library is Vast's new, model-first way to deploy AI. Instead of starting from the runtime-level (like vLLM, ComfyUI, or PyTorch), you can browse and discover model-level templates, sorted by modality and use case, then launch a running instance in seconds with full custom control over the underlying runtime, parameters, and infrastructure.

Every template includes recommended hardware presets and clear dedicated GPU rental pricing, so you can quickly get started with the right cost/performance setup and deploy with confidence, or tweak to fit your needs.

Coming soon: more runtimes, more model variations, and Vast Serverless model templates.

## The Best AI Image Generation Models

With text-based prompts, you can produce images from a range of popular AI models. From hyper-realistic renderings to cartoonish depictions, these models are among your best bets for generating images.

- **[HiDream I1 Full](https://vast.ai/model/hidream-i1-full)**: Released in May 2025, HiDream produces images in a variety of styles. Whether you want realistic photo imagery or stylized cartoon graphics, HiDream can deliver. With speed and precision, it can produce images without sacrificing quality.

- **[Qwen Image (FP8)](https://vast.ai/model/qwen-image)**: With a specific focus on generating images containing text, Qwen Image can produce high-quality images that provide a strong eye for detail. This is true not only of images with English characters, but even intricate characters for multilingual projects. It has a particular focus on generating images containing Chinese characters.

- **[FLUX.1 [dev]](https://vast.ai/model/flux.1-dev)**: Whether prototyping concepts, marketing, or digital art, FLUX.1 is an open-source tool that rivals many closed-source competitors in its functionality and precision. Developed by Black Forest Labs, this 12-billion-parameter image generator serves for a range of use cases from concepting to marketing materials and scientific visualization.

- **[Juggernaut XI v11](https://vast.ai/model/juggernaut-xi-v11)**: With a particular emphasis on photorealistic images, Juggernaut XI v11 is trained to provide exceptional detail on challenging human features like hands and facial expressions. It also provides exceptional detail for landscapes and other detailed imagery that have historically fallen into the realm of "uncanny valley."

- **[RealVisXL V5.0](https://vast.ai/model/realvisxl-v5.0)**: Developed by Evgeny, RealVisXL renders photorealistic images and produces anatomically accurate and high-fidelity visuals thanks to inference parameters like DPM++ SDE Karras sampling. This makes it equally adept at producing architectural designs and photorealistic images.

- **[Stable Diffusion XL Base 1.0](https://vast.ai/model/stable-diffusion-xl-base-1.0)**: Developed by Stability AI, Stable Diffusion XL Base improves image quality by providing two models. With a brainstorming mode in its base model, you can parse through a variety of images and then hone in on specific imagery with the more detailed refinement module. This holistic approach provides you with everything you need to craft the specific images you need.

## The Best AI Video Generation Models

[Vast.ai](http://vast.ai) offers some of the best video generation options available on the market. With options from a variety of AI engines and developers, you can produce stunning videos that rival cost-and-labor-intensive CGI and video productions.

- **[LTX Video](https://vast.ai/model/ltx-video)**: The first Diffusion Transformer (DiT)-based video generation model, LTX Video, produces high quality videos at resolutions and framerates that exceed much of the competition. Videos are available at 30 FPS and a 1216×704 resolution. This gives it an advantage over many of its competitors.

- **[Mochi 1 Preview](https://vast.ai/model/mochi-1-preview)**: Specializing in photorealistic video generation, Mochi 1 has 10 billion parameters that are built on an innovative Asymmetric Diffusion Transformer architecture. This helps it process over 44,000 visual tokens to produce high-quality videos.

- **[Wan2.2 I2V A14B (FP8)](https://vast.ai/model/wan-2.2-i2v-a14b)**: Wan2.2 I2V generates 720p videos from static image prompts. With a high-noise expert option for background imagery and a low-noise expert option for movement, you can make still images come to life with striking detail.

- **[Wan2.2 T2V A14B (FP8)](https://vast.ai/model/wan-2.2-t2v-a14b)**: Released in July 2025, Wan2.2 T2V generates 5-second videos at both 480P and 720P resolutions. With cinematic aesthetics and complex motion capabilities, it provides greater detail than many previous open-source and commercial models.

## The Best Audio Generation AI Models

Whether you need to produce a podcast, narrate an audiobook, engineer music production, or just explore the possibilities of ai-generated sounds, [Vast.AI](http://vast.ai) provides affordable audio generation options from some of the best-known models available.

- **[ACE Step V1 3.5B](https://vast.ai/model/ace-step-v1-3.5b)**: ACE Step V1 produces realistic audio thanks to its 3.5-billion-parameter model. It can be leveraged for a range of use cases: podcasts, audiobooks, and a wide range of music production prompts.

- **[Dia 1.6B](https://vast.ai/model/dia-1.6b)**: Developed by Nari Labs, Dia 1.6B is an audio generation model that specializes in multi-speaker production. This works great for podcasts and other text-to-audio prompts and excels at inflecting speakers' voices and producing sighs, laughter, and other non-verbal flourishes.

## The Best Text Generation AI Models Available

Whether you need agentic support, assistance on scientific research, or multilingual text translation, there are many incredible choices available. These are the best text generation AI models available from [Vast.AI](http://vast.ai).

- **[GLM 4.6](https://vast.ai/model/glm-4.6)**: This model provides support for language models that work on multi-step workflows. With over 200,000 tokens and sophisticated reasoning logic, it can excel at anything from coding applications to agentic capabilities.

- **[Kimi K2 Thinking](https://vast.ai/model/kimi-k2-thinking)**: Released under a Modified MIT License, Kimi K2 Thinking supports both commercial and research applications. Its chain-of-thought reasoning allows it to operate through hundreds of steps without degrading its response quality.

- **[DeepSeek V3.2 Exp](https://vast.ai/model/deepseek-v3.2-exp)**: Built for complex research and debugging scenarios, DeepSeek V3.2 Exp is an incredible assistant for very labor-intensive processes. With the ability to parse and analyze large bodies of text, you can automate some of the most tedious parts of your day in seconds.

- **[Kimi K2 Instruct 0905](https://vast.ai/model/kimi-k2-instruct-0905)**: With over a trillion parameters, Kimi K2 Instruct excels at language models including agentic intelligence and software development. Its massive 256k token window provides advanced decision-making capabilities for a range of issues and use cases.

- **[Qwen3 235B A22B Thinking 2507](https://vast.ai/model/qwen3-235b-a22b-thinking-2507)**: Developed by Alibaba, this 235-billion-parameter Mixture-of-Experts (MoE) language model is specifically engineered for extended reasoning tasks by activating 8 experts and 22 billion parameters per token. This makes it ideal for complex projects including mathematics, scientific research, and code generation.

- **[Qwen3 Coder 480B A35B Instruct](https://vast.ai/model/qwen3-coder-480b-a35b-instruct)**: This 480-billion-parameter Mixture-of-Experts (MoE) model developed by Alibaba is designed for advanced code generation and is optimized for actionable code output by operating exclusively in an efficient non-thinking mode.

- **[DeepSeek R1 0528](https://vast.ai/model/deepseek-r1-0528)**: This advanced reasoning language model developed by DeepSeek AI utilizes advanced chain-of-thought logic for complex mathematical and logical problems. This makes it a great choice for tool and API integrations.

- **[DeepSeek V3.1](https://vast.ai/model/deepseek-v3.1)**: This hybrid language model operates in both thinking and non-thinking modes. This dual-mode architecture is equally suited for deep reasoning with visible thought processes or generating fast responses without intermediate reasoning.

- **[GPT-OSS-120b](https://vast.ai/model/gpt-oss-120b)**: GPT-OSS-120b is an open-weight reasoning model from OpenAI designed for production use cases. With a range of low, medium, and high reasoning settings it balances speed and accuracy for agentic functions.

- **[GPT OSS 20b](https://vast.ai/model/gpt-oss-20b)**: Built for lower latency and specialized use cases. For everything from web browsing to function calling and API integration tasks, GPT OSS 20b is a great choice for agentic support.

- **[Llama 4 Maverick 17B 128E Instruct](https://vast.ai/model/llama-4-maverick-17b-128e-instruct)**: With seamless processing of both text and visual inputs, Llama 4 Maverick 17B supports up to a 10 million token context length and can process up to 5 input images simultaneously. It supports 12 languages for a wide array of agentic functions including English, Spanish, Tagalog, French, German, Hindi, Indonesian, Italian, Portuguese, Arabic, Thai, and Vietnamese.

- **[Llama 4 Scout 17B 16E Instruct](https://vast.ai/model/llama-4-scout-17b-16e-instruct)**: Released in April 2025, Scout 17B 16E Instruct employs a mixture-of-experts (MoE) and has a total of 109 billion parameters. It functions as a great assistant for analyzing data, parsing large quantities of data and is compatible with 12 languages including English, Spanish, Tagalog, French, German, Hindi, Indonesian, Italian, Portuguese, Arabic, Thai, and Vietnamese.

- **[Qwen3 VL 235B A22B Instruct](https://vast.ai/model/qwen3-vl-235b-a22b-instruct)**: The most powerful vision-language model in the Qwen series, VL 235B A22B Instruct combines 236 billion parameters across visual understanding, agent applications, context processing, and multimodal reasoning tasks. It excels at recognizing a range of items from landmarks, to plants and animals, to works of art and celebrities. This makes it a great research tool for imaging, robotics, and even technical documentation.

## The Best Computer Vision Models Available

When it comes to choosing an AI model that can interpret visual elements from real-world scenarios, these are among the best:

- **[DeepSeek OCR](https://vast.ai/model/deepseek-ocr)**: This vision-language model specializes in optical character recognition and document understanding. This makes it great for extracting text from images or scanned documents, processing PDFs, and much more.

- **[GLM 4.5V](https://vast.ai/model/glm-4.5v)**: Whether summarizing longform videos, analyzing data, or scanning documents for text, GLM 4.5V provides support for advanced reasoning tasks. With so many use cases, you can even toggle between quick responses or deeper analysis of data to make it work for you the way you want.

- **[InternVL3 78B](https://vast.ai/model/internvl3-78b)**: Whether analyzing technical documentation, interpreting results from medical images, or providing agentic support, InternVL3 78B has many use cases from text-only applications to a mixture of visual applications.

- **[Llama 4 Maverick 17B 128E Instruct](https://vast.ai/model/llama-4-maverick-17b-128e-instruct)**: Featuring 17 billion activated parameters distributed across 128 total experts, this model combines text and image understanding capabilities within a unified architecture. This model provides everything from an assistant-like conversational experience to image captioning, document analysis, and multilingual support.

- **[Llama 4 Scout 17B 16E Instruct](https://vast.ai/model/llama-4-scout-17b-16e-instruct)**: With efficiency being front and center, this model runs much more leanly than Maverick. This gives it a more conversational tone while still providing agentic support, robust analysis of charts, multilingual functionality.

- **[Qwen3 VL 235B A22B Instruct](https://vast.ai/model/qwen3-vl-235b-a22b-instruct)**: Qwen3's most powerful vision-language model, it provides incredible PC and mobile GUI navigation for automation tasks, visual coding support across a range of coding languages, and agentic support.

---

If you're ready to take your AI workflows to the next level, [Vast.ai](http://vast.ai) is here to help. From an [AI model library](https://vast.ai/model-library) with a host of tools and use cases to [GPU servers](https://cloud.vast.ai/) that can accelerate your research, we're here to help empower and scale your discovery every step of the way.

---


Explore Vast.AI's comprehensive model library featuring the best AI models across image generation, video production, audio, text, and computer vision.

Vast.AI's Model Library: Choosing from the Best AI Models Available


# Running ComfyUI on Vast.ai Serverless: Simple, Scalable Image Generation

ComfyUI has become one of the most popular open-source AI tools for image generation, for good reason.

Rather than typing a prompt and hoping for the best, you build the entire process yourself through a **node-based interface that gives you full, visual control** over how images are created. Every part of the workflow – from models, prompts, and parameters – can be adjusted or branched however you like, and workflows themselves are also shareable and reusable.

While ComfyUI runs well locally for smaller projects, it's a whole other challenge to use it for heavier custom workflows or at production scale. Complex ComfyUI graphs can exceed consumer GPU memory or demand more throughput than a single machine can handle – or simply take too long to iterate on locally.

That's where **Vast.ai Serverless** comes in.

### ComfyUI Meets Vast.ai Serverless

Vast.ai Serverless makes it easy to run your ComfyUI workflows at scale, thanks to a ready-to-use template that handles all of the setup for you – and no need to manage GPUs or monitor capacity.

Simply send requests to ComfyUI and have your generated assets automatically uploaded to S3-compatible storage. The template will return pre-signed URLs in response to requests, as well as detailed process updates while the workflow runs.

It's a straightforward way to integrate ComfyUI into your applications or pipelines without worrying about infrastructure overhead.

### What Comes in the ComfyUI Serverless Template

Our ComfyUI Serverless template provides [everything you need](https://www.google.com/url?q=https://docs.vast.ai/documentation/serverless/comfy-ui&sa=D&source=editors&ust=1766188232265630&usg=AOvVaw0k4v0rLcf_0MltEhly3_ZW) right out of the box:

- **ComfyUI** and its **API wrapper** (which manages asset uploads)
- **Stable Diffusion 1.5** for benchmarking
- **PyWorker** for processing JSON workflows
- **Provisioning script** for custom configuration

The template saves you from building or maintaining your own Docker image and ensures you get a consistent environment every time a worker spins up. Once the template is in place, several capabilities come built into the workflow by default.

#### 1. Built-In Asset Handling with S3 Storage

Generated assets are automatically uploaded to your private S3-compatible storage buckets, and pre-signed URLs provide secure, temporary access to download results directly. This keeps storage and retrieval simple, even as workloads scale.

Just add your credentials once in your account settings, and every worker can use them. If you need different buckets for different jobs, you can override these variables on a per-request basis.

#### 2. Flexible Configuration When You Need It

With ComfyUI Serverless, you can keep things simple or tailor your setup with optional configurations. For instance:

- Add a webhook to be notified after generation completion or failure
- Use a custom PyWorker git repository URL
- Adjust benchmark settings to better reflect your workload

With these configuration tools, you're free to adapt the template to your unique needs.

#### 3. Distributing Work Through Benchmarking

Each worker runs a quick benchmark when it initializes, to validate GPU performance and calculate a performance score. This score is what determines how requests are distributed across workers.

The template includes a default Stable Diffusion 1.5 benchmark for this purpose, but you can configure it to match your actual workload complexity or even supply your own custom benchmark.

### Try It Out Interactively Before Going Serverless

The ComfyUI Serverless template is designed for serverless execution, of course, but you can test it as an interactive instance first. Interactive testing helps confirm that everything works as expected before you start sending requests through Serverless.

You'll have to enable port forwarding to access applications over the public interface, but the benchmarking process will be visible in the instance logs either way.

### A Smoother Path to Production-Scale ComfyUI

Vast.ai Serverless and our ComfyUI template lets you go from a local workflow to a scalable, on-demand system with minimal setup. The environment is preconfigured – with customizable adjustments as needed – and assets are handled automatically while workloads are routed to the right GPUs based on real performance.

For creative work beyond images, we offer Serverless templates for [text-to-video](https://www.google.com/url?q=https://docs.vast.ai/documentation/serverless/comfyui-wan-2.2&sa=D&source=editors&ust=1766188232275066&usg=AOvVaw33TRQTBXX80tZ040XucSdk) and [text-to-music](https://www.google.com/url?q=https://docs.vast.ai/documentation/serverless/comfyui-acestep&sa=D&source=editors&ust=1766188232275242&usg=AOvVaw3Dnzq-bbQYJjA76I3vHcU3) workflows, as well.

Ready to see ComfyUI in action? Just launch the [ComfyUI Serverless template](https://www.google.com/url?q=https://cloud.vast.ai/?ref_id%3D62897%26creator_id%3D62897%26name%3DComfyUI%2520(Serverless)&sa=D&source=editors&ust=1766188232275749&usg=AOvVaw0clIXR2u6vdhxqRdqoPmHE) on Vast.ai and start running workflows at scale. It's that easy.


ComfyUI has become one of the most popular open-source AI tools for image generation, for good reason.

Running ComfyUI on Vast.ai Serverless: Simple, Scalable Image Generation


Vast.ai is thrilled to announce the launch of our Serverless offering for GPU workloads, enabling automatic, pay-per-use scaling for AI inference without hyperscaler pricing or fixed infrastructure constraints.

With [Vast.ai Serverless](https://vast.ai/products/serverless), users run inference workloads through a fully serverless API on Vast's globally distributed GPU cloud. That means no manual instance management and no capacity planning. Instead, the platform automatically applies predictive optimization and flexible scaling, with access to a diverse GPU fleet -- so teams can deploy production AI systems that scale when needed and yet stay cost-efficient at all times.

How does this serverless model work in practice? Let's take a closer look!

<div style={{ margin: "48px auto", maxWidth: "960px", width: "100%" }}>
  <div style={{ position: "relative", paddingBottom: "56.25%", height: 0, overflow: "hidden", borderRadius: "12px", background: "#000" }}>
    <iframe
      src="https://www.youtube-nocookie.com/embed/0PAPzSZa3tA?rel=0"
      title="Vast.ai Serverless overview"
      allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
      allowFullScreen
      referrerPolicy="strict-origin-when-cross-origin"
      loading="lazy"
      style={{ position: "absolute", top: 0, left: 0, width: "100%", height: "100%", border: 0 }}
    ></iframe>
  </div>
</div>

### Where GPU Cloud Meets Serverless AI

Vast.ai Serverless provides access to a wide range of GPUs, from RTX-class consumer GPUs to enterprise-grade machines like A100s, H100s, and B200s. Rather than managing individual instances, all you have to do is define performance targets, and Vast handles the rest.

Behind the scenes, Vast.ai continuously benchmarks GPUs across a global network of more than **17,000 GPUs hosted by 1,400+ providers in over 500 locations worldwide**. As workloads run, the platform dynamically selects, provisions, and routes jobs to the most efficient hardware available at that moment.

The key is predictive optimization.

### Predictive Optimization Built for Real-World Workloads

Most serverless platforms react to demand after it appears. Vast.ai Serverless goes further.

Our predictive optimization feature analyzes historical usage patterns, real-time load, and ongoing market benchmarking to anticipate demand before it peaks. Based on those signals, the platform **proactively provisions GPU workers** that balance cost and latency -- ready to activate on demand rather than scrambling to scale once performance has already degraded.

These reserve workers help avoid the laggy cold starts and unpredictable costs common with other GPU serverless offerings. Not to mention having to pay for excess GPU capacity that isn't actively being used.

### Multiple Workergroups per Endpoint

Unlike traditional serverless systems that restrict endpoints to a single hardware profile, Vast.ai Serverless [supports multiple Workergroups per Endpoint](https://docs.vast.ai/documentation/serverless/architecture). A Workergroup is a collection of GPU workers that are managed as a logical unit to automatically adjust capacity to meet demand.

In our Serverless offering, each Workergroup in an Endpoint can specify different GPU types or hardware configurations. This lets a single API endpoint serve workloads using whichever Workergroup is most cost-effective or the best fit for your performance targets. For example, lighter requests might route to consumer GPUs, while heavier inference jobs scale onto H100s -- with no manual intervention needed.

In practice, this makes it possible to optimize performance and cost-efficiency in real time within a single deployment. And that brings us to pricing.

### Lowest-Cost Transparent Serverless Pricing

Vast.ai Serverless is the **lowest-cost autoscaling GPU cloud** on the market today.

Our Serverless workloads are billed per second, with support for On-Demand, Interruptible, and Reserved pricing. There are [no tiers and no limits](https://docs.vast.ai/documentation/serverless/pricing) -- and all you need is $5 to get started.

Because Vast draws from a competitive global market of GPU providers, pricing reflects real supply and demand rather than preset SKUs. As demand scales up, Vast.ai Serverless automatically selects the most cost-effective GPUs available. When demand drops, resources are released immediately, and billing stops.

As a result, teams can run production workloads at a fraction of the cost compared to traditional providers with centralized infrastructure. However, lower costs don't mean lower standards. Security and compliance remain core to our platform.

### Enterprise-Ready Security and Compliance

Vast.ai is built with security and compliance as foundational principles. Our platform is backed by **SOC 2 Type II** [certification](https://vast.ai/article/vast-soc2-typeII-certification), with ongoing audits every 12 months to ensure continual coverage.

For customers with the highest security requirements, Vast.ai offers a tiered compliance structure with our [Secure Cloud](https://cloud.vast.ai/?secure_cloud=true) offering. In the Secure Cloud, workloads run on isolated instances with direct SSH, CLI, and API access, hosted by our vetted datacenter partners that meet **ISO 27001** standards at minimum. Many partners maintain additional certifications. (Full details are available on our [compliance page](https://vast.ai/compliance).)

Other enterprise security features -- including private VPN access, optional audit trails, and enterprise-grade compliance support -- can be enabled as needed.

In all cases, data sovereignty remains fully in your control. Models, data, and workloads persist only as long as you choose and can be deleted at any time.

### How to Automatically Scale GPU Compute for AI Inference

With Vast.ai Serverless, you can go from [zero to compute](https://docs.vast.ai/documentation/serverless) in seconds. In short, here's how the process works:

1. Bring your own machine learning model
2. Send an inference request
3. Vast.ai Serverless routes and provisions compute on ready GPUs
4. GPU executes and returns results
5. Serverless predictively scales up and down based on usage

That's all there is to it! From request to response, Vast.ai handles all of the heavy lifting. You also gain access to ample metrics and debugging tools, including logs and Jupyter/SSH.

### Get Started with Vast.ai Serverless

Our Serverless offering makes it easy to run AI inference without managing infrastructure or overpaying for capacity. Vast.ai Serverless gives you a simple and straightforward path from experiment to production -- and it's ready when you are.

Leverage Vast's distributed GPU fleet on your own terms. Check out our [Serverless Overview](https://docs.vast.ai/documentation/serverless) and start building today!


Announcing Vast.ai Serverless for GPU inference with predictive optimization, multi-workergroup endpoints, pay-per-second pricing, and enterprise-grade compliance.

Vast.ai Serverless: Automated GPU Scaling for AI Inference - Without the Overhead


All startups rely on infrastructure that can grow with them, but AI startups in particular face an added challenge: supporting compute-heavy workloads and iterating fast without slowing progress or overspending. 

Whether you're bootstrapping or venture-funded, it's always best to maximize the return on every GPU hour and dollar spent. But how do you do that? 

For AI startups, the right GPU infrastructure is absolutely critical. Training large models and running compute-heavy experiments can quickly consume budgets if you're not careful. Traditional cloud providers often charge a premium for the level of compute that AI workloads demand, while locking users into long-term contracts or inflexible pricing models. 

Buying your own dedicated hardware rarely solves the problem. The high upfront cost can be prohibitive for early-stage teams, and delays can stall progress. You also risk being tied to specific hardware that may be inadequate for your startup's next phase of development.

At Vast.ai, we take a different approach. Our cloud GPU rental platform offers high-performance machines with flexible spot pricing, so you can get the compute you need, when you need it, as affordably as possible.

Let's take a closer look at what makes Vast.ai's [spot GPU rental](https://cloud.vast.ai/) an ideal choice for AI startups.

## The Cost Advantage of Spot GPUs for AI Startups

To keep cloud costs under control as an AI startup, it's important to carefully consider which [pricing model](https://docs.vast.ai/documentation/instances/choosing/instance-types) is right for you. Vast.ai offers on-demand GPU instances as well as spot instances – each balancing cost and reliability in different ways.

**On-demand instances** offer guaranteed uptime at a fixed price, so you pay a slightly higher rate to access a GPU without the risk of sudden interruption. This option is well suited for continuous training jobs or production inference, or basically any workload where you need to prioritize stability over cost.

**Spot instances**, also known as interruptible instances on [Vast.ai](https://vast.ai), let you rent unused GPU capacity at lower rates determined by bidding. The highest bid wins the instance. The trade-off is the possibility that your instance may be interrupted or preempted with little to no notice, requiring you to restart the job later (when you regain priority) or move it to another GPU. This option is nevertheless ideal for workloads that can tolerate occasional interruptions in exchange for significant cost savings. 

For AI startups, the main draw of spot instances is simple: they deliver the same high-performance GPUs at a drastic discount. On Vast.ai, interruptible instances can be **up to 80% cheaper** than traditional cloud rates.

These savings can give early-stage AI startups the momentum to scale faster. The next move is scaling smarter.

## How Spot GPUs Help AI Startups Scale Smarter

As a startup, you want to accelerate your AI development thoughtfully and efficiently. Lowering your cloud GPU costs is the first step, and then you can reinvest those savings to accelerate growth.

With spot pricing, access to high-performance GPUs remains cost-effective. As a result, larger-scale training becomes affordable enough for early-stage AI startups to run more jobs and test new ideas efficiently – ultimately iterating faster. 

Additionally, because spot instances are billed on a pay-as-you-go basis, you can run workloads on cutting-edge GPUs that may otherwise be out of reach – like the H200 – for limited periods as needed, and then scale back down quickly.

Lower costs and operational agility translate into a competitive edge that can make all the difference for an early-stage startup. 

From there, how can you make spot instances work best for you? Here are a few quick tips:

**Use spot GPUs for flexible, restartable jobs.** Batch data processing and any experiments or workloads that can resume from checkpoints are excellent options when you're using interruptible instances on Vast.ai. 

**Make sure your workflows are tolerant of interruptions, and if you can, automate recovery.** With spot or interruptible instances, disruptions will almost certainly happen on occasion, and you'll want to be ready for that.

**Switch to on-demand instances for critical workloads.** Live inference and customer-facing applications are good candidates for on-demand. Anytime you want to minimize productivity loss from an interruption (for example, when a member of your team is actively debugging or deploying code), guaranteed uptime may be worth the higher cost.

## The Bottom Line

Interruptible instances on Vast give AI startups a cost-effective way to iterate quickly and scale smarter. By delivering the same high-performance hardware at a fraction of the cost, spot GPUs help extend your runway and free up budget to reinvest in growth. 

To learn more about why startups of all stripes choose Vast.ai, check out our previous post about how our platform is tailored to meet your needs from day one.

Ready to put the Vast.ai advantage to work for you? Explore our [spot GPU options](https://cloud.vast.ai/) and see how much faster – and smarter – your AI startup can scale!


For AI startups working with limited budgets and evolving product requirements, spot (interruptible) GPUs offer a flexible, cost-effective path to building and scaling AI infrastructure without breaking the bank.

Starting Smart: Why Spot GPUs Are Ideal for AI Startups


When you're training a custom AI model, access to the right GPU compute resources is crucial. That can be a tall order – but you don't have to drain your budget or get locked into long-term contracts to get the compute power you need. With **Vast.ai**, there's a better way.

Our cloud GPU rental platform gives you direct access to high-performance GPUs through a transparent, pay-as-you-go model that lets you train on your own terms: anytime, anywhere, and at the right price.

In this post, we'll take a closer look at how Vast.ai helps you train custom AI models faster and more affordably, without compromising performance or scalability.

## 1. Flexible, Budget-Friendly Pricing

Training AI models can get expensive. Vast.ai offers different [pricing options](https://docs.vast.ai/instances/rental-types) to help you keep cloud costs under control:

**Interruptible instances** let you rent GPUs at lower prices through interruptible or bid-based access. While these instances can be interrupted with little notice, they're a great fit for workloads that can handle brief disruptions in exchange for major cost savings – **up to 80% cheaper** than traditional cloud rates. For many users, that tradeoff is worth it for the reduction in training costs.

**On-demand instances** give you uninterrupted GPU access at a fixed rate. This setup is best for continuous training, real-time inference, or workloads where stability is crucial.

Of course, price is just one part of the equation: the next step is choosing the right GPU for the job. Vast.ai gives you that flexibility, as well.

## 2. GPU Options for Every AI Workload

There's no one-size-fits-all solution when you're training custom AI models. That's why our wide range of cloud GPU options is such an advantage. You can choose exactly what your workload needs, whether that's an enterprise-grade accelerator like the **NVIDIA [A100](https://vast.ai/article/H100-vs-A100-Comparing-two-Powerhouse-GPUs)** or **[H200](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights)**, an ultra-high-performance machine like the **[B200](https://vast.ai/article/nvidia-h200-vs-b200-comparing-datacenter-grade-accelerators)**, or a top-tier consumer card like the **[RTX 4090](https://vast.ai/article/nvidia-rtx-4090-vs-a100-two-powerhouses-two-purposes)**, **[5090](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know)**, or **[L40S](https://vast.ai/article/comparing-nvidia-l40-vs-l40s-and-more)**.

Each listing on Vast.ai displays **transparent instance specs**, making it easy to compare GPUs by type, RAM, storage, location, and more. Simply filter and select the best fit for you!

Most importantly, you'll never have to worry about overpaying for capacity you don't need or being held back by hardware limitations. When your workloads grow, scaling up is seamless. Vast.ai lets you spin up multiple GPU instances and customizable, on-demand [GPU/CPU clusters](https://vast.ai/clusters), so you can expand training capacity instantly – and only pay for what you use.

## 3. Supporting Real-World Use Cases

Our platform is ideal for anyone training, fine-tuning, or experimenting with custom AI models. Whether you're an AI researcher or developer, an enterprise team, or a [startup](https://vast.ai/article/7-reason-why-startups-uses-vast-ai), Vast.ai provides the compute power you need to train models for use cases like automating customer inquiries, identifying sales trends and other business insights, and analyzing massive research datasets.

You can build and refine domain-specific models, fine-tune foundation models like Llama and Mistral on proprietary data, and retrain as your datasets evolve. With support for LLMs and diffusion models, our platform is **compatible with all major machine learning (ML) frameworks** – and you can even run custom containers and tooling.

## 4. Quick Start with Pre-Built Templates

You don't have to start from a blank slate. Vast.ai offers pre-built [templates](https://docs.vast.ai/templates) for popular frameworks – **PyTorch, TensorFlow, vLLM, ComfyUI, and more** – that are designed to help you launch a training environment in minutes.

Each template includes the dependencies and configurations you need for model training or fine-tuning, so you can spend less time setting up and more time improving your results. Customize templates with your own datasets, scripts, or containers. You have complete flexibility to tailor them to your project needs!

Finally, Vast.ai also ensures that you can train your AI models in a trusted environment [with control over data exposure](https://vast.ai/article/running-private-ai-models-without-the-risk-of-data-exposure).

## 5. Enterprise-Grade Security

With Vast.ai, your workloads run on a secure, trusted infrastructure designed to [protect your data](https://vast.ai/compliance) – crucial when training custom AI models. Choose either **verified hosts** with strong reliability metrics or **vetted datacenter partners** who maintain third-party compliance certifications and are bound by comprehensive Data Processing Agreements (DPAs) with Vast.ai.

Our **Secure Cloud** offering gives you the option to select GPU infrastructure only from our vetted datacenter partners. Each of these providers is located in a certified facility that's current with ISO 27001 and/or Tier 3/4 standards at minimum.

We've also [completed](https://vast.ai/article/vast-soc2-typeII-certification) our own SOC 2 Type II certification, affirming our adherence to industry-leading compliance frameworks.

## Get Started with Vast.ai

Training custom AI models shouldn't be limited to those with deep pockets or powerful infrastructure. **Vast.ai** levels the playing field.

With high-performance GPUs, flexible pricing, and pre-configured templates to help you get started, you can move from idea to implementation faster and more affordably than ever with Vast.ai. You'll also enjoy peace of mind knowing that your data and models are protected by enterprise-grade security every step of the way.

Ready to train your own model? Spin up your first instance today at Vast.ai!


When you're training a custom AI model, access to the right GPU compute resources is crucial. Vast.ai gives you direct access to high-performance GPUs through a transparent, pay-as-you-go model that lets you train on your own terms.

Why Choose Vast.ai to Train Custom AI Models


We're excited to announce a new deployment guide for MiniMax-M2, the breakthrough 230 billion parameter language model that's making waves in the open-source AI community. If you've been looking for a cost-effective way to run state-of-the-art LLMs, this guide is for you.

## What Makes MiniMax-M2 Special?

MiniMax-M2 achieves the #1 composite score among open-source models while maintaining incredible efficiency. Despite its 230B parameter size, it activates only 10B parameters per inference, delivering fast responses without the typical computational overhead of large models.

Key features include:
- **Interleaved Thinking**: Transparent reasoning via `<think>...</think>` tags
- **131K Token Context**: Handle large documents and complex conversations
- **OpenAI Compatible**: Drop-in replacement for existing applications
- **MIT Licensed**: No restrictions on commercial use

## What's in the New Guide

Our latest documentation example walks you through deploying MiniMax-M2 on Vast.ai from start to finish. The guide includes:

**Complete Deployment Instructions**
- Hardware requirements and instance selection
- Step-by-step provisioning with the Vast.ai CLI
- API key authentication for secure deployments
- Configuration parameters validated through real deployments
- Expected timelines and what to watch for

**API Integration Examples**
- Python code using the OpenAI SDK with authentication
- cURL examples for quick testing
- Streaming response implementations
- Best practices for multi-turn conversations

**Troubleshooting Section**
Based on real deployment testing, we've documented the four critical issues you might encounter and their solutions:
- Docker image version requirements
- Disk space allocation
- GPU memory optimization
- CUDA driver compatibility

**Performance Data**
All metrics in the guide come from actual deployments on Vast.ai infrastructure:
- 30-minute initial deployment time
- ~7 seconds for 100-token responses
- 131K token context window
- Production-ready performance on 4x H100 GPUs

## Why Deploy on Vast.ai?

Vast.ai's GPU marketplace gives you access to enterprise hardware at competitive rates, making it economical to run powerful models like MiniMax-M2 for production workloads. Our guide uses a 4x H100 (80GB) configuration with the vLLM nightly build. You get:

- Enterprise GPU access (H100 recommended for best compatibility)
- Predictable costs with no usage-based pricing
- Full control over your infrastructure
- OpenAI-compatible API for easy integration

The guide shows you how to leverage these advantages for cost-effective LLM inference that rivals cloud API quality.

## Scaling for Production

While the 4x H100 configuration demonstrated in our guide provides an excellent starting point for deploying and testing MiniMax-M2 on Vast.ai, production deployments typically require larger GPU configurations to support longer context lengths and higher concurrent request volumes. For production use cases, consider configurations such as 8x H100, 4x H200, or 8x H200, which provide substantially more GPU memory for handling concurrent requests and extended context windows.

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Developers** building AI-powered applications who want cost control and flexibility
- **Research teams** running experiments that need high-volume inference
- **Startups** prototyping AI features without committing to expensive cloud APIs
- **Anyone** curious about deploying open-source LLMs at scale

## Get Started

The complete guide is now available in our documentation:

**[Read: Running MiniMax-M2 on Vast.ai →](https://docs.vast.ai/examples/text-generation/minimax-m2#minimax-m2)**

Whether you're exploring open-source AI options or ready to deploy your first large language model, this guide provides everything you need to get MiniMax-M2 running on Vast.ai infrastructure.

Ready to try it? [Sign up for Vast.ai](https://cloud.vast.ai) and follow the guide to deploy your first instance.

---

*All performance metrics and cost estimates in this article are based on actual deployment testing on Vast.ai infrastructure. Results may vary based on instance availability and configuration.*


Learn how to deploy MiniMax-M2, the 230B parameter language model with state-of-the-art performance, on Vast.ai infrastructure.

New Guide: Deploy MiniMax-M2 on Vast.ai

	
Choosing an RTX 6000 GPU isn't quite as straightforward as it might seem. The name "RTX 6000" now spans multiple cards across two architectures – the NVIDIA RTX 6000 Ada, as well as three newer Blackwell-based variants built for very different workloads. The four GPUs share some impressive fundamentals, but their performance profiles and a few important design differences set them apart.

Today we're looking at these NVIDIA RTX 6000 GPU options, what they're built for, and how to decide which one best fits your needs. So let's get into it!

### The RTX 6000 Family at a Glance

For starters, here's a brief overview of the four GPU models in the RTX 6000 lineup:

-   [RTX 6000 Ada Generation](https://cloud.vast.ai/?gpuModelNames=rtx6000AdaGeneration&instanceDiskSizeMin=700&instanceDurationMin=604800&instanceType=onDemand&isAvxSupported=false&isHostSecure=false&isMachineIpStatic=false&isOfferAvailable=true&isOfferCompatible=true&isOfferVerified=true&isQueryInverted=false&machineReliabilityMin=0.9&sorts=score-desc) – An enthusiast-class professional graphics card with solid all-around performance.

-   [RTX PRO 6000 Blackwell Workstation Edition](https://cloud.vast.ai/?gpuModelNames=rtxPro6000BlackwellWorkstation&instanceDiskSizeMin=700&instanceDurationMin=604800&instanceType=onDemand&isAvxSupported=false&isHostSecure=false&isMachineIpStatic=false&isOfferAvailable=true&isOfferCompatible=true&isOfferVerified=true&isQueryInverted=false&machineReliabilityMin=0.9&sorts=score-desc) – A full-power 600W desktop card, best used for single-GPU high-throughput workflows.

-   **RTX PRO 6000 Blackwell Max-Q Workstation Edition** – Basically the same hardware as the standard Workstation Edition above, but with a few design differences that make it better suited for multi-GPU setups.

-   [RTX PRO 6000 Blackwell Server Edition](https://cloud.vast.ai/?gpuModelNames=rtxPro6000BlackwellServer&instanceDiskSizeMin=700&instanceDurationMin=604800&instanceType=onDemand&isAvxSupported=false&isHostSecure=false&isMachineIpStatic=false&isOfferAvailable=true&isOfferCompatible=true&isOfferVerified=true&isQueryInverted=false&machineReliabilityMin=0.9&sorts=score-desc) – A passive-cooled variant designed for rackmount systems and high-density environments.

Next, we'll break down each GPU in more detail, then wrap up with some benchmarks and a side-by-side comparison table to help guide your decision.

### RTX 6000 Ada Generation: Solid and Dependable

The RTX 6000 Ada is the previous-generation flagship graphics card, and it still holds its own as a solid, versatile workstation GPU. Built on the Ada Lovelace architecture, it delivers strong performance for enhanced content creation and light-to-moderate AI training.

Overall, it's a well-rounded option. Featuring 48GB of GDDR6 ECC memory, a respectable 18,176 CUDA cores, up to 91.1 TFLOPS of single-precision performance, and 300W TDP, the RTX 6000 Ada can handle complex tasks and multi-application workflows without the more extreme price or power demands of a Blackwell-class GPU. It remains dependable for many professional use cases, especially if your projects don't require cutting-edge AI acceleration.

The RTX 6000 Ada Generation may be the right choice for you if:

Your workloads span creative applications, 3D tools, and mixed GPU tasks.

You don't need 96GB of VRAM but still want professional-grade stability and performance.

You're looking for a more cost-effective alternative to Blackwell for workstation use.

### RTX PRO 6000 Blackwell Workstation: Top-Tier Desktop Performance

The RTX PRO 6000 Blackwell Workstation Edition is the top-performing desktop card in this group. Built on NVIDIA's Blackwell architecture, it boasts 96GB of GDDR7 memory, a wide 512-bit memory bus, and extremely high memory bandwidth at nearly 1.8 TB/s. Plus, it has a staggering 125 TFLOPS of single-precision performance. This combination makes it well suited for the most demanding AI workloads as well as complex visualization, simulation, and neural rendering tasks.

The Workstation Edition can sustain peak performance under a 600W power load, without thermal throttling – but at the same time, it's not recommended for dense multi-GPU configurations due to its higher heat output and flow-through dual-fan design. Instead, it works best as a single GPU in a workstation setup.

The RTX PRO 6000 Blackwell Workstation Edition makes sense for you if:

You want the highest sustained throughput the RTX PRO 6000 series can deliver in a workstation, for the most demanding workloads.

You don't need to run multiple GPUs in the same system.

Your workstation has the power and cooling to support a 600W GPU.

### RTX PRO 6000 Blackwell Max-Q Workstation: Multi-GPU Friendly

RTX PRO 6000 Blackwell Max-Q Workstation Edition is ideal for anyone who wants Blackwell-level performance in a cooler, more efficient card that works well in multi-GPU configurations. For simplicity's sake, let's call it the Max-Q variant as opposed to the standard Workstation Edition.

The Max-Q features the same basic hardware as the full-power standard Workstation Edition – including 96GB of GDDR7 memory, a 512-bit bus, and 1.8 TB/s memory bandwidth – but it runs at a reduced 300W TDP and uses a blower-style cooler. Its enclosed single-fan design pulls air in and then exhausts it out the rear of the chassis. With lower heat output and superior management of airflow, the Max-Q is a much better fit for multi-GPU configurations than the standard Workstation Edition.

You may want to go with the Max-Q Workstation model if:

You're looking for Blackwell-class performance and memory capacity in a cooler, more efficient card.

You can't support a 600W GPU but still need 96GB of VRAM and high bandwidth.

You plan to run 2-4 GPUs and your workloads benefit from multi-GPU scaling.

### RTX PRO 6000 Blackwell Server Edition: Scalable and Server-Ready

The RTX PRO 6000 Blackwell Server Edition is built specifically for datacenter and rackmount environments. It lacks an active cooling fan, instead relying entirely on external airflow with a passive heatsink. It can also be configured up to the full 600W power envelope – as long as your airflow solution can handle the cooling required, of course!

Like the other Blackwell variants, the Server Edition includes 96GB of GDDR7 memory and a 512-bit bus. Its memory bandwidth is slightly lower at 1.6 TB/s, compared to 1.8 TB/s on the workstation cards, but the Server Edition is still more than capable of handling large models, massive datasets, and multi-GPU workloads. What really sets it apart is its ability to deliver stable, scalable performance in high-density environments.

Choose the RTX PRO 6000 Blackwell Server Edition if:

You need stable performance for continuous workloads across large-scale multi-GPU setups.

You want Blackwell-level memory and bandwidth in a form factor designed for dense compute.

You're deploying into rackmount servers where the chassis provides adequate high-pressure airflow.

### RTX 6000 Series: Performance Benchmarks

Now that we've covered design differences and use cases, let's check out how these GPUs actually perform. The benchmarks below show how each RTX 6000 variant handles real workloads, so you can get a clearer picture of what to expect in practice.

![RTX 6000 benchmark comparison](/uploads/vast_blog_6000_comparison_benchmarks.png)

And finally, for reference, here's a side-by-side specs comparison table showing how the four GPUs stack up against each other:

<table class="w-full border-collapse border border-gray-300">
  <thead>
    <tr class="bg-gray-100">
      <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
        Spec
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX 6000 Ada
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX PRO 6000 Blackwell Workstation
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX PRO 6000 Max-Q
      </th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
        RTX PRO Blackwell Server Edition
      </th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Architecture
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Ada Lovelace</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Blackwell</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Blackwell</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Blackwell</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        VRAM
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">48GB GDDR6</td>
      <td class="border border-gray-300 px-4 py-2 text-center">96GB GDDR7</td>
      <td class="border border-gray-300 px-4 py-2 text-center">96GB GDDR7</td>
      <td class="border border-gray-300 px-4 py-2 text-center">96 GB GDDR7</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        ECC
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Yes</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Memory Bandwidth
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">960 GB/s</td>
      <td class="border border-gray-300 px-4 py-2 text-center">1,792 GB/s</td>
      <td class="border border-gray-300 px-4 py-2 text-center">1,792 GB/s</td>
      <td class="border border-gray-300 px-4 py-2 text-center">1,597 GB/s</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Memory Interface
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">384-bit</td>
      <td class="border border-gray-300 px-4 py-2 text-center">512-bit</td>
      <td class="border border-gray-300 px-4 py-2 text-center">512-bit</td>
      <td class="border border-gray-300 px-4 py-2 text-center">512-bit</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        CUDA Cores
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">18,176</td>
      <td class="border border-gray-300 px-4 py-2 text-center">24,064</td>
      <td class="border border-gray-300 px-4 py-2 text-center">24,064</td>
      <td class="border border-gray-300 px-4 py-2 text-center">24,064</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Tensor Cores
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">568 (4th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">752 (5th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">752 (5th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">752 (5th gen)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Ray Tracing Cores
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">142 (3rd gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">188 (4th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">188 (4th gen)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">188 (4th gen)</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Single-Precision Performance
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">91.1 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">125 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">110 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">120 TFLOPS (FP32)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        RT Core Performance
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">210.6 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">380 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">333 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">355 TFLOPS</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Total Board Power
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">300W</td>
      <td class="border border-gray-300 px-4 py-2 text-center">600W</td>
      <td class="border border-gray-300 px-4 py-2 text-center">300W (power-optimized)</td>
      <td class="border border-gray-300 px-4 py-2 text-center">400–600W (configurable)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Form Factor
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">4.4" H x 10.5" L, dual slot</td>
      <td class="border border-gray-300 px-4 py-2 text-center">5.4" H × 12" L, dual-slot, extended height</td>
      <td class="border border-gray-300 px-4 py-2 text-center">4.4" H × 10.5" L, dual-slot</td>
      <td class="border border-gray-300 px-4 py-2 text-center">4.4" H × 10.5" L, dual-slot</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Thermal (Fan Configuration)
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Single-fan active</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Dual-fan active</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Single-fan active</td>
      <td class="border border-gray-300 px-4 py-2 text-center">No-fan passive (server chassis)</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">
        Date Released
      </td>
      <td class="border border-gray-300 px-4 py-2 text-center">Dec. 2022</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Mar. 2025</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Mar. 2025</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Mar. 2025</td>
    </tr>
  </tbody>
</table>

### Final Thoughts

Each RTX 6000 variant brings something different to the table. As we explored above, the right choice ultimately comes down to how you plan to use it.

To sum things up, the Ada model is solid and dependable for a broad range of workloads, while the full-power Blackwell Workstation card offers the strongest single-GPU performance. The Max-Q variant is the more practical choice for lower-power or multi-GPU setups, and the Server Edition is designed for dense deployments and continuous workloads.

At the same time, these GPUs aren't exactly budget-friendly. That's where Vast.ai comes in. Our cloud GPU rental platform allows you to access this level of performance on demand, eliminating the upfront cost of purchasing the hardware yourself.

With Vast, you can easily match your compute to your workload and only pay for what you use. Best of all, you'll save 5-6X on average compared to traditional cloud providers.

Ready to see what Vast.ai can do for you? Spin up an RTX 6000 GPU instance today, or explore other powerful GPU options like the A100, H200, and B200 – available right now!


A practical guide to every RTX 6000 option, from Ada to the newest Blackwell workstation and server GPUs.

Which NVIDIA RTX 6000 Is Right for You?


When you think of a large language model (LLM), you probably picture the big household names like ChatGPT, Claude, Gemini, and Llama. They're dominant players for a reason – delivering cutting-edge performance that sets the standard for general-purpose AI – but they're also overwhelmingly centered around the English language and Western culture.

If you look further, there are plenty of other models gaining traction globally. Fueled by government investment as well as tech giants, startups, and research labs, these regional LLMs are trained on local languages and optimized for diverse needs.

With that in mind, let's take a look at some examples from around the world.

### Middle East: Arabic-First AI

Across the Middle East, government partnerships and large-scale investments are driving the rapid expansion of the region's AI sector. Much of this effort is focused on Arabic-first and bilingual LLMs designed for regional sovereignty and cultural awareness. For instance, the following are some of the leading projects in the region:

*   Jais offers open bilingual models up to 30B parameters, trained on large Arabic-English datasets, with the ability to switch between Arabic dialects. Instruction-tuned "Chat" variants, optimized for conversational applications, have also been released.

*   ALLaM powers Saudi Arabia's popular HUMAIN Chat conversational assistant and emphasizes Arabic fluency and cultural nuance. Its series of models is designed to support the ecosystem of Arabic Language Technologies (ALT).

*   Falcon, Abu Dhabi's flagship open LLM series, ranges from 7B to 180B parameters and combines commercially permissive licensing with strong performance that prioritizes resource efficiency – making it popular for academic, commercial, and government uses.

### Asia-Pacific: Scale, Localization, and Linguistic Diversity

The AI ecosystem in the Asia-Pacific region spans the largest and most linguistically diverse populations in the world. Both government-backed initiatives and major tech firms are creating models that serve dozens of languages and dialects while delivering performance that in many cases stands on par with the most advanced models worldwide.

*   [DeepSeek](https://vast.ai/article/serving-deepseek-models-on-vast-ai-with-vllm-and-langchain), one of the most capable open-source models to emerge from China, uses a Mixture-of-Experts (MoE) architecture to balance accuracy and computational efficiency. It offers more fluent support for Asian languages, with cultural nuance that may be missed by Western-centric models.

*   [Qwen](https://vast.ai/article/alibabas-qwen-an-open-source-ai-model-that-surpasses-deepseek), developed by Alibaba, covers 119 languages and dialects, with specialized multimodal variants and built-in support for Model Context Protocol. It features hybrid "thinking" and "non-thinking" modes for flexible reasoning – and you can even set a "thinking budget" to control how much computational effort it devotes to reasoning versus how quickly it replies.

*   tzusumi is a lightweight Japanese LLM (0.6B and 7B parameters) designed for on-prem use, offering strong Japanese-language processing and multimodal support with minimal hardware requirements.

*   Solar Pro 2 offers frontier performance in a comparatively compact 31B model, with multilingual capabilities and standout fluency in Korean. It's designed for real-world agent-like workflows across domains.

*   SeaLLM is a multilingual model built for Southeast Asia that outperforms most open-source LLMs across domains like science, physics, and economics in the languages of the region. Notably, it supports non-Latin scripts and low-resource languages like Lao and Khmer.

### Europe: Responsible AI and Data Sovereignty

In Europe, AI development tends to emphasize data sovereignty and compliance. The region has produced a number of LLMs that serve multilingual users and align with the EU AI Act and GDPR while competing with U.S. and Asian counterparts in raw performance.

*   Mistral, developed in France, offers a suite of open and proprietary models for reasoning, multimodal, coding, and other applications, including its flagship Mistral Large 2 and the open-source Devstral family – and even a multilingual chatbot called Le Chat that boasts deep research and image editing capabilities.

*   Luminous by Aleph Alpha is a multilingual model family (13B–70B parameters) built for enterprise and sovereign-AI use. It emphasizes compliance and transparency and provides support for English, German, French, Italian, and Spanish. The company also offers Pharia, a newer generation of models designed for concise, length-controlled responses.

*   BLOOM is a 176 billion-parameter open-access model produced through a year-long volunteer research effort to provide a transparent alternative to proprietary AI models. It can generate text in 46 natural languages and 13 programming languages.

### Other Regions: Emerging Innovation

Outside the major AI centers, new regional efforts are gaining momentum. For instance, Latam-GPT is an open initiative by Chile's nonprofit CENIA to develop a collaborative AI model completely within and for Latin America – trained on regional data and languages. The project's goal is to advance technological independence and cultural representation across the region.

In Africa, InkubaLM (named after the IsiZulu term for a dung beetle) addresses the need for lightweight African-language models. Trained on English and French as well as five African languages – Hausa, Swahili, IsiXhosa, IsiZulu, and Yoruba – the compact 0.4B-parameter model delivers efficient natural-language capabilities without heavy compute demands.

These projects reflect how emerging regional players are similarly building language models that prioritize inclusion and linguistic diversity, broadening the reach of global AI innovation.

### How to Choose a Regional LLM

Choosing a regional model depends on your goals and infrastructure. Here are a few considerations to keep in mind:

*   Look for open-weight models if you're prioritizing sovereignty or on-prem use.

*   Pick the right model size for your infrastructure! Many regional LLMs offer compact variants.

*   Consider dialect coverage and code-switching capabilities (e.g., training on colloquialisms) – along with nuanced cultural awareness – for social and chat applications.

*   If your use case requires meeting strict data protection and compliance standards, EU-built models developed with these priorities in mind may be ideal.

Once you've found the right fit, the next step is putting these models to work!

### The Bottom Line

Regional LLMs reflect how AI is evolving across a global spectrum of languages, cultural norms, regulatory priorities, and specialized technical needs.

Ultimately, they represent a more inclusive approach to model development. In turn, this helps bridge the AI gap for underrepresented groups worldwide. For developers and organizations, these LLMs provide context-aware and culturally fluent performance that's both versatile and cost-efficient.

With platforms like [Vast.ai](https://vast.ai), you don't need massive in-house infrastructure to take full advantage of these models. Our affordable cloud GPU rental makes it easy to train, fine-tune, and deploy regional LLMs at scale – helping you find just the right fit for your use case and bring AI closer to the communities and markets you serve.

> **Start experimenting with regional LLMs on [Vast.ai](https://vast.ai) today!**


From Jais to Qwen, Falcon to BLOOM - a look at how regional large language models are reshaping AI across languages, cultures, and continents.

Global Growth of Regional Large Language Models (LLMs)


The world’s smallest desktop AI supercomputer is now available in the form of the NVIDIA DGX™ Spark. Originally [announced in January 2025](https://www.theverge.com/news/798775/nvidia-spark-personal-ai-supercomputer) under the name Digits, NVIDIA CEO Jensen Huang praised Spark as a tool for data scientists, AI researchers, and students “to engage and shape the age of AI.”

Powered by a 240 watt power supply that uses a standard wall outlet and with a golden asteroid-like textured casing, it looks more like a high-end PC than an AI supercomputer. Only an inch wider and longer than the M4 Mac Mini, Spark measures just 6” x 6” x 2” and weighs just 2.6 pounds. This compact size makes it all the more impressive that it can fuel enterprise AI development with ease. If you’re looking for a mini AI companion to run local AI models, the NVIDIA DGX Spark is the best-looking, most-talked-about option on the market. But does it deliver on the hype?

### **Inside the NVIDIA DGX Spark**

Spark runs on a 20-core ARM64 CPU and a NVIDIA GB10 [Grace Blackwell](https://vast.ai/article/report-nvidia-developing-new-blackwell-based-ai-chip-for-china) GPU. It’s capable of handling AI models with up to 200 billion parameters at a million billion calculations per second. Its 128GB of unified memory can be seamlessly allocated to both its CPU and GPU without static partitioning. Plus, its stock 4TB NVME.M2 SSD hard drive provides more storage than many of its closest competitors.  
With four USB-C ports and an HDMI port, you can plug it into an array of peripheries, monitors, or even external GPUs. You can also connect two Sparks together and run them simultaneously to double your performance.

#### NVIDIA DGX Spark Technical Specifications

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Specification
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                NVIDIA DGX Spark
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2">
                NVIDIA Grace Blackwell
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">GPU</td>
            <td class="border border-gray-300 px-4 py-2">
                Blackwell Architecture
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">CPU</td>
            <td class="border border-gray-300 px-4 py-2">
                20 core Arm, 10 Cortex-X925 + 10 Cortex-A725 Arm
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                CUDA Cores
            </td>
            <td class="border border-gray-300 px-4 py-2">
                Blackwell Generation
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Performance<sup>1</sup>
            </td>
            <td class="border border-gray-300 px-4 py-2">1 PFLOP</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                System Memory
            </td>
            <td class="border border-gray-300 px-4 py-2">
                128 GB LPDDR5x, coherent unified system memory
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">OS</td>
            <td class="border border-gray-300 px-4 py-2">NVIDIA DGX™ OS</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                System Dimensions
            </td>
            <td class="border border-gray-300 px-4 py-2">
                150 mm L x 150 mm W x 50.5 mm H
            </td>
        </tr>
    </tbody>
</table>

See full specs at [NVIDIA.com](https://www.nvidia.com/en-us/products/workstations/dgx-spark/).

### **Should You Buy the NVIDIA DGX Spark?**

Following a delayed launch, Spark was released earlier this month. It is [available on](https://www.nvidia.com/en-us/products/workstations/dgx-spark/) [NVIDIA.com](http://NVIDIA.co) and other retailers for $3,999. However, Spark was initially slated to retail for only $2,999 — the price of the similarly spec’d ASUS Ascent GX10. Though the Ascent comes with only a 1TB hard drive and is far less inspired design-wise. Beyond its delayed release and initial sticker shock, early reviews have been mixed.

Many early adopters praise Spark’s ability to run larger AI models and champion the GP10’s ability to outperform many PC GPUs. Additionally, remote access through tools like Tailscale have made Spark even more accessible. Using it as a companion to developers’ existing PCs and other devices, you can work with your Spark from anywhere on earth.

However, Spark is being criticized for not reaching its listed capabilities. Though rated for 240 watts, noted programmer and video game developer [John Carmack reported](https://www.pcgamer.com/hardware/graphics-cards/og-doom-legend-john-carmack-says-his-nvidia-dgx-spark-box-has-thermal-throttling-issues-raising-doubts-over-the-viability-of-nvidias-ai-apu-for-laptop-pcs/) less than half of that. As a result, he suggested it only delivers about half of its advertised performance. With wide reports of latency, throttling, and reboot issues, some shoppers have voiced reservations about buying it for more data-intensive projects.

For now, Spark’s full range of ideal use cases have yet to be explored and its ecosystem is still evolving. Recent [news that open source platforms](https://blogs.nvidia.com/blog/open-source-aerial-ai-native-6g/) like NVIDIA Sionna Research Kit and Aerial Testbed are available for the DGX network could provide Spark with native tools to accelerate its capabilities. It has shined brightest for agentic AI and reasoning models. For a mini AI supercomputer, it does a lot — it may just not be enough for everyone.

### **Bringing Powerful Compute Within Reach**

Spark's steep price tag makes it an expense that may be hard to justify. While on-premise, local AI development sounds great, Spark may be too niche for wide adoption at its price point. But the tiny supercomputer does nonetheless reflect a push toward making advanced AI computing more widely available.

Here at Vast.ai, we share that goal of making AI development accessible. Our mission is to democratize AI by making powerful GPUs available to anyone, anywhere — without the upfront cost of purchasing and maintaining costly hardware. With our cloud GPU rental platform, you can leverage machines that exceed DGX Spark with affordable, on-demand compute power designed to scale with your needs.

After all, democratizing AI starts with access. On Vast, you'll save 5–6X compared to traditional cloud providers. Find your compute and [get started](https://vast.ai/pricing) today\!


Explore the NVIDIA DGX Spark — the world’s smallest desktop AI supercomputer — its specs, performance, reviews, and how Vast.ai offers a more scalable alternative.

The NVIDIA DGX Spark: The New Desktop AI Supercomputer That Fits in Your Hand


What if your Integrated Development Environment (IDE) could talk back to you, suggest solutions, and write half your code for you? That's essentially what vibe coding does.

Coined earlier this year, "vibe coding" is an interactive approach where developers partner with AI to generate code. Instead of typing every line manually, they use natural language to collaborate with an AI model – guiding and prompting the model to generate and refine code through an iterative "vibing" process.

So why is vibe coding taking off? What are some of its benefits and challenges? Let's take a closer look at this very new approach to coding.

### Why Vibe Coding Is Growing in Popularity

Vibe coding has lowered the barrier to entry for software development, allowing non-technical individuals and hobbyists to build applications by simply describing what they want. Suddenly just about anyone can create their own custom tools and prototypes.

It's no wonder vibe coding is catching on. The following are some key reasons why:

#### 1\. Context-aware Assistance

In vibe coding, the AI model understands the broader context of the project. It can interpret the developer's intent and suggest code that aligns with the project's purpose, while also tracking variables, dependencies, comments, and recent edits in order to maintain a seamless workflow.

#### 2\. Productivity Gains

Research has [found](https://itrevolution.com/articles/new-research-reveals-ai-coding-assistants-boost-developer-productivity-by-26-what-it-leaders-need-to-know/) that AI coding assistants can boost the productivity of software developers by 26% in real-world business settings. AI models automate mundane and repetitive tasks like writing boilerplate code and generating tests, increasing the speed of iteration and time-to-prototype.

#### 3\. Cognitive Offloading

Instead of focusing on syntax and structure, developers can spend more time thinking about architecture and design. Delegating low-level tasks to the AI model enables them to focus on higher-level decisions like what the software should do, rather than how to make it happen.

### Benefits of Vibe Coding

Beyond speed and efficiency, here are a few benefits of vibe coding:

-   **On-demand collaboration**: The AI model is always available to collaborate as a coding partner. For those new to coding, it can effectively act as a built-in teacher, and for seasoned developers, it helps brainstorm and validate ideas.

-   **Creative workflow**: By handling the mundane, repetitive parts of coding, AI lets developers concentrate on the big picture, which encourages deeper thinking and creative problem-solving.

-   **Democratizing software development**: With vibe coding, non-technical users can build and prototype their own tools. Programming expertise is no longer required to turn ideas into working applications, opening the door to greater creativity and inclusion in tech.

Vibe coding isn't without drawbacks, however.

### Potential Pitfalls of Vibe Coding

Despite its advantages, vibe coding does introduce real challenges that must be thoughtfully dealt with, such as:

-   **Hallucination or incorrect code**: AI models can generate convincing but incorrect code, which could include security flaws. Developers should review and test everything that comes from the AI, essentially treating it as an assistant that requires careful supervision.

-   **Version drift**: Over time, inconsistencies may emerge between human-written and AI-generated sections of code, and the code may increasingly deviate from the intended architectural design. Careful integration pipelines and regular "re-anchoring" to the original goals are crucial to enforce version control.

-   **Governance and compliance issues**: With AI contributing portions of code, organizations must ensure that all of the code they use is auditable and properly licensed while also being safe to deploy. Human oversight at every step is vital.

Overall, vibe coding is a pretty exciting new phase in software development. It just needs to be used carefully, at the right time and in the right place.

### Final Thoughts

Most vibe coding happens on public platforms powered by cloud-hosted AI models. This is perfectly fine for experimentation but may not be ideal for projects involving sensitive code or proprietary data. For individuals and teams that need privacy or operate under strict compliance requirements, a closed and controlled environment may be a better fit.

With **Vast.ai**, it's easy to spin up secure, isolated GPU instances on demand – so you can host open-source models like Code Llama or DeepSeek Coder and get started vibe coding while keeping your data fully contained. Save up to **5–6X** compared to traditional cloud providers and enjoy both privacy and performance without the heavy price tag.

Our platform offers a practical way to experiment with vibe coding on your own terms – securely, affordably, and under your control. Explore Vast.ai today\!


Discover vibe coding — an AI-driven way to code faster and smarter. Collaborate with AI to generate, refine, and prototype code securely on Vast.ai.

Less Coding, More Creating: The Rise of Vibe Coding


One of the most important considerations when you're running private AI models on a cloud platform is data security. You want to protect your training data and the outputs your models generate – and even the models themselves.

At Vast.ai, we recognize that there's always some level of risk when sensitive data moves through cloud infrastructure. That's why we've built safeguards into every layer of our platform.

From containerized environments to our Secure Cloud offering and ongoing compliance efforts, here's how we help ensure you can run private AI models with confidence on Vast.

### Docker Container Isolation

When you rent GPUs on a cloud platform like Vast.ai, your workloads run inside Linux Docker instances. Each of these instances is a container created from a Docker image, where the container packages your processes and dependencies into a self-contained unit. This container runs in its own isolated environment, separate from the host system and from other clients' containers.

Why is this important? On a multi-tenant platform, multiple users may be running workloads on the same physical server. But with the isolation provided by a containerized environment, **each client only has access to their own data.**

Additionally, when you delete your instance, all data stored on it is also deleted. If you stop your instance, your data will still be stored on the disk and able to be copied in and out, but it's crucial to destroy your instances when you no longer need them. This helps protect your data while also avoiding unnecessary disk storage charges.

If you require the extra assurance of enterprise-grade data protection, you're in luck. Our Secure Cloud option may be just what you're looking for.

### Vast.ai's Secure Cloud Offering

Our **Secure Cloud offering** is designed for clients with the strictest data protection requirements – particularly well suited for when you're running your own private AI models or working with proprietary or sensitive datasets. It's our highest security tier, giving you the option to select GPU infrastructure only from our vetted datacenter partners.

The datacenter providers in our Secure Cloud all hold a **minimum of ISO 27001 certification**. Many are also HIPAA, NIST, PCI, and/or SOC 1-3 certified, as well as GDPR compliant. Each datacenter provider signs comprehensive Data Processing Agreements (DPAs) with Vast.ai and undergoes both internal and third-party audits on a regular basis.

Furthermore, we compile documentation on the identity of each provider and ensure they follow industry-recognized best practices for incident response, continuous monitoring systems, and security and compliance training for employees.

Our Secure Cloud offering provides the strongest safeguards available on Vast.ai. However, we also maintain our own ongoing efforts to reinforce data protection across our platform.

### Ongoing Compliance Efforts

At Vast.ai, we understand that compliance is an ongoing responsibility. Our integrated approach to security and trust weaves robust compliance measures throughout our platform to meet the unique needs of every client.

Some of our ongoing efforts include:

-   [**SOC 2 Type I and II and SOC 3 Certification**](https://vast.ai/article/vast-soc2-typeII-certification)s – Independent audits of our internal controls, validating that your sensitive data is secure, available, and privately stored across all of our systems (along with a standard 12-month Type II verification cycle for continuous coverage year round).

-   [**Vulnerability Bounty Program**](https://vast.ai/article/announcing-the-vast-ai-vulnerability-bounty-program) – A structured way to report bugs and responsibly disclose any security issues on our platform, as well as suggest new features.

Our [**Trust Center**](https://trust.vast.ai/) provides a detailed overview of our controls, compliance stance, FAQs and Subprocessors, and details about our other certifications and security practices.

### Securing Your Peace of Mind

Running private AI models on a cloud GPU platform requires careful attention to security. You must be able to protect not only the data you train on, but also the models you've built and the outputs they produce. Addressing these concerns is a top priority for us.

At Vast.ai, our goal is to give you the flexibility to scale workloads while maintaining the privacy and integrity of your data. As a result, you can be confident that your sensitive assets stay secure. As we continue to advance the security infrastructure we've built over 6+ years of trusted service, we remain focused on making powerful GPU compute accessible and secure for everyone.

Explore how **Vast.ai** can support your private AI workloads at scale, today\!


Learn how Vast.ai protects your sensitive data while running private AI models in the cloud. Discover our Secure Cloud offering, Docker container isolation, and ongoing compliance measures that ensure your AI workloads stay private and secure.

Running Private AI Models Without the Risk of Data Exposure


We're back with a new round of updates here at Vast.ai\! These past few months, we've expanded our offerings so you can get the most out of your time and budget – with new hardware options, fresh templates, and continued enhancements to elevate your experience.

To that end, we're pleased to offer **NVIDIA B200 GPUs** for rent\! These incredibly powerful machines deliver exceptional performance for develop-to-deploy pipelines and all of your most demanding enterprise AI tasks.

But that's not all. Here are the details on what's new\!

### NVIDIA Cloud GPU Updates

As mentioned, the NVIDIA B200 joins our growing GPU lineup, bringing you more power and options than ever.

Featuring **192 GB** of HBM3e memory with extremely high bandwidth, the B200 is ideal for high-throughput AI workloads like fine-tuning, Retrieval-Augmented Generation (RAG), and LLM inference. For additional information and pricing details, learn more [here](https://vast.ai/pricing/gpu/B200).

### Fresh New Templates

We've also rolled out several templates that expand what you can build and run on Vast. Jump-start your creative and technical projects with templates such as:

-   [**InvokeAI**](https://cloud.vast.ai/?ref_id=62897&template_id=c309df384010cf9d3dc9abb9e0a5a6fd) – A creative engine providing a professional web interface for AI image generation using Stable Diffusion models.

-   [**Langflow \+ Ollama**](https://docs.vast.ai/langflow-ollama) – A visual AI workflow platform with an intuitive drag-and-drop interface in Langflow and a hosted Ollama API server with models like Llama, Mistral, and more.

-   [**Ostris AI Toolkit**](https://cloud.vast.ai/?ref_id=62897&template_id=81e6e9dd266d9bdbcbb0fd48b9afe357) – A complete environment for training custom Stable Diffusion models, LoRAs, and other AI image generation models using your own datasets.

-   [**ACE Step**](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ACE%20Step%20Music%20Generation) – An AI-powered music generation system that lets you create original music through a simple web interface.

### Other Improvements

We've recently updated our Discord server to make conversations more focused, relevant, and helpful for you. Didn't know we had a Discord? [**Join us**](https://discord.gg/vast) there today for direct access to updates, technical guidance, and our supportive user community\!

We've also made an important update to how API keys are handled. To strengthen security, **API keys are now one-time viewable and not readable anywhere on Vast.** Be sure to store your key off of Vast when it's created, as it won't be visible again afterward. Vast.ai API endpoints will no longer return API keys as part of their response (see details in change log below).

That wraps up the updates for October 2025\!

### Our Commitment

From the new B200 GPUs to convenient templates that help you move from idea to execution, this month's updates continue our mission to make high-performance compute accessible to everyone. We're here to help you build faster and create more, with a platform that grows with your needs.

Need assistance? Reach out to our customer support team at [support@vast.ai](mailto:support@vast.ai) or join our [Discord](https://discord.gg/vast) server anytime for tips, troubleshooting, and community insights.

### Change Log

#### **InvokeAI Template now available\!**

InvokeAI is a leading creative engine for Stable Diffusion models that provides a professional web interface for AI image generation. It's perfect for artists, designers, content creators, or anyone wanting to generate stunning AI artwork. Find the InvokeAI template on Vast.ai [here](https://cloud.vast.ai/?ref_id=62897&template_id=c309df384010cf9d3dc9abb9e0a5a6fd).

#### **Langflow Template & Guide are now live\!**

This template gives you a complete visual AI workflow platform with Langflow and a hosted Ollama API server running in Docker containers. Langflow provides an intuitive drag-and-drop interface for building complex AI applications, while Ollama serves as your private LLM backend with models like Llama, Mistral, and more. Learn how to launch Langflow on Vast.ai [here](https://docs.vast.ai/langflow-ollama).

#### **Ostris AI Toolkit Template now available\!**

This template gives you a complete Stable Diffusion training environment with the Ostris AI Toolkit running in a Docker container. The toolkit provides an intuitive web interface for training custom Stable Diffusion models, LoRAs, and other AI image generation models with your own datasets. Find the Ostris template on Vast.ai [here](https://cloud.vast.ai/?ref_id=62897&template_id=81e6e9dd266d9bdbcbb0fd48b9afe357).

#### **ACE Step Template now available\!**

This template gives you a hosted ACE Step music generation server running in a Docker container. ACE Step is an AI-powered music generation system that lets you create original music through a web interface. It's perfect for musicians, content creators, developers, or anyone who wants to experiment with AI music generation. Find the ACE-Step template on Vast.ai [here](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ACE%20Step%20Music%20Generation).

# **Vast Discord Update: Unlock Your Channels\!**

We’ve streamlined our Discord to make it cleaner, faster, and more relevant to you. Here’s what’s changed.

## **Connect your Vast account to instantly unlock channels designed for your role:**

-   **Hosts** → Access to \#Hosting, \#Hosting-rigs, \#Support-hosting. (More host channels coming soon\!)
-   **Clients** → Unlock support rooms for templates, feature requests, and more.

# **API Key Security Update \+ API Breaking Changes**

**API keys are now one-time viewable and not readable anywhere on Vast.** You will need to store your API key off of Vast at the moment it is created, as it will only be visible this one time. If you need access to your existing API key, and did not store it off of Vast, you will need to delete it and create a new one. **Vast.ai API endpoints will no longer return API keys as part of their response.** However, we've added **key_id** to responses specifically for cases like this. The newly added key_id can be used to verify a user instead of api_key in certain scenarios. The key_id associated with a user's API key is returned from the show api keys endpoint / command.

## **key_id is now available for these API calls:**

-   create_user \- POST /api/v0/users
-   log in (regular and oauth and tfa) \- PUT /api/v0/users/current
-   update_user \- PUT /api/v0/users
-   get_current_user \- GET /api/v0/users/current
-   get_root_user \- GET /api/v0/users/root
-   connect_discord \- GET /api/v0/oauth/connect-discord/
-   disconnect_discord \- DELETE /api/v0/admin/disconnect-discord/
-   get_subaccounts \- GET /api/v0/subaccounts/

## **The following API key endpoints are now deprecated:**

-   GET /api/v0/auth/apikeys/&#123;id&#125;/
-   PUT /api/v0/commands/reset_apikey/

## **And the following will now have null for the key values:**

-   GET /api/v0/auth/apikeys/
-   GET /api/v0/team/apikeys/

## **New NVIDIA B200s added\!**

B200 GPUs are ideal for high-throughput AI workloads like fine-tuning, Retrieval-Augmented Generation, and LLM inference. These GPUs feature 192 GB of HBM3e memory with extremely high bandwidth.

-   [Learn more](https://vast.ai/pricing/gpu/B200)
-   [Rent now](https://cloud.vast.ai/?gpu_option=B200)


Discover what's new at Vast.ai! From the powerful NVIDIA B200 GPUs to fresh AI templates like InvokeAI, Langflow + Ollama, and Ostris, plus security and API updates — explore how we're making high-performance compute faster, safer, and more creative for everyone.

October 2025 Product Update


By now, just about everyone has either heard of large language models (LLMs) or has used them before. Not as many people are familiar with small language models (SLMs), however.

The names offer a clue – large and small – but size isn't the only thing that sets them apart.

So what are the differences between LLMs and SLMs? When do you actually need a large model, and when might a small one be the better choice?

In this post, we'll answer these questions and more. Here's what you need to know.

### Large Language Models vs. Small Language Models

An AI language model is a type of artificial intelligence that is trained on huge text datasets to **understand and generate human language.** These models use probabilistic machine learning to predict which words are most likely to appear next in a phrase sequence – and, in doing so, are able to generate text that resembles how people actually write and speak.

In practice, this ability is what underpins natural language processing (NLP) systems that let machines interact with us in useful ways.

When it comes to LLMs and SLMs specifically, though, let's look at how the two compare.

#### What Are LLMs?

With **massive parameter counts** in the billions and even trillions, most modern LLMs rely on the transformer architecture, which uses a **self-attention mechanism** to make sense of relationships between words across a sequence. This allows them to model incredibly complex patterns and long-range dependencies in language.

Since they're **trained on extensive datasets over a variety of domains**, LLMs are highly versatile. They have broad general knowledge and can generalize well across different tasks – even multitasking effectively across domains.

The downside is that training and running LLMs requires **significant computational power.** It's a resource-intensive process that often involves costly hardware and large-scale distributed computing infrastructure.

#### What Are SLMs?

At the basic level, an SLM is a smaller version of an LLM. It has **fewer parameters** – in the millions to low billions – and its architecture is focused on efficiency.

For instance, some SLMs use a **sliding window attention** mechanism (where the model focuses on a fixed-length "window" that slides across the text), and others rely on techniques like **grouped-query attention** (GQA), **sparse attention patterns**, and **low-rank adaptation** (LoRA) to improve efficiency and lower compute costs.

Unlike LLMs, SLMs are **trained on domain-specific data** or smaller datasets tailored to particular tasks. Although they may lack broad general knowledge, they do well in their specific domains and can even be fine-tuned for niche or regulated industries like finance and healthcare.

Because of their smaller size, SLMs need **less computational power** and can often be trained and deployed on more modest hardware.

### LLMs vs. SLMs: Strengts and Limitations

Both LLMs and SLMs have advantages and drawbacks. Here's a quick side-by-side comparison of their respective pros and cons:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Model Type
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Strengths
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Limitations
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">LLM</td>
            <td class="border border-gray-300 px-4 py-2">
                Broad general knowledge across different domains
                <br />
                Can handle complex, open-ended tasks with more contextual
                understanding
                <br />
                Greater multimodal potential
            </td>
            <td class="border border-gray-300 px-4 py-2">
                Requires extensive compute resources and costly hardware
                <br />
                Slower inference and higher latency
                <br />
                Not optimized for specific tasks, and higher risk of bias due to
                unfiltered training data
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">SLM</td>
            <td class="border border-gray-300 px-4 py-2">
                Easier to fine-tune for specific domains
                <br />
                Uses less compute resources
                <br />
                Faster inference and lower latency
                <br />
                Stronger data control and easier on-prem deployment
            </td>
            <td class="border border-gray-300 px-4 py-2">
                Narrower focus and less general knowledge; may require
                retraining for new tasks
                <br />
                Struggles with highly complex or long-context tasks
                <br />
                Limited multitasking capability
            </td>
        </tr>
    </tbody>
</table>

So how do you decide which type of model is right for your needs?

### Choosing the Right Model

As we've covered, neither model type is universally superior. The value of each one depends on what you're trying to accomplish and the resources you have available. That said, here are some guidelines to keep in mind.

#### An LLM may be the best choice if:

-   You need a model for **open-ended, multi-domain applications** such as general-purpose chatbots or creative content generation.  

-   Your use case involves **long-range context** across different subject areas.  

-   You have access to **powerful compute resources** and can manage the costs involved.

#### An SLM may be right for you if:

-   You prefer a model that is **lightweight and efficient**, suited for **resource-constrained environments.**  

-   Your application is **narrow or domain-specific**, such as FAQ bots or translation from one language to another.  

-   You want to **fine-tune with proprietary data** or maintain stricter control in regulated industries.

At the same time, you don't necessarily even have to choose one or the other; a hybrid approach could be the way to go. An SLM can take on routine tasks while an LLM handles more nuanced issues. For instance, you could use an SLM for standard customer queries and escalate more complex problems to an LLM.

### Final Thoughts

Both LLMs and SLMs offer unique advantages. Getting the most out of them often depends on having scalable, affordable compute – and that's where **Vast.ai** comes in.

Our cloud GPU platform gives you the flexibility to run a lightweight SLM on a single GPU or to train a massive LLM across [distributed clusters](https://vast.ai/products/clusters), all at a fraction of the typical cost. With Vast.ai, you can **save up to 5–6X** compared to traditional cloud providers – and train, fine-tune, and deploy AI models on your own terms.

Ready to get started? Spin up GPUs on demand with [Vast.ai](http://Vast.ai) today\!


Learn the key differences between Large Language Models (LLMs) and Small Language Models (SLMs), their strengths, limitations, and when to use each.

LLMs vs. SLMs: What's the Difference, and Why Does It Matter?


Are you looking for a faster way to turn your AI application ideas into working prototypes?

Langflow may be just the solution you need. Its visual editor simplifies the process, allowing you to build and test a functional AI application workflow quickly and easily. You can develop anything from chatbots and document analysis systems to content generators and agentic applications – and much more.

### What Is Langflow?

Put simply, [Langflow](https://docs.langflow.org/about-langflow) is an **open-source, node-based agent builder** that you can use right from your web browser. In minutes, you can create and serve flows: that is, functional representations of application workflows. All you have to do is connect and configure component nodes using the drag-and-drop visual editor, where each component represents one step in the workflow.

Then you can test your flows in real time using the Playground. This is a dynamic interface that lets you examine and experiment with different inputs, outputs, and memories – and get feedback about flow logic and response generation – before you move forward with more formal app development. Flows can even be embedded into your application code using the Langflow API.

Plus, Langflow is model-agnostic. It supports all major LLMs and vector databases you want to use, and it has built-in support for the Model Context Protocol (MCP), as well. It even has a **convenient Ollama integration** that makes it easy to use with open-weight models as well as custom fine-tunes.

With Vast, you can run Langflow with Ollama on powerful GPU instances at affordable rates, enabling cost-effective AI app development and prototyping. This guide will show you how to set up [Langflow \+ Ollama](https://docs.vast.ai/langflow-ollama) using our pre-built template in just minutes.

### Getting Started with Langflow \+ Ollama

Before you do anything else, set up your Vast account and add credit. If you need help with this step, review our [QuickStart guide](https://docs.vast.ai/quickstart) for a walkthrough of the setup process.

Now it's time to configure your template. Minimal changes are needed here, but you'll probably want to customize the template so that Ollama will download your preferred model automatically.

Follow our [step-by-step guide](https://docs.vast.ai/langflow-ollama) to get you started in our documentation.

### Conclusion

There is so much more to Langflow than it's possible to cover in this short introduction. The endlessly customizable framework lowers the barrier to building and testing AI applications, while Ollama keeps model choice open and flexible. Both beginners and seasoned developers can benefit from its ease of use and support for creating more complex projects.

Running Langflow \+ Ollama on **Vast.ai** gives you the cost-effective GPU infrastructure to go from prototype to production on your own terms.

Turn your ideas into working real-world solutions today\!


Run Langflow with Ollama on Vast.ai GPUs for fast AI prototyping. Step-by-step guide to build chatbots, agents, and custom AI apps with a visual editor.

Build and Test AI Applications with Langflow + Ollama on Vast.ai


[Video generation with AI](https://docs.vast.ai/video-generation) has come a long way. If you saw the infamous AI-generated video of Will Smith eating spaghetti a couple of years ago, you probably remember how unnatural and distorted the clip looked\! But that was just the beginning.

Things look very different today, with more advanced video generation tools producing much higher quality results. Today's models can create photorealistic visuals with smooth, consistent motion as well as stylized and cinematic effects if that's what's called for.

AI video generation has become a practical tool for anyone who needs an easy way to go from text to video, no matter the project. The right templates make that process even simpler.

With that in mind, let's take a look at a few of the best [AI video generation](https://docs.vast.ai/video-generation) templates available today.

### 1\. [ComfyUI \+ LTX Video](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%20%2B%20LTX%20Video)

[**LTX Video**](https://github.com/Lightricks/ComfyUI-LTXVideo) is a group of open-source AI video models built for high-quality diffusion-based video generation. It offers both speed and precision, generating high-resolution videos **up to 60 seconds long** (with some versions of the model limited to **10 seconds**), often in less time than it takes to watch the resulting video.

How is this possible? LTX Video uses a unique approach where it **compresses video data 192 times smaller than the original** – past the limits of what most models can achieve – while still maintaining crisp visuals and smooth, realistic motion. A bonus is that this drastically reduces the amount of memory and compute required. In fact, LTX Video is optimized specifically for consumer-grade GPUs.

While most diffusion models use a two-step pipeline where a second-stage model sharpens details, LTX Video does decoding and cleanup in just one step. With a Video-VAE (variational autoencoder) and a denoising transformer combined into a single integrated system, there's no need to patchify the video stream, and high-quality results are possible even under heavy compression.

This [template](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%20%2B%20LTX%20Video) pairs LTX Video with **ComfyUI**, a modular, node-based interface for Stable Diffusion workflows. Together, they serve as a seamless way to run efficient, high-compression video pipelines with a user-friendly interface.

### 2\. [Mochi](https://vast.ai/article/generating-videos-with-mochi-on-vast.ai)

Designed to excel in quality of motion as well as prompt adherence, [**Mochi**](https://github.com/genmoai/mochi) is a state-of-the-art open-source video generation model that performs competitively with leading closed models.

It demonstrates strong motion quality, generating smooth videos at **30 frames per second for durations of up to 5.4 seconds.** Fluid dynamics, fur and hair simulation, and human movement are all rendered with impressive realism.

Mochi's alignment with text prompts is also exceptional. Generated videos accurately depict the given input, allowing users to control characters, settings, and actions with a high degree of detail. Prompt adherence is benchmarked with a vision-language model and generated videos are evaluated using Gemini to ensure accurate, consistent results.

However, keep in mind that the initial release of Mochi only generates videos at **480p**, and the model is optimized for photorealistic styles, so animated content is not recommended.

### 3\. [Open-Sora](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Open-Sora)

[**Open-Sora**](https://github.com/hpcaitech/Open-Sora) is an open-source video generation model with a streamlined, user-friendly platform that lowers the barrier to efficient, high-quality video content creation. It can produce videos **up to 15 seconds long, at resolutions of up to 720p, with any aspect ratio.**

The model also supports a wide range of visual generation tasks: text-to-image, text-to-video, and image-to-video generation. This flexibility means users can run diverse workflows within a single framework.

Notably, Open-Sora introduces two innovations. The first is the Spatial-Temporal Diffusion Transformer (STDiT), a framework that decouples spatial and temporal attention. The second is a highly compressive 3D autoencoder that not only makes representations more compact but also accelerates training with its own ad hoc training strategy.

These features allow Open-Sora to generate high-quality results more efficiently – and with a ready-to-use template, it's easier than ever to start experimenting.

### Final Thoughts

AI video generation has matured rapidly in just a few short years. Instead of glitchy experiments, we now have reliable tools that deliver polished video content.

At **Vast.ai**, we believe in democratizing [access to these advanced AI tools](https://docs.vast.ai/video-generation), making them easy to use and available to everyone without the overhead of complex setup.

With our pre-built templates for [LTX Video](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=ComfyUI%20%2B%20LTX%20Video), [Mochi](https://vast.ai/article/generating-videos-with-mochi-on-vast.ai), and [Open-Sora](https://cloud.vast.ai/?ref_id=62897&creator_id=62897&name=Open-Sora), you can launch cutting-edge video generation models in minutes – backed by affordable, on-demand cloud GPUs that save you from having to purchase and maintain your own expensive hardware. Use only the compute you need, when you need it.

Ready to experiment? Explore the [video generation templates](https://docs.vast.ai/video-generation) on Vast.ai and bring your ideas to life today\!


Generate AI videos with LTX Video, Mochi, and Open-Sora on Vast.ai. Step-by-step templates for high-quality, photorealistic text-to-video workflows with affordable GPU hosting.

Vast.ai: Best Video Generation Templates 2025


NVIDIA is [reportedly](https://www.reuters.com/world/china/nvidia-working-new-ai-chip-china-that-outperforms-h20-sources-say-2025-08-19/) working on a new China-specific AI chip that will be more powerful than the aging HGX H20 model the company is currently authorized to sell in China.

Recent reports [suggest](https://www.tomshardware.com/tech-industry/artificial-intelligence/nvidia-responds-to-reports-that-its-h20-gpu-for-china-is-ending-production-next-gen-b30a-green-light-up-to-the-united-states-government-according-to-ceo-jensen-huang) that the H20 – originally developed as a scaled-down alternative to the A100 and H100 that were banned from export to China – is nearing the end of its production run. Its successor: the tentatively named B30A chip.

Based on the latest Blackwell architecture, the B30A will feature a single-die design delivering roughly half the raw computing power of NVIDIA's flagship B300 accelerator, according to sources. The new chip will include high-bandwidth memory and NVLink support for fast data transfer between processors – both of which are also present in the Hopper-based H20.

NVIDIA aims to send test batches of the B30A chip to Chinese customers as early as sometime this month, sources said.

The move comes following remarks from President Donald Trump suggesting that more advanced NVIDIA chips could be cleared for sale in China. However, regulatory approval remains uncertain. There are ongoing concerns about granting China broader access to U.S. AI technology – which could weaken the United States' competitive edge in the sector.

At the same time, analysts warn that restricting China too heavily risks driving developers toward rivals like Huawei. NVIDIA argues that it's important to retain market share in China – which generated 13% of the company's revenue in the past year – so that its software ecosystem stays relevant in one of the world's largest AI markets.

NVIDIA is also preparing a second Blackwell-based China-specific chip called the RTX6000D, designed primarily for AI inference tasks, which is expected to start shipping this month. With weaker specs than the H20 and bandwidth just under U.S. export limits, it's set to be a compliant, lower-cost alternative within NVIDIA's lineup.

For those outside China, however, these region-specific models highlight the stark difference between scaled-down chips and the unrestricted performance available elsewhere.

### On-Demand Access to Powerful GPUs, Anytime

While NVIDIA adapts its offerings to meet specific market regulations, users elsewhere don't have to work around restricted versions.

On [**Vast.ai**](http://Vast.ai), you can access a wide range of powerful GPUs – from the [RTX 5090](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know) to the [A100](https://vast.ai/article/nvidia-rtx-4090-vs-a100-two-powerhouses-two-purposes), [H200](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights), [B200](https://vast.ai/article/nvidia-h200-vs-b200-comparing-datacenter-grade-accelerators), and more – on demand, at a fraction of the cost of traditional cloud providers. Our cloud GPU rental platform makes it easy. Just choose the hardware you need, spin up an instance in seconds, and pay only for the compute you use.

Get started with Vast today, and enjoy flexible, cost-effective GPU compute that scales with your needs\!


NVIDIA is developing the B30A AI chip for China, a Blackwell-based successor to the H20 with half the power of the B300. Learn what it means for AI exports, competition, and how you can still access unrestricted GPUs like the H200, B200, and A100 on Vast.ai.

Report: NVIDIA Developing New Blackwell-Based AI Chip for China


Remember when ChatGPT couldn't tell you the weather or gather top headlines? Are you still frustrated asking Gemini to create a JIRA ticket from a bug report email? Perhaps you're tired of Cursor hallucinating database column names because it's walled off from your live schema, so you're constantly copy-pasting the same context across your Claude and Cursor apps. Despite their eloquent email-writing powers, out-of-the-box Large Language Models (LLMs) are powerful predictors, not contextually-aware assistants. They operate in silos, forcing you to start with fresh context \-- or none at all \-- for every new task.

This constant context-juggling isn't just annoying, it's fragile. When a better model is released or your data governance rules change, do you have to re-implement your entire workflow? What if you want to easily swap out Gemini's new video generation model for one from Stable Diffusion?

With [Model Context Protocol](https://modelcontextprotocol.io/) (MCP), an open-source framework released by Anthropic for building truly helpful, environment-agnostic AI assistants, you can create a persistent, shareable context layer across models and tools. And with [Vast.ai](http://vast.ai)'s secure and affordable cloud GPU instances, implementing your own MCP has never been easier.

**Model Context Protocol Architecture**

MCP architecture is built on three key components:

-   Clients: The user-facing applications (like Cursor, a custom chatbot, or Claude) that coordinate between the user, the AI model, and the MCP server.
-   Servers: The adapter between your tools and AI models, serving tools, resources, and prompts to your LLM provider and other services.
-   Service Providers: The software tools and data sources (like Slack, Notion, GitHub, or your company's internal database) that have existing APIs.

With MCP, any developer can build a server to connect an API to any MCP-compatible client. You no longer have to wait for official integrations from LLM providers.

An MCP server provides three categories of capabilities to the model – tools, resources, and prompts. In the following guide, you'll focus on creating MCP tools and learn to create useful actions to spawn from your clients via a remote MCP server on [Vast.ai](http://Vast.ai)'s platform.

**Remote MCP Servers**

Initially, MCP architecture only supported locally running servers using stdio transport. Since summer 2025, it supports remote access via streamable-http, enabling scalable, AI-enabled workflows that integrate LLMs across your organization. Now teams can more easily integrate data services and build custom workflows within their workspaces.

​

[Vast.ai](http://vast.ai)'s affordable GPU instance rentals provide flexible and affordable hosting for as many MCP servers as you want to create – scaling up your team's AI workflows quickly and securely.

**Tutorial: Build a Stock Sentiment Analysis MCP**

Today, you'll build an MCP server for stock sentiment analysis correlation hosted on [Vast.ai](http://vast.ai)'s platform. This setup provides a foundation for team-based workflows. For example, a financial analyst company may use this MCP tool to probe further into market conditions all within the Claude interface, or spur discussions by connecting this output to a shared Slack channel, providing up to date and collective information to multiple teams. To begin, let's review what you'll need for this process.

[Vast.ai](http://vast.ai)'s GPU rentals offer space for a vast variety of MCP servers, easily accommodating 70B large language models or image diffusion models on the most up-to-date graphics card offerings. Their platform creates opportunities for creative, custom server integrations like creating a shared, product-specific image prompt for your product teams or pulling in proprietary climate energy data to your coding and marketing workflows.

By the end of this tutorial, you will understand how to build a similar bespoke MCP server for your organization's needs, and have a running blueprint for creating your collection of remote MCP servers on [Vast.ai](http://vast.ai). Let's get started.

**Prerequisites**

-   A [Vast.ai](http://vast.ai) account with an API key
-   A Hugging Face account with Mistral7B access token
-   Python 3.10+ environment

1. **Rent and Configure Your [Vast.ai](http://vast.ai) GPU Instance**  
   Start by selecting a GPU Instance from the Vast.ai console. For this project, you'll use a **Mistral 7B LLM** to summarize and analyze market sentiments, so we recommend using an **RTX 4090 GPU with 24GB of VRAM**. [Vast.ai](http://vast.ai) offers a variety of powerful graphics cards, including A100 and 80GB VRAM for larger LLM and diffusion tasks.  
   Begin by selecting a PyTorch (Vast) template. Modify this template to launch as an interactive shell server without a Jupyter notebook instance and with 24 GB of disk space. For more information on selecting templates, please refer to the [template guide.](https://docs.vast.ai/instances/templates)  
   ![][image3]Now select an RTX 4090 offering at your required price point and select "Rent". Navigate to the "Instances" section in the sidebar. When the server is up and running, you should see "Running" next to "Status". Now you're ready to create your MCP server.  
   ![][image4]
2. **Create Your Environment**  
   In your terminal, install uv and set up your Python environment using the following command:

​​

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Now, restart your terminal to activate the \`uv\` command. Then, initialize a Python project.

```bash
uv init vast_mcp_server
cd vast_mcp_server
```

​​

Finally, create and activate your virtual environment to manage project dependencies.​

```bash
uv venv
source .venv/bin/activate
```

3. **Install Project Dependencies**  
   Create a requirements.txt file to outline project dependencies. You'll use this to write your server code locally and then install dependencies on your [Vast.ai](http://vast.ai) instance later.​

```bash
touch requirements.txt
```

Then add the following dependencies:​

```
fastmcp
pydantic
transformers
torch
requests
uvicorn
accelerate
```

Install the dependencies using the following command.

```bash
pip install -r requirements.txt
```

4. **Create Your MCP Server**  
   Now you'll create your first MCP server. This server will facilitate summarizing and analyzing news headlines using an LLM and correlating them with real-time stock data. This is a real-time task that out-of-the-box LLMs like Claude may struggle with. You'll create three MCP tools for stock analysis retrieval, sentiment analysis, and correlation – all available via your remote server.  
   This example uses [FastMCP](https://gofastmcp.com/getting-started/welcome) to facilitate simple protocol implementation. FastMCP uses simple decorators to handle the low-level implementation details, similar to the FastAPI approach.

Create a 'server.py' and 'test_server.py' file using the command:​

```bash
touch server.py test_server.py
```

First, you'll configure a Mistral 7B model for summarization and sentiment analysis for a given news article. You'll also set up a FastMCP server. Add the following code to the 'server.py' file.​

```python
import random
import logging
import asyncio
import os
import json
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelForCausalLM
from fastmcp import FastMCP
import torch


logging.basicConfig(level=logging.INFO)

# Initialize the MCP server
mcp = FastMCP("Financial Market Analyzer")

# Configure Mistral 7B LLM model
MODEL_NAME = "mistralai/Mistral-7B-Instruct-v0.2"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {DEVICE}")

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    device_map="auto"
)

class NewsItem(BaseModel):
    title: str
    description: str
```

Now, you'll create three tools. The first uses the Mistral 7B LLM to summarize and analyze a provided news article. In a real-world implementation, you may want to update this example to accept a URL as input. Add the following code:

```python
# Analyzes the sentiment and provides a summary of a news headline using a large language model.
@mcp.tool()
def analyze_sentiment(news_item: NewsItem):
    prompt = (
        f"You are a financial news analysis expert. "
        f"Analyze the following news headline and description. "
        f"Provide a brief summary and determine the sentiment as POSITIVE, NEGATIVE, or NEUTRAL. "
        f"Respond only with a JSON object containing two keys: 'summary' and 'sentiment'.\n\n"
        f"Headline: \"{news_item.title}\"\n"
        f"Description: \"{news_item.description}\"\n\n"
        f"JSON Response:"
    )

    inputs = tokenizer(prompt, return_tensors="pt").to(DEVICE)

    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)

    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

    try:
        json_start_index = generated_text.find("{")
        if json_start_index != -1:
            json_str = generated_text[json_start_index:]
            analysis_result = json.loads(json_str)
            summary = analysis_result.get("summary", "No summary found.")
            sentiment_label = analysis_result.get("sentiment", "NEUTRAL").upper()
        else:
            summary = "Could not generate a structured summary."
            sentiment_label = "NEUTRAL"
            logging.warning("LLM response did not contain a valid JSON object.")

    except json.JSONDecodeError as e:
        summary = "Could not parse LLM response."
        sentiment_label = "NEUTRAL"
        logging.error(f"Failed to parse JSON from LLM: {e}")

    logging.info(f"Analyzed '{news_item.title}': Sentiment={sentiment_label}, Summary='{summary}'")
    return {"title": news_item.title, "sentiment": sentiment_label, "summary": summary}
```

Now you'll add a stock retrieval tool. In this example, you'll mock the stock data. However, in an actual implementation, you might pull this data from a stock-related API. Add the following code:

```python
# Fetches mock real-time stock data. In a real-world implementation, you would connect to a secure financial database API.
# For this example, we are using a simple mock to simulate real-time data.
@mcp.tool()
def get_stock_data(symbol: str):
    price = round(random.uniform(100.0, 500.0), 2)
    volume = random.randint(100000, 5000000)
    logging.info(f"Fetched mock data for {symbol}: Price={price}, Volume={volume}")
    return {"symbol": symbol, "price": price, "volume": volume}
```

Finally, you'll add a tool that correlates the stock prices and news sentiment. Add the following code:

```python
@mcp.tool()
async def correlate_data(stock_data: dict, news_analysis: dict):
    sentiment = news_analysis['sentiment']
    price = stock_data['price']

    correlation_message = "No clear correlation found."
    if sentiment == "POSITIVE" and stock_data['volume'] > 2000000:
        correlation_message = f"POSITIVE news coincided with a high trading volume. Possible impact. News Summary: {news_analysis['summary']}"
    elif sentiment == "NEGATIVE" and price < 300:
        correlation_message = f"NEGATIVE news for a low-priced stock. Potential further decline. News Summary: {news_analysis['summary']}"

    logging.info(f"Correlated data: {correlation_message}")
    return {"stock_data": stock_data, "news_analysis": news_analysis, "correlation": correlation_message}
```

Now, you'll create the ability to run your MCP remote server via streamable-http. Add the following code:

```python
# Runs the MCP server remotely using streamable-http transport
if __name__ == "__main__":
    logging.info(f"MCP server started on port {os.getenv('PORT', 8080)}")
    asyncio.run(
        mcp.run_async(
            transport="streamable-http",
            host="0.0.0.0",
            port=os.getenv("PORT", 8080),
        )
    )
```

Great work! Now you have three MCP tools defined on your server. After deploying via [Vast.ai](http://vast.ai), you can access any of these tools remotely by integrating with Claude, Cursor, or other MCP supporting apps. Let's upload it to your [Vast.ai](http://vast.ai) GPU instance.

5. **Upload Your MCP Server to [Vast.ai](http://Vast.ai)**  
   First, retrieve your API key from the Vast.ai Console. Then, install the Vast.ai CLI in your Linux terminal using the following command.

```bash
pip install --upgrade vastai
```

Set your API key in your local Vast CLI configuration.

```bash
vastai set api-key #your-api-key-here
```

Now you'll be able to connect by SSH to your remote GPU instance. Navigate to your [Vast.ai](http://vast.ai) Console and select your GPU instance. Click on "Open Terminal Access". Here, you should see your SSH commands to connect to your GPU instance with the following structure:

```bash
ssh -p XXXXX root@XXX.XXX.XXX.XXX -L XXXX:localhost:XXXX
```

Navigate back to your terminal and use the command to connect to your instance. If your instance is running, you should see your terminal login to the remote [Vast.ai](http://vast.ai) instance with a welcome message.

Since you are using a Hugging Face model, you'll need to login to the Hugging Face CLI using the following:

```bash
huggingface-cli login
```

This command with prompt you for your access token from the Hugging Face website.

Then install the dependencies using:

```bash
pip install -r requirements.txt
```

Finally, you'll securely copy your [server.py](http://server.py) and requirement.txt files to the remote GPU. Navigate to the terminal you used to create your server environment and ensure the virtual environment is activated. Use the following command to move your files.

```bash
scp -P 29946 server.py requirements.txt root@172.81.127.37:/root/
```

Now you're ready to run and test your server!

6. **Run and Test Your Server**  
   In your open terminal connected to your [Vast.ai](http://vast.ai) GPU instance, simply run

```bash
uv run server.py
```

If all is well, you should see the models downloading and the server startup success message:

```
Starting MCP server 'Financial Market Analyzer' with transport 'streamable-http' on http://0.0.0.0:8080/mcp
INFO: Started server process [2135]
INFO: Waiting for application startup.
INFO:mcp.server.streamable_http_manager:StreamableHTTP session manager started
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080
```

As a final step, test the server connection to ensure your tools are available. In your Python environment, create a file called 'test_server.py' and paste the script below.

```python
import asyncio
import json

from fastmcp import Client

# Test the MCP server using streamable-http transport.
async def test_server():
    async with Client("http://localhost:8080/mcp") as client:

        # List available tools
        tools = await client.list_tools()
        for tool in tools:
            print(f"Tool found: {tool.name}")

        # Call the news analysis tool
        print("Calling stock tool for INVC")
        stock_result = await client.call_tool("get_stock_data", {"symbol": "INVC"})
        print(f"Result: {stock_result[0].text}")

        # Call the sentiment analysis and summarization tool
        print("2. Calling 'analyze_sentiment' tool...")
        news_item_payload = {
            "news_item": {
                "title": "Tech Giant Announces Record Profits",
                "description": "Shares surged after the company reported quarterly earnings that exceeded all analyst expectations, signaling strong growth."
            }
        }
        sentiment_result = await client.call_tool("analyze_sentiment", news_item_payload)
        print(f"Result: {sentiment_result[0].text}")

        # Call the correlation tool
        print("3. Calling 'correlate_data' tool...")
        correlation_payload = {
            "stock_data": json.loads(stock_result[0].text),
            "news_analysis": json.loads(sentiment_result[0].text)
        }
        correlation_result = await client.call_tool("correlate_data", correlation_payload)
        print(f"Result: {correlation_result[0].text}")

if __name__ == "__main__":
    asyncio.run(test_server())
```

```bash
uv run test_server.py
```

Navigate back to your Python environment terminal and run the script.

​If your server is working correctly, you should see output showing each of the tools and calling each tool in turn.

Congratulations! You have created your first MCP server on [Vast.ai](http://vast.ai). Now you can integrate this into Claude, Cursor, Windsurf, or any other MCP-supporting client, so you can customize, scale, and create more MCP servers for your organization and AI-enabled workflows.


Learn how to build and host a custom Model Context Protocol (MCP) server on Vast.ai's affordable GPU cloud. Step-by-step guide for creating AI-enabled workflows, integrating stock sentiment analysis, and scaling remote MCP servers with Claude, Cursor, and more.

Building Your First MCP Server on Vast.ai

Deploy OpenAI's GPT-OSS-20B on Vast.ai with Harmony SDK for transparent multi-channel AI reasoning. Step-by-step tutorial to build a weather assistant with function calls and reasoning visibility.

Build a Weather Assistant with OpenAI GPT-OSS and Harmony SDK on Vast.ai


## Introduction

Qwen-Image revolutionizes image generation with its exceptional ability to render complex text directly within images, supporting multiple languages including English and Chinese with perfect typographic detail. This powerful foundation model from Alibaba's Qwen series goes beyond simple image creation, offering advanced editing capabilities, style transfer, and precise control over visual elements.

What sets Qwen-Image apart is its nuanced understanding of text-image relationships. While other models struggle with text rendering, Qwen-Image seamlessly integrates readable text into visuals, making it ideal for creating signage, posters, infographics, and artistic compositions that require precise text placement and formatting.

Vast.ai provides the perfect platform for deploying Qwen-Image, offering access to high-performance GPUs at competitive prices with the flexibility to scale based on your needs.

In this guide, you'll deploy Qwen-Image on Vast.ai and explore its capabilities for text-integrated image generation.

## Setting Up the Environment

First, install the Vast CLI and configure your API key:

```bash
pip install --upgrade vastai
export VAST_API_KEY="<your-api-key>"
vastai set api-key $VAST_API_KEY
```

You can obtain your API key from the [Vast.ai Account Page](https://cloud.vast.ai/account/).

## Choosing the Right Hardware

Qwen-Image requires substantial GPU resources for optimal performance:

-   **GPU Memory**: Minimum 80GB VRAM for the full model
-   **Recommended Hardware**: NVIDIA A100 80GB or H100
-   **Storage**: 200GB disk space for model weights and dependencies

Search for suitable instances with these specifications:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 80 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 150 \
rentable = true"
```

This search filters for:

-   Modern GPUs with compute capability 7.5+
-   At least 80GB VRAM for the model
-   Static IP for stable connections
-   Verified providers for reliability
-   Adequate storage for model and outputs

## Deploying Qwen-Image

Deploy the instance using a pre-configured Docker image optimized for transformer models:

```bash
export INSTANCE_ID=<instance-id>
vastai create instance $INSTANCE_ID \
    --image huggingface/transformers-pytorch-gpu \
    --env '-p 8888:8888' \
    --disk 200 \
    --args bash -c "pip install jupyter && jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root --NotebookApp.token='' --NotebookApp.password=''"
```

**Key parameters explained:**

-   `--image`: HuggingFace's PyTorch GPU image with ML libraries pre-installed
-   `--env '-p 8888:8888'`: Maps Jupyter's default port for external access
-   `--disk 200`: Allocates 200GB for model weights and generated images
-   `--args`: Auto-installs Jupyter and launches a passwordless notebook server

## Connecting to Your Instance

After deployment:

1. Navigate to the [Instances tab](https://cloud.vast.ai/instances/)
2. Find your instance and wait for "Running" status
3. Look for the "Open Ports" section in instance details
4. Click the port mapping link (e.g., `123.123.123.123:12345 -> 8888/tcp`)
5. This opens Jupyter directly in your browser
6. Create a new notebook or upload the example notebook if available

## Installing Dependencies

Install the required packages in your notebook:

```bash
apt install -y git
pip install git+https://github.com/huggingface/diffusers
```

The latest diffusers library from GitHub includes Qwen-Image support.

## Loading the Model

Initialize Qwen-Image with optimal settings for GPU deployment:

```python
from diffusers import DiffusionPipeline
import torch

model_name = "Qwen/Qwen-Image"

# Load the pipeline
if torch.cuda.is_available():
    torch_dtype = torch.bfloat16
    device = "cuda"
else:
    torch_dtype = torch.float32
    device = "cpu"

pipe = DiffusionPipeline.from_pretrained(model_name, torch_dtype=torch_dtype)
pipe = pipe.to(device)
```

The model uses bfloat16 precision on GPU for optimal memory usage and performance.

## Using Qwen-Image

Now let's explore Qwen-Image's text rendering capabilities with examples:

### Example 1: Complex Text Integration with Multiple Languages

```python
import os

positive_magic = {
    "en": "Ultra HD, 4K, cinematic composition.", # for english prompt
}

# Create img folder if it doesn't exist
os.makedirs("img", exist_ok=True)

# Generate image
prompt = '''A coffee shop entrance features a chalkboard sign reading "Qwen Coffee 😊 $2 per cup," with a neon light beside it displaying "Welcome". Next to it hangs a poster showing a beautiful Chinese woman, and beneath the poster is written "π≈3.1415926-53589793-23846264-33832795-02384197". Ultra HD, 4K, cinematic composition'''
negative_prompt = " " # using an empty string if you do not have specific concept to remove

# Set image dimensions (16:9 aspect ratio)
width, height = 1664, 928

image = pipe(
    prompt=prompt + positive_magic["en"],
    negative_prompt=negative_prompt,
    width=width,
    height=height,
    num_inference_steps=50,
    true_cfg_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(42)
).images[0]

# Save image to img folder
image.save("img/example_1.png")

# Display the image
display(image)
```

### Output

![Qwen Coffee Shop with Text](/uploads/example_1.webp)

This example demonstrates Qwen-Image's ability to:

-   Render multiple text elements (chalkboard, neon sign, poster)
-   Handle emojis and special characters
-   Integrate mathematical constants with precise formatting
-   Maintain readability across different text styles

### Example 2: Creative Fantasy Scene

```python
positive_magic = {
    "en": "Ultra HD, 4K, cinematic composition.", # for english prompt
}

# Create img folder if it doesn't exist
os.makedirs("img", exist_ok=True)

# Generate image
prompt = '''Rare photo: UFO hovering over a yeti riding the Loch Ness monster.'''
negative_prompt = " " # using an empty string if you do not have specific concept to remove

# Set image dimensions (16:9 aspect ratio)
width, height = 1664, 928

image = pipe(
    prompt=prompt + positive_magic["en"],
    negative_prompt=negative_prompt,
    width=width,
    height=height,
    num_inference_steps=50,
    true_cfg_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(42)
).images[0]

# Save image to img folder
image.save("img/example_2.png")

# Display the image
display(image)
```

### Output

![Fantasy Scene with UFO and Mythical Creatures](/uploads/example_2.webp)

This showcases Qwen-Image's creative capabilities in composing imaginative scenarios.

### Example 3: Futuristic Space Scene

```python
positive_magic = {
    "en": "Ultra HD, 4K, cinematic composition.", # for english prompt
}

# Create img folder if it doesn't exist
os.makedirs("img", exist_ok=True)

# Generate image
prompt = '''Now I want you to show a spaceships approaching Andromeda Galaxy!, it should be futuristic spaceships out on exploring universe'''
negative_prompt = " " # using an empty string if you do not have specific concept to remove

# Set image dimensions (16:9 aspect ratio)
width, height = 1664, 928

image = pipe(
    prompt=prompt + positive_magic["en"],
    negative_prompt=negative_prompt,
    width=width,
    height=height,
    num_inference_steps=50,
    true_cfg_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(42)
).images[0]

# Save image to img folder
image.save("img/example_3.png")

# Display the image
display(image)
```

### Output

![Futuristic Spaceships Approaching Andromeda](/uploads/example_3.webp)

This demonstrates Qwen-Image's ability to create detailed sci-fi scenes.

## Advanced Features and Capabilities

### Flexible Aspect Ratios

Qwen-Image supports various aspect ratios for different use cases:

```python
aspect_ratios = {
    "1:1": (1328, 1328),    # Social media posts
    "16:9": (1664, 928),    # Widescreen displays
    "9:16": (928, 1664),    # Mobile/portrait content
    "4:3": (1472, 1140),    # Traditional displays
    "3:4": (1140, 1472),    # Portrait photos
    "3:2": (1584, 1056),    # Standard photos
    "2:3": (1056, 1584),    # Portrait standard
}
```

### Generation Parameters

Fine-tune your outputs with these key parameters:

-   `num_inference_steps`: Higher values (50-100) for better quality
-   `true_cfg_scale`: Controls prompt adherence (3.0-7.0 range)
-   `generator`: Set seed for reproducible results
-   `negative_prompt`: Specify elements to avoid

## Practical Applications

Qwen-Image's unique text rendering capabilities enable numerous applications:

-   **Marketing Materials**: Create posters, banners, and advertisements with perfectly rendered text
-   **Educational Content**: Generate infographics and diagrams with clear labels and annotations
-   **Social Media**: Design posts with integrated text overlays and captions
-   **Multilingual Content**: Produce visuals with text in multiple languages
-   **Creative Projects**: Combine artistic imagery with typographic elements
-   **UI/UX Mockups**: Generate interface designs with readable text elements

## Performance Optimization

To maximize performance on Vast.ai:

1. **Use bfloat16 precision**: Reduces memory usage while maintaining quality
2. **Batch generation**: Process multiple prompts together when possible
3. **Cache the model**: Keep it loaded between generations to avoid reload overhead
4. **Optimize resolution**: Start with lower resolutions for testing, then increase for final outputs

## Conclusion

Qwen-Image represents a significant advancement in image generation technology, particularly in its ability to seamlessly integrate readable text into visual content. Its support for multiple languages, mathematical notation, and various text styles makes it uniquely suited for projects requiring precise text-image integration.

By deploying on Vast.ai, you gain access to powerful GPUs capable of running this advanced model efficiently. Whether you're creating marketing materials, educational content, or artistic compositions, Qwen-Image provides the tools to generate professional-quality visuals with integrated text.

### Next Steps

Try these to explore further:

-   **Experiment with multilingual prompts**: Test the model's capabilities with Chinese, Japanese, or mixed-language text
-   **Create branded content**: Generate marketing materials with your company's text and logos
-   **Build an API service**: Wrap the model in a REST API for integration with your applications
-   **Explore style transfer**: Use the model's editing capabilities to transform existing images

[Get started on Vast.ai](https://cloud.vast.ai/) and experience Qwen-Image today!


Deploy Qwen-Image on Vast.ai to create stunning visuals with perfectly rendered text, multilingual support, and advanced editing features.

Deploying Qwen-Image for Advanced Text-Integrated Image Generation on Vast.ai


If you're working at the cutting edge of AI and high-performance computing (HPC), NVIDIA's H200 and B200 GPUs are likely already very much on your radar. They're each designed to handle the massive compute demands of datacenter-class workloads – such as training trillion-parameter models and running high-speed inference at scale.

While both GPUs deliver incredible power, it's crucial to understand the differences between the two in order to make the right choice for your needs.

Today, we'll explore how the **H200** and **B200** measure up against each other and how each fits into real-world deployments. Which one is right for you? Let's take a look.

### NVIDIA H200: Hopper-Based Heavyweight

Building on the foundation laid by the H100, the NVIDIA H200 is based on the same Hopper architecture while bringing substantial improvements in memory capacity and bandwidth. This means fewer bottlenecks, an advantage that becomes increasingly important as you move further into enterprise scale.

The H200 nearly doubles the memory of its predecessor, boasting **141 GB of HBM3e memory** and **4.8 TB/s of bandwidth** – delivering about **1.4X faster** data access. MLPerf benchmarks using Llama 2 70B demonstrated the H200's power: it reached over 31,000 tokens per second, about **45% faster** than the H100.

With the same **fourth-gen NVLink**, **PCIe Gen5**, **MIG support**, and full Tensor Core and Transformer Engine capabilities, the H200 is a solid next step for existing Hopper-based deployments. You don't have to rethink your entire infrastructure; you just get more horsepower.

To sum up, here are some of the most notable features of the H200:

-   **141 GB of HBM3e memory at 4.8 TB/s** – almost double the memory of the H100 with 1.4X the bandwidth.

-   **Up to 1.9X improved LLM inference performance**, with more tokens per second and higher throughput for gen AI workloads compared to its predecessor.

-   **Lower energy use and total cost of ownership** (TCO) over the H100 thanks to improved energy efficiency and thermal management.

-   **An upgrade path exists on qualified HGX/MGX platforms**, so that little to no reconfiguration is required.

In short, the H200 represents the very best of the Hopper-based lineup. When it was introduced in late 2024, it set a new standard for memory-intensive and high-throughput workloads in datacenter GPUs – and it's still one of the most powerful GPUs on the market today.

The B200, on the other hand, is an entire generational leap forward.

### NVIDIA B200: The New Generation

Built on NVIDIA's advanced Blackwell architecture, the B200 features **192 GB of ultra-fast HBM3e memory** with **8.0 TB/s of memory bandwidth.** Those are some staggering numbers, even compared to the H200's respectable 141 GB at 4.8 TB/s.

The Blackwell architecture also introduces **fifth-gen Tensor Cores** and **dual Transformer Engines**, which are particularly useful for AI workloads with long context windows and heavy token parallelism. The B200 scales much more efficiently than the H200 for the latest and most advanced model classes.

In fact, scaling is a major area where the B200 stands out. With **fifth-gen** **NVLink**, its inter-GPU bandwidth reaches **1.8 TB/s** within a node – compared to Hopper's 900 GB/s – with cross-node scaling supported via the NVLink Switch System.

Of course, this may not matter to you very much if you're only using a single card. At cluster scale, however, or in multi-GPU systems like the DGX B200 (which connects eight B200 GPUs together), the improvements in distributed training efficiency really make a difference.

The B200 truly sits at the frontier of performance. Its raw capacity and speed can support next-gen inference pipelines and AI workloads at hyperscale. According to NVIDIA, it delivers **3X the training performance** and **15X the inference performance** of the previous generation.

These gains do come at a cost in power draw. The B200 has a max thermal design power (TDP) of **1000 W** compared to the H200 at 700 W – which means it also requires far more robust cooling solutions. But the extra power consumption may just be worth it for the boost in performance.

To recap, a few advantages of the B200 include:

-   **192 GB of HBM3e memory at 8.0 TB/s** – ultra-high capacity that's well suited for trillion-parameter models and allows training of AI models _entirely in GPU memory_.

-   **Fifth-gen Tensor Cores and second-gen Transformer Engines** for higher efficiency in FP8, FP16, and mixed precision performance, especially in long-context and multi-modal workloads.

-   **NVLink 5 interconnect** with up to **1.8 TB/s** GPU-to-GPU bandwidth for multi-node scaling.

-   **Fine-grain micro-tensor scaling** that optimizes performance and accuracy, enabling 4-bit floating point (**FP4**) AI.

The B200 was built to shatter performance records. As you can imagine, it has an eyewatering price tag to match. Nonetheless, it can be helpful to compare the B200 and H200 based on specs alone.

### H200 vs. B200: Key Specs Compared

Here's a side-by-side look at some key features of the H200 and B200:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Feature
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                H200
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                B200
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2">Hopper</td>
            <td class="border border-gray-300 px-4 py-2">Blackwell</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory
            </td>
            <td class="border border-gray-300 px-4 py-2">141 GB HBM3e</td>
            <td class="border border-gray-300 px-4 py-2">192 GB HBM3e</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Bandwidth
            </td>
            <td class="border border-gray-300 px-4 py-2">4.8 TB/s</td>
            <td class="border border-gray-300 px-4 py-2">8 TB/s</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                CUDA Cores
            </td>
            <td class="border border-gray-300 px-4 py-2">16,896</td>
            <td class="border border-gray-300 px-4 py-2">16,896 x2</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Cores
            </td>
            <td class="border border-gray-300 px-4 py-2">528 (4th Gen)</td>
            <td class="border border-gray-300 px-4 py-2">528 x2 (5th Gen)</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Boost Clock
            </td>
            <td class="border border-gray-300 px-4 py-2">1.98 GHz</td>
            <td class="border border-gray-300 px-4 py-2">1.98 GHz</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Transformer Engine
            </td>
            <td class="border border-gray-300 px-4 py-2">Single</td>
            <td class="border border-gray-300 px-4 py-2">Single (2nd gen)</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Form Factor
            </td>
            <td class="border border-gray-300 px-4 py-2">SXM</td>
            <td class="border border-gray-300 px-4 py-2">SXM</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Interconnect
            </td>
            <td class="border border-gray-300 px-4 py-2">
                NVLink 4 (900 GB/s) per GPU
            </td>
            <td class="border border-gray-300 px-4 py-2">
                NVLink 5 (1.8 TB/s) per GPU
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Precision Formats
            </td>
            <td class="border border-gray-300 px-4 py-2">
                FP8, FP16/BF16, TF32, FP32, FP64, INT8
            </td>
            <td class="border border-gray-300 px-4 py-2">
                FP4, FP6, FP8, FP16/BF16, TF32, FP32, FP64, INT8
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Multi-Instance GPUs
            </td>
            <td class="border border-gray-300 px-4 py-2">Up to 7</td>
            <td class="border border-gray-300 px-4 py-2">Up to 7</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Max Thermal Design Power (TDP)
            </td>
            <td class="border border-gray-300 px-4 py-2">700 W</td>
            <td class="border border-gray-300 px-4 py-2">1000 W</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Recommended Power Supply
            </td>
            <td class="border border-gray-300 px-4 py-2">1100 W</td>
            <td class="border border-gray-300 px-4 py-2">1400 W</td>
        </tr>
    </tbody>
</table>

Choosing the right GPU often comes down to more than just specs, however.

### Use Cases: Which GPU Should You Choose?

While both the H200 and B200 deliver game-changing performance capabilities, they each have distinct advantages depending on your needs.

#### You should consider the H200 if you prefer:

-   **A high-performance GPU** ideal for fine-tuning LLMs and high-throughput inference at the enterprise level without getting into exascale territory.

-   **A proven workhorse** that balances strong performance with lower TDP and cost.

-   **Being able to seamlessly upgrade** from existing Hopper deployments without overhauling infrastructure.

#### Or the B200 might be the right choice if you need:

-   **To push model boundaries** and handle long-context and/or multi-modal foundation model training at scale.

-   **The absolute maximum throughput and efficiency** for multi-GPU or multi-node clusters.

-   **A future-proof investment** for infrastructure built to support the next generation of cutting-edge AI and AGI.

### Conclusion

At **Vast.ai**, we know that the choice between GPUs like the H200 and B200 can depend on a lot more than just performance. Access and cost are important factors, as well.

The H200 and B200 represent the very best of NVIDIA's current lineup, and they're among the most powerful and expensive GPUs in the world today. Buying them outright is undoubtedly out of reach for most teams.

That's what we're here for. With Vast, you can rent high-end GPUs like the [A100](https://vast.ai/article/H100-vs-A100-Comparing-two-Powerhouse-GPUs), [H100](https://vast.ai/article/nvidia-h100-vs-l40s-power-meets-versatility), [H200](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights), and soon the B200, with on-demand pricing and spot instances – and pay only for what you use.

On average, **Vast users save 5–6X on GPU compute** compared to traditional cloud providers. You'll find a variety of options on our platform to fit your needs, so you can focus on what really matters: your work, not your budget.

Explore Vast today and enjoy enterprise-grade compute without enterprise-level costs.


The H200 and B200 GPUs represent the top end of NVIDIA's current lineup for AI and high-performance computing (HPC). Both are designed for serious, large-scale workloads – from training foundation models to running complex scientific simulations.

NVIDIA H200 vs. B200: Comparing Datacenter-Grade Accelerators

Deploy DeepSeek R1 on Vast.ai with vLLM and integrate Crew AI for scalable multi-agent workflows. Create cost-effective, OpenAI-compatible AI endpoints.

Running DeepSeek R1 + Crew AI on Vast.ai


Vast.ai GPUs can be seamlessly rented through SkyPilot, an open-source framework that automates AI, LLM, and batch job execution. This integration makes it easy to access Vast.ai's low-cost, high-performance GPUs while SkyPilot handles provisioning, scaling, and cost optimization.

Whether you're training deep learning models, fine-tuning LLMs, or running large-scale batch jobs, this partnership provides an efficient and cost-effective solution for developers everywhere.

## Why This Integration Matters

Running large-scale AI workloads can be expensive and resource-intensive, especially as GPU availability fluctuates across providers. With Vast.ai's extensive GPU supply and SkyPilot's intelligent cloud orchestration, you can enjoy streamlined access to affordable, on-demand compute resources.

SkyPilot automatically selects the most cost-effective and available infrastructure, provisioning GPUs to ensure seamless execution of AI and batch jobs at scale. As the market leader in low-cost cloud GPU rental, Vast.ai provides an ideal foundation for this approach.

## What Is SkyPilot?

Put simply, SkyPilot is a powerful framework for managing cloud workloads in a way that abstracts away infrastructure complexity. It was developed in the UC Berkeley lab that proposed the concept of "sky computing," which aims to make cloud providers function as a single compute pool.

With SkyPilot, you can run AI models and batch jobs efficiently without worrying about infrastructure management. SkyPilot includes improvements to managed jobs and Kubernetes support, faster GPU provisioning, and ready-to-use LLM recipes.

## Vast.ai's Role

You can easily access and manage GPU resources on Vast.ai through SkyPilot. Vast.ai provides a vast supply of cost-effective GPU compute, while SkyPilot automates the selection and provisioning process to match your workload needs – allowing you to run thousands of concurrent jobs and scale efficiently.

With Vast.ai's extensive GPU marketplace, you get reliable, affordable compute on demand. SkyPilot enhances this by simplifying deployment, ensuring a seamless experience while maximizing access and failover protection.

## Getting Started: Practical Tutorial

Now let's walk through a complete example of deploying a language model on Vast.ai using SkyPilot.

### Setting Up the Environment

First, install SkyPilot with Vast.ai support:

```bash
pip install -U "skypilot[vast]"
pip install "vastai-sdk>=0.1.12"
```

Configure your Vast.ai API key:

```bash
# Get your API key from https://vast.ai → Account → API Keys
echo "YOUR_VAST_API_KEY_HERE" > ~/.vast_api_key
chmod 600 ~/.vast_api_key

# Verify setup
sky check
```

Expected output:

```
Vast: enabled ✓
```

### Creating the YAML Configuration

Create a file named `deepseek-r1-inference.yaml`:

```yaml
resources:
  accelerators: H100:1
  cloud: vast
  image_id: docker:vllm/vllm-openai:latest
  disk_size: 100

run: |
  vllm serve deepseek-ai/DeepSeek-R1-0528-Qwen3-8B \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.9 \
    --reasoning-parser deepseek_r1
```

This configuration:

- Requests 1x H100 GPU on Vast.ai
- Uses the pre-built vLLM Docker image
- Allocates 100GB disk space for the model
- Serves the DeepSeek R1 model on port 8000

### Deploying the Service

Launch your service with SkyPilot:

```bash
sky launch deepseek-r1-inference.yaml --cluster deepseek-r1
```

This command will:

1. Find an available H100 GPU on Vast.ai
2. Provision the instance
3. Download the DeepSeek R1 model (~15GB)
4. Start the vLLM API server

The deployment takes 5-10 minutes. Monitor progress with:

```bash
sky logs deepseek-r1 --follow
```

Verify deployment:

```bash
sky status deepseek-r1
```

You should see `Status: UP` when ready.

### SSH Port Forwarding

Since direct port mapping isn't available, use SSH tunneling to access your service:

```bash
ssh -L 8000:localhost:8000 deepseek-r1
```

**Keep this terminal open!** The tunnel needs to stay active for API access.

### Testing the API

In a new terminal, test your deployed service:

```bash
# Test health endpoint
curl http://localhost:8000/health

# List available models
curl http://localhost:8000/v1/models
```

#### Python Client Example

```python
import openai

# Configure client for your local tunnel
client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="sk-fake-key"  # vLLM doesn't require real API key
)

# Make a simple request
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{
        "role": "user",
        "content": "Write a brief history of Los Angeles in one paragraph."
    }],
    max_tokens=300,
    temperature=0.7
)

print(response.choices[0].message.content)
```

### Managing Your Deployment

```bash
# Check status
sky status deepseek-r1

# Stop cluster (saves money)
sky stop deepseek-r1

# Start stopped cluster
sky start deepseek-r1

# Terminate completely
sky down deepseek-r1
```

## Conclusion

You now have a production-ready AI API running on Vast.ai's cost-effective H100 GPUs, managed through SkyPilot's streamlined interface. This combination provides enterprise-grade AI capabilities at a fraction of traditional cloud costs.

The integration between Vast.ai and SkyPilot represents a powerful solution for developers who need reliable, affordable GPU access without the complexity of manual infrastructure management. Start experimenting with your own AI workloads using this cost-effective stack!


Easily rent Vast.ai’s low-cost GPUs via SkyPilot for seamless AI, LLM, and batch job execution with automated provisioning, scaling, and cost optimization.

Vast.ai GPUs Can Now Be Rented Through SkyPilot


## Introduction

OpenAI has just released the GPT-OSS family of models, marking their return to open-weight model releases. The family includes GPT-OSS-120B (120 billion parameters) and GPT-OSS-20B (20 billion parameters), offering developers access to state-of-the-art language capabilities previously locked behind proprietary APIs.

GPT-OSS offers configurable reasoning capabilities with reasoning levels - from "low" for quick responses to "high" for complex problem-solving. The model uses OpenAI's harmony encoding system, which structures conversations and reasoning in a sophisticated format that enables fine-grained control over the model's thinking process.

This breakthrough means you can now:

-   Deploy OpenAI-quality models on your own infrastructure
-   Scale inference according to your exact needs

Vast.ai provides the perfect platform for running GPT-OSS models. With its marketplace of GPUs, you can choose the right hardware for your needs - whether running the efficient 20B model or the powerful 120B variant.

In this guide, we'll show you how to deploy both GPT-OSS models on Vast.ai using vLLM for optimized inference, with a focus on the 120B model. You'll learn how to interact with these models using the harmony encoding system for different reasoning levels.

## Setting Up the Environment

Before we can deploy our model, we need to set up our Vast.ai environment. First, install the Vast SDK:

```bash
# Install required packages
pip install --upgrade vastai
pip install --upgrade openai
pip install --upgrade openai-harmony
```

Set up your Vast API key (available from your [Account Page](https://cloud.vast.ai/account/)):

```bash
# Set your Vast.ai API key
export VAST_API_KEY="" # Your key here
vastai set api-key $VAST_API_KEY
```

## Choosing the Right Hardware

The GPT-OSS models have different hardware requirements:

-   **GPT-OSS-120B**: Requires an H100 GPU with 80GB VRAM
-   **GPT-OSS-20B**: Requires 16GB+ VRAM

For this guide, we'll demonstrate with the 120B model on H100. Let's search for suitable instances:

```bash
# Search for suitable GPU instances
vastai search offers " \
gpu_name = H100_SXM \
geolocation=US \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 120 \
rentable = true"
```

## Deploying GPT-OSS with vLLM

Now let's deploy our GPT-OSS instance using vLLM's optimized inference server. We'll use the latest vLLM image with special GPT-OSS support. The pip install commands are from the model's Hugging Face pages ([120B](https://huggingface.co/openai/gpt-oss-120b), [20B](https://huggingface.co/openai/gpt-oss-20b)).

### Deploying GPT-OSS-120B (H100 recommended)

```bash
# Deploy vLLM instance for 120B model
export INSTANCE_ID= # Insert instance ID from search results

vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 160  --onstart-cmd 'uv pip install --system --upgrade transformers kernels torch openai; uv pip install --system --pre vllm==0.10.1+gptoss --extra-index-url https://wheels.vllm.ai/gpt-oss/ --extra-index-url https://download.pytorch.org/whl/nightly/cu128; vllm serve openai/gpt-oss-120b --max-model-len 80000'
```

### Alternative: Deploying GPT-OSS-20B on A100

Here's how to deploy the 20B model on an A100 GPU:

```bash
# Deploy vLLM instance for 20B model
export INSTANCE_ID= # Insert instance ID from search results

vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 80  --onstart-cmd 'uv pip install --system --upgrade transformers kernels torch openai; uv pip install --system --pre vllm==0.10.1+gptoss --extra-index-url https://wheels.vllm.ai/gpt-oss/ --extra-index-url https://download.pytorch.org/whl/nightly/cu128; vllm serve openai/gpt-oss-20b'
```

After deployment, wait for the model to download and start serving. You can monitor the logs to see when it's ready. Once running, find your instance's IP address and port from the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console.

## Interacting with GPT-OSS Using Harmony Encoding

GPT-OSS uses OpenAI's harmony encoding system, which provides structured conversation formatting and enables different reasoning levels. Let's set up our client:

```python
from openai import OpenAI
from openai_harmony import (
    load_harmony_encoding,
    HarmonyEncodingName,
    Role,
    Message,
    Conversation,
    SystemContent,
    DeveloperContent,
)

# Your server details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize client
client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

# Load harmony encoding
enc = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)

## Simple Chat Function
def chat_gpt_oss(prompt, reasoning="medium"):
    """
    Simple chat with GPT-OSS
    reasoning: "low", "medium", or "high"
    """
    # Create conversation with harmony format
    conversation = Conversation.from_messages([
        Message.from_role_and_content(Role.SYSTEM, SystemContent.new()),
        Message.from_role_and_content(
            Role.DEVELOPER,
            DeveloperContent.new().with_instructions(f"Reasoning: {reasoning}")
        ),
        Message.from_role_and_content(Role.USER, prompt),
    ])

    # Get formatted prompt
    formatted_prompt = enc.render_conversation_for_completion(conversation, Role.ASSISTANT)

    # Make API call
    response = client.completions.create(
        model="openai/gpt-oss-120b",
        prompt=formatted_prompt,
        max_tokens=2000,
        temperature=0.7,
    )

    return response.choices[0].text


prompt = "Explain quantum computing to a 4th grader. Keep in concise."
```

## Comparing Reasoning Levels

One of GPT-OSS's most interesting features is its configurable reasoning levels. Let's see how the model responds differently to the same prompt with low and high reasoning:

### Low Reasoning Response

```python
result = chat_gpt_oss(prompt, "low")
print("----LOW REASONING----")
print(result)
```

Output:

```
----LOW REASONING----
analysisWe need to explain quantum computing to a 4th grader, concise. Use simple analogies, like bits as light switches, qubits as coins that can be heads, tails, or both. Keep short.assistantfinal**Quantum Computing in a Nutshell (for a 4th‑grader)**

1. **Regular computers = light switches**
   - A normal computer works with bits that are either **0** (off) or **1** (on), just like a light switch that's either down or up.

2. **Quantum computers = magic coins**
   - A quantum computer uses **qubits**.
   - Think of a qubit like a coin that can be **heads (0)**, **tails (1)**, **or both at the same time** while it's spinning in the air. That "both at once" is called **superposition**.

3. **Working together**
   - When you have many regular bits, you have to try every possible combination one after another.
   - With many qubits, they can be in many combinations **all at once** because they can be linked together in a special way called **entanglement**. It's like a group of magic coins that always know what the others are doing, even if they're far apart.

4. **Why it matters**
   - Because of superposition and entanglement, a quantum computer can solve some puzzles (like cracking codes or simulating molecules) **much faster** than a regular computer.

**Bottom line:** A regular computer is like a row of on/off switches; a quantum computer is like a bunch of spinning magic coins that can be many things at once, letting it try many solutions at the same time.
```

### High Reasoning Response

```python
result = chat_gpt_oss(prompt, "high")
print("----HIGH REASONING----")
print(result)
```

Output:

```
----HIGH REASONING----
analysisWe need to explain quantum computing to a 4th grader, concise. Use simple analogies, avoid jargon, but keep accurate. Should be short. Provide analogies like coins, dice, superposition, entanglement. Keep concise.assistantfinal**What is a quantum computer?**

Imagine you have a super‑smart robot that can solve puzzles much faster than a regular computer. A quantum computer is that robot, but instead of using normal "on/off" switches (bits) like a regular computer, it uses tiny, magical switches called **qubits**.

### Bits vs. Qubits
- **Bit (regular computer):** Like a tiny light that can be **off (0)** or **on (1)**.
- **Qubit (quantum computer):** Like a tiny spinning top that can be **0**, **1**, **or both at the same time** (this is called *superposition*).

### Why is that cool?
If you have 10 bits, they can show only one of 2¹⁰ = 1,024 possible patterns at a time.
If you have 10 qubits, they can be in all 2¹⁰ = 1,024 patterns **at once**! That lets a quantum computer try many solutions together.

### A Simple Analogy
Think of a maze puzzle:

- **Regular computer:** Sends a single mouse down the maze, step by step, until it finds the exit.
- **Quantum computer:** Sends a whole swarm of mice that can explore every possible path at the same time and tell you the exit almost instantly.

### Entanglement (Teamwork)
Sometimes two qubits become "best friends" and link together, no matter how far apart they are. Changing one instantly changes the other. This is called **entanglement**, and it lets quantum computers coordinate their work in a super‑efficient way.

### Bottom Line
A quantum computer uses qubits that can be 0, 1, or both at once, and they can be entangled with each other. This lets the machine look at many possibilities at the same time, solving certain problems way faster than ordinary computers.

So, it's like a super‑fast, super‑clever puzzle‑solver that can try lots of answers all at once!
```

## Comparing Low vs High Reasoning Outputs

The quantum computing explanations demonstrate the distinct capabilities of GPT-OSS-120B's reasoning levels. With low reasoning, the model produced a concise, structured explanation using simple analogies like "magic coins" and "light switches" - delivering clear concepts in a compact format suitable for quick understanding.

The high reasoning response revealed significantly more sophisticated thinking. The model provided detailed analogies like the maze puzzle with mice, explained mathematical concepts (2¹⁰ patterns), and structured the explanation with clear sections and formatting. Most notably, it offered multiple perspectives on the same concepts - from the spinning top analogy to the teamwork explanation of entanglement.

This difference in reasoning depth represents a significant advancement in model control. Rather than simply adjusting temperature or other parameters, GPT-OSS allows direct specification of how thoroughly the model should think through problems. The low reasoning approach prioritized efficiency while maintaining accuracy, while high reasoning prioritized comprehensive understanding with rich examples and detailed explanations.

## Conclusion

OpenAI's first open-source model thus introduces not just accessibility to advanced language capabilities, but a new paradigm for controlling model reasoning depth - enabling applications to dynamically adjust between quick responses and thorough analysis based on context and requirements.


Learn how to deploy and run OpenAI's GPT-OSS models on Vast.ai using vLLM, with configurable reasoning levels and harmony encoding system.

Running OpenAI's GPT-OSS on Vast.ai


At **Vast.ai**, we believe privacy and security are inextricably woven into our mission to equip customers worldwide with accessible, scalable GPUs. In honor of this commitment, we are pleased to announce that we have achieved **SOC 2 Type II certification** – an independent assurance that with us, your data is secure, reliably available, and continuously protected.

Vast.ai’s Enduring Security Commitment

Earlier this year, our [SOC 2 Type I audit](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) confirmed we have the right internal controls in place to meet trust services criteria for data security, integrity, and privacy. The Type II audit put those controls to the test, evaluating their operational effectiveness over an extended period.

Achieving **SOC 2 Type II certification** is independent validation that the security infrastructure we've built over 6+ years of trusted service continues to meet rigorous [compliance standards](https://vast.ai/article/security-and-compliance-at-vast-ai) in day-to-day execution. This achievement strengthens our ability to support the evolving security needs of enterprises, research institutions, and organizations building on our infrastructure.

Here's a quick look at SOC 2 Type II and what it entails.

### What Is SOC 2 Type II?

System and Organization Controls 2 (SOC 2\) audits – developed by the American Institute of Certified Public Accountants (AICPA) – protect customer data by rigorously evaluating an organization’s internal controls across security, confidentiality, privacy, and processing integrity benchmarks.

- **Type I** reports provide a control design snapshot at a single point in time.

- **Type II** reports go further, assessing how those controls perform over months of continuous operation – confirming they're both well-designed and consistently implemented.

Achieving both Type I and Type II certifications means your sensitive data is secure, available, and privately stored across all our systems. This establishes a foundation of enduring excellence aligned with rigorous industry standards for all customers, which we can tailor to your unique data security and regulatory compliance needs to scale infrastructure and further reduce risks.

After completing our initial 3-month Type II audit, we’ve launched a standard 12-month cycle to ensure we have no gaps in coverage. This two-phase strategy lets us move fast on improvements, then settle into an annual rhythm of verification. We are now in the 12-month cycle with continuous coverage year round.

### Vast.ai's Secure Cloud Offering

For customers with stricter security and compliance requirements, our [**Secure Cloud**](https://cloud.vast.ai/?secure_cloud=true) offering provides high-security GPU access through our certified datacenter partners. This environment, already backed by stringent operational and security controls, now benefits from the added assurance of a SOC 2 Type II audit across our platform infrastructure.

Our **[Trust Center](https://trust.vast.ai/?utm_source=blog)** offers a deeper look into our controls, compliance stance, FAQs and Subprocessors, and our other compliance certifications provide additional safeguards.

Why This Matters

Earning our **SOC 2 Type II certification** is another step forward in strengthening our foundation of trust and security that our customers rely on every day. It reflects both our adherence to proven industry practices and our ongoing commitment to transparency.

This achievement isn't the end of our compliance journey – it's part of an ongoing initiative. Vast.ai will undergo SOC 2 Type II audits every 12 months to ensure continual coverage.

Additionally, we've requested an [**SOC 3 report**](https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-3), which will provide a publicly available summary of our security posture for those who want to review our controls without accessing the full SOC 2 report.

### Our Ongoing Commitment

Achieving the **SOC 2 Type II certification** is a crucial step in our compliance roadmap as we continue to build trustworthy and dependable GPU infrastructure for our users. Alongside other security and [compliance measures](https://vast.ai/compliance) across cloud and AI services, such as our [SOC 2 Type I certification](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) and [Vulnerability Bounty Program](https://vast.ai/article/announcing-the-vast-ai-vulnerability-bounty-program), we are weaving security and trust throughout our platform for every member of our community.

Through this integrated approach, we’re dedicated to protecting the privacy and integrity of the data entrusted to us by developers, researchers, and enterprise teams around the world. We take this responsibility seriously as a distributed peer-to-peer platform powering global compute, and we look forward to sharing future updates as we continue to advance our security and compliance roadmap.

For more information on Vast.ai's comprehensive security and compliance stance, or to request a copy of our SOC 2 reports, please contact us at [compliance@vast.ai](mailto:compliance@vast.ai).

Thank you to our team at Vast.ai and our audit partners for helping us achieve this milestone.

We’re committed to protecting your data and upholding the trust you place in us, and we’ll keep you updated on our ongoing progress in our mission to provide powerful, dependable GPUs.


Vast.ai has achieved SOC 2 Type II certification, demonstrating our commitment to enterprise-grade security, data protection, and compliance. Learn about our comprehensive security controls, Secure Cloud offering, and ongoing compliance roadmap that ensures your data remains secure and protected.

Vast.ai Achieves SOC 2 Type II Certification


The landscape of AI inference is rapidly evolving, with organizations seeking cost-effective solutions that don't compromise on performance or control. While cloud providers offer convenient managed services, they often come with premium pricing and limited customization options. On the other hand, on-premises deployments provide control but require significant hardware investments.

A hybrid approach offers the best of both worlds: maintaining local control over your inference pipeline while leveraging cost-effective remote GPU resources. This blog post demonstrates how to implement such a system using LiteLLM as a local proxy and Vast.ai for remote GPU hosting.

## The Hybrid Architecture Advantage

This architecture combines local control with cost-effective cloud GPU access through two key components:

**LiteLLM** is a Python SDK and proxy server that provides a unified OpenAI-compatible interface for 100+ LLM APIs from different providers. It includes features like cost tracking, rate limiting, and logging capabilities when used as a proxy server.

**Vast.ai** is a GPU marketplace that offers access to over 20,000 GPUs from secure datacenters. According to their website, users can save up to 80% compared to traditional cloud services with their pay-as-you-go pricing model.

## What You'll Learn

In this blog post, we'll deploy a complete hybrid inference pipeline:

1. Deploy a vLLM server with the DeepSeek-R1 model on Vast.ai
2. Configure LiteLLM locally to proxy requests to the remote server
3. Test the complete pipeline using OpenAI client libraries
4. Demonstrate advanced reasoning capabilities

The result is a flexible, cost-effective inference setup that you control locally while leveraging remote GPU resources.

## Deploy LiteLLM + vLLM Pipeline on Vast.ai

Let's start by installing and configuring the Vast AI API client.

### Install Vast AI Client

Get your API key from the [Account Page](https://cloud.vast.ai/account/) in the Vast Console.

```bash
pip install --upgrade vastai
```

```bash
# Here we will set our api key
export VAST_API_KEY="" #Your key here
vastai set api-key $VAST_API_KEY
```

### Hardware Requirements

For our LiteLLM + vLLM pipeline, we need a GPU with 24GB+ RAM for the DeepSeek-R1 model weights and KV cache, plus a static IP address with direct ports for stable connections.

Let's search for suitable instances:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 24 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

## Deploy vLLM on Vast.ai

Choose an instance ID from the search results above and deploy using the Docker command below.

This deployment uses the vllm/vllm-openai:latest image to serve the deepseek-ai/DeepSeek-R1-0528-Qwen3-8B model with reasoning capabilities on port 8000:

```bash
export INSTANCE_ID="" #insert instance ID

vastai create instance $INSTANCE_ID \
  --image vllm/vllm-openai:latest \
  --env '-p 8000:8000' \
  --disk 60 \
  --args \
    --model deepseek-ai/DeepSeek-R1-0528-Qwen3-8B \
    --served-model-name deepseek \
    --max-model-len 4096 \
    --reasoning-parser qwen3 \
```

## Get Vast IP and Port

After deployment, go to the [Instances Tab](https://cloud.vast.ai/instances/) in the Vast AI Console and find your instance.

Click the IP address button at the top of the instance. A panel will show the IP address and forwarded ports:

```
Open Ports
XX.XX.XXX.XX:YYYY -> 8000/tcp
```

Note down VAST_IP (XX.XX.XXX.XX) and VAST_PORT (YYYY) for the next steps.

## Installing LiteLLM and Dependencies

Now let's install LiteLLM locally to proxy requests to our Vast.ai server:

```bash
pip install litellm
pip install 'litellm[proxy]'
```

### Configure LiteLLM

Set your Vast.ai instance details from the previous step:

```python
# Configure your Vast.ai instance details
VAST_IP = ""
VAST_PORT = ""
MODEL_NAME = "deepseek"

# Configure LiteLLM settings
LITELLM_PORT = 4000  # Change this if port 4000 is in use

# Create the API base URL
API_BASE = f"http://{VAST_IP}:{VAST_PORT}/v1"
print(f"Vast.ai API Base URL: {API_BASE}")
print(f"LiteLLM will run on port: {LITELLM_PORT}")
```

```python
# Write config file with variables
config_content = f"""model_list:
  - model_name: {MODEL_NAME}
    litellm_params:
      model: openai/{MODEL_NAME}
      api_base: {API_BASE}
      api_key: fake-key
general_settings:
  master_key: fake-key
"""

with open('litellm_config.yaml', 'w') as f:
    f.write(config_content)
    
print("Config file created with:")
print(f"- Model: {MODEL_NAME}")
print(f"- API Base: {API_BASE}")
```

```python
import subprocess
import time

# Kill any existing LiteLLM processes
subprocess.run("pkill -f litellm || true", shell=True)
time.sleep(1)

# Start LiteLLM
cmd = f"nohup litellm --config litellm_config.yaml --port {LITELLM_PORT} --host 0.0.0.0 > litellm.log 2>&1 &"
subprocess.run(cmd, shell=True)
time.sleep(5)

print(f"✅ LiteLLM running on port: {LITELLM_PORT}")
print(f"URL: http://localhost:{LITELLM_PORT}/v1")

# Save port for other cells
with open('.litellm_port', 'w') as f:
    f.write(str(LITELLM_PORT))
```

Output:
```
✅ LiteLLM running on port: 4000
URL: http://localhost:4000/v1
```

## Install OpenAI Python Client

Now install the OpenAI client to test our setup:

```bash
pip install openai
```

## Test the Pipeline

Let's test our complete LiteLLM + Vast.ai setup with a basic API call:

```python
from openai import OpenAI

client = OpenAI(
    api_key="fake-key",
    base_url=f"http://localhost:{LITELLM_PORT}/v1"
)

# Basic test
try:
    response = client.chat.completions.create(
        model="deepseek",
        messages=[
            {"role": "user", "content": "Hello! How are you?"}
        ],
        max_tokens=1000
    )
    print("Response:", response.choices[0].message.content)
    
except Exception as e:
    print(f"❌ Error: {e}")
```

Output:
```
Response: 
Hello! I'm just a friendly AI here, so I don't get tired or have feelings—but I'm fully ready and happy to help you! How are you doing today? Let me know if there's anything you'd like to chat about or need assistance with.
```

Perfect! Now let's test the reasoning capabilities of the DeepSeek model:

```python
try:
    response_1 = client.chat.completions.create(
        model="deepseek",
        messages=[
            {"role": "user", "content": "what is 8 * 7?, Give me your answer in step by step reasoning."}
        ],
        max_tokens=3000
    )
    print("Response:", response_1.choices[0].message.content)
    
except Exception as e:
    print(f"❌ Error: {e}")
```

Output:
```
Response: 
### Step-by-Step Reasoning for 8 * 7

Multiplication is a basic arithmetic operation that can be thought of as repeated addition. To find the product of 8 and 7, you can add the number 8 to itself 7 times (or equivalently, add the number 7 to itself 8 times). I'll use the repeated addition method with the number 8 to make it clear.

Start with the first 8:
- Add the second 8: 8 + 8 = 16
- Add the third 8: 16 + 8 = 24
- Add the fourth 8: 24 + 8 = 32
- Add the fifth 8: 32 + 8 = 40
- Add the sixth 8: 40 + 8 = 48
- Add the seventh 8: 48 + 8 = 56

As shown, after adding 8 seven times, the result is 56.

Alternatively, you can verify this with another method, such as multiplying 7 eight times or subtracting 8 from 8 * 8 (since 8 * 7 = (8 * 8) - 8 = 64 - 8 = 56). Both methods confirm the result.

**Final Answer:** 56
```

The pipeline is working correctly. You now have a complete LiteLLM + Vast.ai setup with local proxy control and remote GPU inference.

**Note**: For production deployments, consider using Docker containers for LiteLLM rather than the command-line approach shown here.

## Conclusion

This hybrid inference architecture demonstrates how modern AI deployment can balance cost, control, and performance. By combining LiteLLM's local proxy capabilities with Vast.ai's GPU marketplace, we've created a system that offers several key advantages:

**Cost Considerations**: Vast.ai's marketplace model and pay-as-you-go pricing can provide cost savings compared to traditional cloud providers, while running LiteLLM locally eliminates additional hosting costs for the proxy layer.

**Local Control**: Running LiteLLM locally provides control over request routing, logging, and configuration. The proxy server includes built-in features for cost tracking, rate limiting, and observability integrations.

**API Compatibility**: LiteLLM's OpenAI-compatible interface allows existing applications to integrate with minimal changes, while supporting multiple LLM providers through a unified API.

**Flexible Architecture**: This setup allows you to experiment with different models and providers while maintaining a consistent interface for your applications.

Whether you're building AI-powered applications, conducting research, or optimizing existing ML pipelines, this hybrid approach provides a practical path to cost-effective, controllable AI inference that doesn't sacrifice capability for economics.

Deploy hybrid AI inference using LiteLLM + Vast.ai. Run local OpenAI-compatible proxies with remote GPU models for scalable, cost-effective LLM performance.

Hybrid AI Inference: Local LiteLLM Proxy with Remote Vast.ai GPU

Deploy Llama 4, Qwen3, and Kimi K2 on Vast.ai in minutes. Use optimized templates like Oobabooga, vLLM, HuggingFace TGI, and Open WebUI—no setup hassle.

Modular MAX vs vLLM Performance Comparison on Vast.ai


Today, we’re announcing the launch of our Vast.ai Vulnerability Bounty Program. This is a new initiative we’ve created to help us improve, expand, and innovate our platform while making it more secure. Vast.ai’s mission is to provide the best GPUs for AI compute at accessible and affordable prices for everyone who needs it. With this bounty program, we’re inviting AI developers, researchers, and enthusiasts to collaborate with us. The launch of this program is in direct response to our community’s feedback, and we appreciate all the collaboration happening already.

### **Responsible Vulnerability Disclosure**

### We take security seriously and encourage responsible disclosure of any [vulnerabilities](https://vast.ai/vulnerability-disclosure-policy).

**What is the [Vast.ai](http://Vast.ai) Vulnerability Bounty program?**

The program provides a structured way to report bugs, suggest new features, or responsibly uncover vulnerabilities. Whether you’re a security researcher or a community builder, there’s an opportunity for you to do the right thing and be rewarded.

### **What Is In Scope?**

-   Vast.ai web console, REST API & billing flows
-   Provider Daemon code (host agent)
-   Match‑making & pricing engine
-   Default Docker & KVM isolation on reference images
-   GPU memory‑isolation / tenant breakout flaws

What is out of Scope?

-   User workloads & third‑party container images
-   Social‑engineering, physical security, or denial‑of‑service (DoS) tests
-   Brute‑force attacks against customer passwords or MFA
-   Any activity that violates applicable law, exports regulations, or provider Terms of Service

### If you discover a security issue, we ask that you:

-   Report it directly to our team at [security@vast.ai](mailto:security@vast.ai?subject=Vulnerability%20Bounty%20Program%20Inquiry).

-   Allow us a reasonable time to address the issue before any public disclosure.

### In return, we commit to:

-   Acknowledging your report promptly and keeping you informed of progress.

-   Recognizing your efforts with a form of bounty rewards, platform credit, or joining our hall of fame.

### **How to Get Started?**

1. Join the [Vast.ai Discord](https://discord.gg/NCgfu5BrdQ) to connect with our team and community.

2. Submit your bounty contributions and start hunting.

Please review [the policy](https://vast.ai/vulnerability-disclosure-policy) for the Rules of Engagement and specifics on scope, which will be updated periodically.


Help secure Vast.ai and earn rewards. Join our Vulnerability Bounty Program to report bugs, suggest features, and collaborate with the AI infrastructure community.

Announcing the Vast.ai Vulnerability Bounty Program


In Part 1 of this series, we created a quantized version of the Qwen3-8B model (Qwen3-8B-W8A8) using LLM-Compressor with 8-bit weight and activation quantization. In this notebook, we'll demonstrate how to deploy and compare this quantized model against its full precision counterpart (Qwen3-8B) using Vast.ai. We'll examine both deployment processes and compare the outputs to assess any quality differences between the models while highlighting the efficiency gains from quantization.

## Key benefits of quantized models:

- Reduced memory footprint (approximately 1.7x smaller in our case)
- Lower inference latency
- Decreased computational requirements
- More affordable deployment options

Let's start by exploring how to deploy our quantized model on Vast.ai.

## Deploying our Model on Vast.ai

First, we will install the Vast.ai SDK and input our API key. This allows us to interact with the Vast.ai platform programmatically and manage our GPU instances.

```bash
#In an environment of your choice
pip install --upgrade vastai
```

```bash
# Here we will set our api key
export VAST_API_KEY="VAST_API_KEY" #Your key here
vastai set api-key $VAST_API_KEY
```

Next, we'll search for an appropriate GPU instance to serve our quantized model. Since this is a W8A8 quantized version of Qwen3-8B, we can use a machine with less VRAM than would be required for the full model. We're looking for machines with at least 24GB of VRAM, which is sufficient for our 8-bit quantized model and its context window:

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 24 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 20 \
rentable = true"
```

Once we've identified a suitable instance from the search results, we'll deploy our quantized model to that instance. We're using the vLLM Docker image with an OpenAI-compatible API, which makes it easy to serve and interact with our model.

Note that we're deploying the W8A8 quantized version of Qwen3-8B that was created and uploaded to Hugging Face in Part 1. Please replace `your-hf-username` with your actual Hugging Face username to use your compressed model:

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 20 --args --model your-hf-username/Qwen3-8B-W8A8
```

## Calling Our Model Using the OpenAI SDK

Now that our model is deployed and running on Vast.ai, we'll set up a client to interact with it. The vLLM server exposes an OpenAI-compatible API, allowing us to use the OpenAI SDK to send requests to our model.

First, we need to install the OpenAI SDK if you haven't already:

```bash
pip install openai
```

Then, we use the OpenAI SDK and the `VAST_IP_ADDRESS` and `VAST_PORT` from our instance to call our model.

Also add your hugginface username `HF_USERNAME` to call the model we saved to hugginface.

```python
from openai import OpenAI

VAST_IP_ADDRESS = "VAST_IP_ADDRESS"
VAST_PORT = "VAST_PORT"

HF_USERNAME="HF_USERNAME"

# Create a client instance pointing to the vLLM server
client = OpenAI(
    api_key="dummy-key",  # vLLM doesn't require a real API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"  # Point to your vLLM server
)

# Call the model
response = client.chat.completions.create(
    model=f"{HF_USERNAME}/Qwen3-8B-W8A8",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What are the major applications of deep learning?"}
    ],
    temperature=0.7,
    max_tokens=1000
)

# Print the response
print(response.choices[0].message.content)
```

Output:

```
<think>
Okay, the user is asking about the major applications of deep learning. I need to provide a comprehensive yet clear answer. Let me start by recalling the different areas where deep learning is commonly used.

First, computer vision comes to mind. That includes image recognition, object detection, and maybe even image generation. Then there's natural language processing, like chatbots, translation, and sentiment analysis.

I should also mention speech recognition, which is used in virtual assistants. Autonomous vehicles rely on deep learning for perception tasks. Healthcare is another big area, such as medical imaging analysis and drug discovery.

Finance uses deep learning for fraud detection and algorithmic trading. Recommender systems in e-commerce and streaming services are also important. Maybe touch on generative models like GANs and diffusion models for creating art or text.

Wait, I should categorize these applications to make it organized. Let me list them under different headings. Also, I need to ensure I don't miss any major areas. Oh, maybe mention robotics and control systems too.

I should check if there are any recent advancements or emerging fields. For example, deep learning in gaming, like reinforcement learning for game AI. Also, maybe something about anomaly detection in various industries.

Need to keep the explanation straightforward without too much jargon. Make sure each application is explained briefly but clearly. Avoid going into too much technical detail unless necessary.

Let me structure the answer with clear sections for each major application. Start with an introduction, then list each area with a brief description. Conclude by summarizing the versatility of deep learning.

Wait, the user might be looking for both current and potential future applications. Should I include that? Maybe a short note at the end about ongoing research and future possibilities.

Also, check for any overlaps or if some applications are better categorized under specific fields. For example, NLP and speech recognition are both related to language processing.

I think that's a solid structure. Now, I'll draft the answer accordingly, making sure it's informative and covers all key areas without being too verbose.
</think>

Deep learning, a subset of machine learning, has revolutionized numerous fields due to its ability to model complex patterns in data. Here are its major applications across various domains:

---

### **1. Computer Vision**
- **Image Recognition**: Identifying objects, scenes, and patterns in images (e.g., facial recognition, medical imaging analysis).
- **Object Detection**: Locating and classifying multiple objects in a single image (e.g., autonomous vehicles, surveillance systems).
- **Image Generation**: Creating new images using Generative Adversarial Networks (GANs) or diffusion models (e.g., AI art tools).
- **Video Analysis**: Action recognition, video summarization, and scene understanding.

---

### **2. Natural Language Processing (NLP)**
- **Text Generation**: Writing articles, stories, or code using models like GPT or BERT.
- **Translation**: Real-time language translation (e.g., Google Translate).
- **Sentiment Analysis**: Determining emotions in text (e.g., social media monitoring).
- **Chatbots & Virtual Assistants**: Interactive AI assistants (e.g., Siri, Alexa, customer service bots).

---

### **3. Speech Recognition & Synthesis**
- **Voice Assistants**: Converting speech to text (e.g., Alexa, Google Assistant).
- **Speech-to-Text**: Transcribing audio for accessibility or transcription services.
- **Text-to-Speech**: Synthesizing natural-sounding speech (e.g., audiobooks, virtual characters).

---

### **4. Autonomous Systems**
- **Self-Driving Cars**: Perceiving environments, navigating, and making real-time decisions.
- **Robotics**: Controlling robotic arms, navigation, and object manipulation.
- **Industrial Automation**: Predictive maintenance and quality control in manufacturing.

---

### **5. Healthcare**
- **Medical Imaging**: Diagnosing diseases (e.g., tumors in X-rays, MRIs) using CNNs.
- **Drug Discovery**: Accelerating the development of new medications through molecular modeling.
- **Personalized Medicine**: Predicting patient outcomes based on genetic data and medical history.

---

### **6. Finance**
- **Fraud Detection**: Identifying unusual transaction patterns.
- **Algorithmic Trading**: Predicting market trends and executing trades.
- **Risk Management**: Assessing credit risk and financial stability.

---

### **7. Recommender Systems**
- **E-commerce**: Personalized product recommendations (e.g., Amazon, Netflix).
- **Streaming Services**: Content suggestions based on user preferences.
- **Social Media**: Curating news feeds and trending topics.

---

### **8. Gaming & Entertainment**
- **Game AI**: Creating intelligent opponents or NPCs (e.g., AlphaGo, Dota 2 bots).
- **Content Creation**: Generating music, art, or video using deep learning models.

---
```

## Compare with full `Qwen/Qwen3-8B` Model

Now that we've seen the output from our quantized model, let's deploy and test the full precision version of the same model for comparison. This will help us evaluate any potential quality differences and understand the tradeoffs of quantization.

First, we need to find a more powerful machine with significantly more GPU memory. The full precision model requires approximately 3-4x more VRAM than its quantized counterpart. Let's search for instances with at least 80GB of VRAM:

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 80 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 40 \
rentable = true"
```

Now we'll deploy the full precision model. Notice the differences in our configuration compared to the quantized model:

1. We're requiring more disk space (40GB vs 20GB)
2. We're using a much more powerful GPU with significantly more VRAM
3. We're deploying the original model from Qwen directly (Qwen/Qwen3-8B)

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 40 --args --model  Qwen/Qwen3-8B
```

With the full precision model deployed, let's query it with the same prompt we used for the quantized model to perform a direct comparison of their outputs:

```python

VAST_IP_ADDRESS = "VAST_IP_ADDRESS"
VAST_PORT = "VAST_PORT"

# Create a client instance pointing to the vLLM server
client = OpenAI(
    api_key="dummy-key",  # vLLM doesn't require a real API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"  # Point to your vLLM server
)

# Call the model
response = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What are the major applications of deep learning?"}
    ],
    temperature=0.7,
    max_tokens=1000
)

# Print the response
print(response.choices[0].message.content)
```

```
<think>
Okay, the user is asking about the major applications of deep learning. Let me start by recalling what I know. Deep learning is a subset of machine learning that uses neural networks with many layers. It's been really popular in various fields. I need to list the main areas where it's applied.

First, computer vision comes to mind. Image recognition, object detection, maybe even things like facial recognition. Then there's natural language processing, like chatbots, translation services, text generation. Speech recognition is another big one, like virtual assistants.

What else? Autonomous vehicles use deep learning for perception tasks, like identifying objects on the road. Healthcare is another area, maybe medical imaging analysis, predicting diseases. Then there's recommendation systems used by companies like Netflix or Amazon.

Wait, what about robotics? Deep reinforcement learning is used in robotics for control systems. Also, maybe gaming, like in AlphaGo. Financial services for fraud detection or algorithmic trading.

I should also mention some more specialized areas like generative models for creating images or music, and maybe something about NLP like sentiment analysis. Oh, and maybe mention that these applications are driven by the ability of deep learning to handle large amounts of data and complex patterns.

Wait, the user might be looking for a comprehensive list, so I should structure it into categories. Let me check if I missed any major areas. Maybe cybersecurity? Or maybe something like anomaly detection in various industries. Also, maybe mention the difference between deep learning and traditional machine learning in terms of application.

I need to make sure each application is clearly explained with examples. Let me organize them into sections. Computer Vision, NLP, Speech Recognition, Autonomous Vehicles, Healthcare, Recommendation Systems, Robotics, Gaming, Financial Services, Cybersecurity, Generative Models, and maybe others.

I should also note that deep learning is used in both supervised and unsupervised tasks. Oh, and maybe mention some specific models like CNNs for images, RNNs or Transformers for NLP. But the user might not need the technical details, just the applications.

Wait, the user might be a student or someone new to the field, so keeping it clear and concise is important. Avoid jargon where possible. Also, make sure to highlight the impact of each application.

Let me make sure I cover all major areas. Maybe add something about drug discovery in healthcare. Or maybe mention that deep learning is used in weather prediction or climate modeling.

Hmm, I think that's a good list. Let me structure it with bullet points and explanations for each application. Also, maybe conclude with a note on the versatility and impact of deep learning.
</think>

Deep learning, a subset of machine learning, has revolutionized numerous fields due to its ability to model complex patterns and process large datasets. Here are the **major applications** of deep learning:

---

### **1. Computer Vision**
- **Image Recognition**: Identifying objects, people, and scenes in images (e.g., Google Photos, medical imaging).
- **Object Detection**: Locating and classifying objects in real-time (e.g., autonomous vehicles, security cameras).
- **Facial Recognition**: Used in biometric authentication (e.g., smartphones, border control).
- **Image Generation**: Creating realistic images (e.g., GANs for art, style transfer).

---

### **2. Natural Language Processing (NLP)**
- **Text Generation**: Writing articles, stories, or code (e.g., chatbots, AI assistants like Siri).
- **Machine Translation**: Languages like Google Translate use neural networks for real-time translation.
- **Sentiment Analysis**: Analyzing emotions in text (e.g., social media monitoring, customer feedback).
- **Speech Recognition**: Converting speech to text (e.g., voice assistants, transcription services).

---

### **3. Autonomous Systems**
- **Self-Driving Cars**: Detecting pedestrians, traffic signs, and obstacles (e.g., Tesla Autopilot, Waymo).
- **Robotics**: Controlling robotic arms and navigation systems (e.g., industrial robots, drones).
- **Reinforcement Learning**: Training agents for tasks like game playing (e.g., AlphaGo, Dota 2).

---

### **4. Healthcare**
- **Medical Imaging**: Diagnosing diseases from X-rays, MRIs, or CT scans (e.g., cancer detection).
- **Drug Discovery**: Accelerating the development of new drugs by predicting molecular interactions.
- **Personalized Medicine**: Tailoring treatments based on patient data (e.g., genomics analysis).

---

### **5. Recommendation Systems**
- **E-commerce**: Product recommendations (e.g., Amazon, Netflix).
- **Streaming Services**: Content suggestions (e.g., Spotify, YouTube).
- **Social Media**: Curating news feeds (e.g., Facebook, Instagram).

---

### **6. Finance**
- **Fraud Detection**: Identifying unusual
```

## Conclusion: Comparing Model Outputs and Size Tradeoffs

After deploying and testing both the quantized model (W8A8) and the full precision model, we can make several key observations:

### Model Size and Resource Requirements

- **Full Precision Model (Qwen3-8B)**:

  - Required ~80GB+ VRAM GPU
  - Needed 40GB disk space
  - Model size: ~16GB (FP16)

- **Quantized Model (Qwen3-8B-W8A8)**:
  - Required only 24GB VRAM GPU (~3.3x less memory)
  - Needed only 20GB disk space
  - Model size: ~9.5GB (8-bit weights and activations)

### Output Quality Comparison

Looking at the outputs from both models, we can observe:

1. **Structural Similarity**: Both models organized their responses with similar categories (Computer Vision, NLP, Healthcare, etc.) and maintained good formatting with markdown.

2. **Content Quality**: Both models produced high-quality, relevant content with appropriate examples for each application domain. The specific details varied slightly, but the core information was consistent.

3. **Reasoning Process**: Both models showed similar thinking processes in their `<think>` sections, demonstrating that the quantized model retained the structured reasoning capabilities of the original model.

4. **Minimal Quality Degradation**: The quantized model showed negligible degradation in output quality compared to the full precision model, despite being significantly smaller.

### Key Takeaways

1. **Deployment Accessibility**: The W8A8 quantized model makes deployment much more accessible, requiring far less expensive hardware while maintaining output quality.

2. **Cost Efficiency**: Using the quantized model can significantly reduce cloud GPU costs by allowing deployment on less expensive hardware.

3. **Production Readiness**: 8-bit quantization appears to be production-ready for this type of model, offering an excellent balance between efficiency and performance.

4. **Use Case Fit**: For most general text generation and understanding tasks, the quantized model appears to be a suitable replacement for the full precision model.

This notebook demonstrates that weight and activation quantization (W8A8) using LLM Compressor is an effective approach for deploying large language models in resource-constrained environments without significant quality compromises.

Vast.ai provides an excellent platform for this workflow, allowing us to use high-end GPUs for model compression and more cost-effective GPUs for efficient inference.


Compare quantized vs. full Qwen3-8B models deployed on Vast.ai. See quality vs. efficiency tradeoffs using W8A8 compression with LLM-Compressor and vLLM.

Using LLM-Compressor to Quantize Qwen3-8B on Vast.ai (Part 2)


As AI models grow increasingly powerful, they're also becoming increasingly expensive to deploy. The latest language models require massive amounts of GPU memory and computational resources, putting them out of reach for many teams and use cases. Model compression has emerged as a critical technique for making these models more accessible while maintaining their performance.

Today, we'll explore how to compress large language models using LLM-Compressor and deploy them cost-effectively on Vast.ai. We'll walk through the complete process of taking a 16GB model and reducing it to approximately 9.5GB while preserving quality—making deployment significantly more affordable and accessible.

## The Tools: LLM-Compressor and Vast.ai

### LLM-Compressor: Making Models Leaner

The vLLM team has developed [LLM-Compressor](https://github.com/vllm-project/llm-compressor) as a comprehensive solution for reducing model size without sacrificing performance. Unlike other compression tools that focus on a single technique, LLM-Compressor offers a complete toolkit for optimization.

What makes it particularly powerful:

- **Comprehensive approach**: Combines quantization, pruning, and distillation techniques that can be mixed and matched
- **Production-ready**: Built specifically for deployment scenarios with seamless vLLM integration
- **Developer-friendly**: The one-shot API handles the complex compression pipeline automatically
- **Proven techniques**: Implements industry-standard methods like GPTQ, AWQ, and SmoothQuant

### Vast.ai: Affordable GPU Power

Traditional cloud providers can make GPU-intensive tasks prohibitively expensive. Vast.ai changes this equation by creating a marketplace where you can access high-end GPUs at competitive rates—often significantly cheaper than major cloud platforms.

For model compression workflows, Vast.ai offers several advantages:

- **Flexible pricing**: Pay-per-hour model means you only pay for compression time, not idle resources
- **Hardware variety**: Choose from different GPU types based on your specific memory and compute requirements
- **Quick deployment**: Get instances running in minutes without complex setup procedures
- **Community-driven**: Access to a wider range of hardware options through the marketplace model

## In this Tutorial

This tutorial demonstrates how to:

1. Compress the `Qwen3-8B` model using LLM-Compressor with 8-bit weight and activation quantization (W8A8)
2. Upload the compressed `Qwen3-8B-W8A8` model to Hugging Face for easy sharing and deployment

In the second part of this series, we'll show how to deploy the compressed `Qwen3-8B-W8A8` model on Vast.ai for efficient inference and compare its performance with the full-precision model.

## GPU Requirements

Model compression requires significant computational resources, especially for larger models. For compressing the Qwen3-8B model, we recommend the following specifications:

### Minimum Requirements:

- **GPU Memory**: 30GB+ VRAM (e.g., A5000, A6000, RTX 4090, V100-32GB)
- **GPU Compute**: CUDA-compatible GPU with compute capability 7.0+
- **System RAM**: 64GB recommended (minimum 32GB)
- **Storage**: 80GB free disk space for model files and datasets
- **CUDA Version**: 11.8 or newer

### Why these requirements?

- The full Qwen3-8B model requires approximately 16GB just to load in FP16 format
- During compression, we need memory for both the original model and compression overhead
- Calibration datasets and intermediate files require additional storage space
- Model compilers and quantization tools have their own memory overhead

For larger models (12B+), you would need even more VRAM (40GB+) or techniques like model sharding.

## Renting an Instance on Vast.ai

1. Ensure that you have a Vast.ai account. If not, sign up at [vast.ai](https://vast.ai)
2. Go to the Vast Templates in the Console https://cloud.vast.ai/templates/
3. Select the `PyTorch (CuDNN Runtime)` Template - this provides a pre-configured environment with PyTorch and CUDA
4. Filter for an instance with the following specifications for optimal performance:
   - 1 GPU with high VRAM (A5000, A6000, RTX 4090, or similar)
   - 30GB+ of VRAM (required for full model loading plus quantization overhead)
   - 80GB+ of storage (for model weights, checkpoints, and datasets)
   - 32GB+ of system RAM (64GB recommended for smoother operation)
5. Select an instance and click rent - hourly pricing will vary based on the selected hardware
6. Install the Vast TLS certificate in your browser to access the notebook server securely https://docs.vast.ai/instances/jupyter#1SmCz
7. Go to your Instances https://cloud.vast.ai/instances/ and click "Open" to access the Jupyter server on your instance
8. Upload this notebook to the server through the Jupyter interface

## Setup and Installation

First we will install and import our required packages. The cell below installs:

- Transformers: For loading and saving the model in Hugging Face format
- PyTorch: The deep learning framework that powers our model
- Hugging Face Hub: For uploading our compressed model
- LLM-Compressor: The toolkit we'll use for quantization

```bash
# Install required packages
# Using specific versions to ensure compatibility
pip install transformers==4.51.3  # Recent Transformers version with good Qwen3 support
pip install torch==2.7.0          # PyTorch version compatible with current CUDA drivers
pip install huggingface_hub       # Core Hugging Face Hub library
pip install huggingface_hub[hf_xet]  # Extension for handling large file transfers
pip install huggingface_hub[cli]     # Command-line interface for HF Hub
pip install llmcompressor            # The model compression toolkit
```

Next, import the necessary libraries:

```python
# Import necessary libraries
import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from llmcompressor.modifiers.quantization import GPTQModifier  # For weight quantization
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier  # For activation quantization preprocessing
from llmcompressor import oneshot  # Simplified API for end-to-end compression
```

## Define Model and Output Path

Next, we'll define our model variables and prepare for downloading the `Qwen/Qwen3-8B` model.

- `MODEL_ID`: The Hugging Face model ID for the original model
- `OUTPUT_MODEL_ID`: A name for our compressed model version
- `OUTPUT_DIR`: Local directory where the compressed model will be saved

```python
# Define model path and output directory
MODEL_ID = "Qwen/Qwen3-8B"  # Original model on Hugging Face
OUTPUT_MODEL_ID = "Qwen3-8B-W8A8"  # Our compressed model name
OUTPUT_DIR = "./Qwen3-8B-W8A8"  # Local directory for saving
os.makedirs(OUTPUT_DIR, exist_ok=True)  # Create output directory if it doesn't exist
```

Download the model files using the snapshot_download function:

```python
from huggingface_hub import snapshot_download

# Download the model files to local directory
# This is a more reliable way to download large models compared to the standard AutoModel approach
# This will download all model files (~16GB for Qwen3-8B) and may take some time depending on connection speed
snapshot_download(
    repo_id="Qwen/Qwen3-8B",
    local_dir_use_symlinks=False  # Set to True to use symlinks instead of copying files (saves space but may cause issues)
)
```

## Configure Quantization

Now we'll define our compression recipe, which consists of two steps:

1. **SmoothQuant**: A technique that redistributes the quantization difficulty between weights and activations, making activations easier to quantize with minimal accuracy loss

2. **GPTQ**: A weight quantization method specifically designed for large language models that uses a reconstruction approach to minimize the impact of quantization on model quality

We're using W8A8 quantization, which means:

- W8: Weights stored in 8-bit precision (instead of FP16/FP32)
- A8: Activations computed in 8-bit precision at inference time

This will reduce the model size by approximately 1.7x and potentially increase inference speed.

```python
# Create quantization recipe
# A recipe is a sequence of compression techniques applied in order
recipe = [
    # First apply SmoothQuant to make activations easier to quantize
    # The smoothing_strength parameter controls how aggressively to shift quantization difficulty
    # from activations to weights (higher = more shifting, 0.8 is a good balance)
    SmoothQuantModifier(smoothing_strength=0.8),

    # Then apply GPTQ for weight and activation quantization
    GPTQModifier(
        targets="Linear",  # Apply to all linear layers in the model
        scheme="W8A8",    # 8-bit weights and activations (other options: W4A16, W8A16, etc.)
        ignore=["lm_head"] # Don't quantize the language modeling head (important for output quality)
    )
]
```

## Apply Quantization with oneshot API

Now we'll apply our compression recipe to the model using the oneshot API.

The oneshot API handles the entire compression workflow:

1. Loading the model from Hugging Face or local path
2. Creating a calibration dataset (or using an existing one)
3. Running calibration to determine optimal quantization parameters
4. Applying the compression recipe
5. Saving the compressed model

We're using the `open_platypus` dataset for calibration, which consists of diverse open-source technical questions and answers. This helps ensure the model maintains accuracy across a range of technical topics.

**Note**: This cell will take a significant amount of time (30-60 minutes) and GPU memory to run.

```python
print(f"Starting compression of {MODEL_ID}...")

# Apply quantization using oneshot API
oneshot(
    model=MODEL_ID,               # HF model ID or local path to load the model from
    dataset="open_platypus",      # Calibration dataset - preset dataset with technical Q&A
    recipe=recipe,                # Our compression recipe defined above
    output_dir=OUTPUT_DIR,        # Where to save the compressed model
    max_seq_length=2048,          # Maximum sequence length for calibration samples
    num_calibration_samples=512,  # Number of samples to use for calibration (more is better but slower)
)
```

## Test the Compressed Model

After compression, it's important to verify that the model still performs reasonably well. This cell loads the compressed model and runs a simple inference test.

We'll evaluate the model on a prompt about AI technology to see how well it generates coherent and relevant text. This helps us verify that the quantization didn't significantly degrade model quality.

Note that a proper evaluation would involve more systematic testing across multiple prompts and metrics.

```python
# Load the compressed model from our output directory
tokenizer = AutoTokenizer.from_pretrained(OUTPUT_DIR)
model = AutoModelForCausalLM.from_pretrained(
    OUTPUT_DIR,
    device_map="auto",      # Automatically place model on available devices (CPU/GPU)
    torch_dtype="auto"      # Use the model's native precision (will be INT8 for quantized weights)
)

# Simple test to verify the model still works
prompt = "AI technology is transforming industries by"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate text with some randomness for creativity
with torch.no_grad():  # Disable gradient calculation for inference
    outputs = model.generate(
        inputs["input_ids"],
        max_new_tokens=100,     # Generate up to 100 new tokens
        do_sample=True,         # Use sampling instead of greedy decoding
        temperature=0.7         # Control randomness (lower = more deterministic)
    )

# Decode the generated text and print it
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Prompt: {prompt}")
print(f"Generated text: {generated_text}")
```

## Upload the Model to Hugging Face

Now that we have successfully compressed the model, we'll upload it to Hugging Face Hub for easy sharing and deployment.

First, we'll set up the necessary credentials and repository information:

```python
from huggingface_hub import HfApi, login
import os

# Define variables for Hugging Face upload
# You'll need to replace these with your actual Hugging Face information
HF_USERNAME = "HF_USERNAME" # Add your Hugging Face username here
HF_TOKEN = "HF_TOKEN"       # Replace with your actual Hugging Face token (from settings page)
HF_MODEL_ID = f"{HF_USERNAME}/{OUTPUT_MODEL_ID}"  # Full repository path on Hugging Face
```

Next, we'll create a proper model card (README.md) with information about our compressed model. This helps others understand what the model is, how it was compressed, and how to use it:

```python
# Create a detailed model card describing the compressed model
model_card_content = f"""---
language:
- en
- zh
library_name: transformers
license: other
datasets:
- open_platypus
---

# Qwen3-8B-W8A8

This is a compressed version of [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) using llm-compressor with the following optimizations:
- 8-bit weight quantization using GPTQ
- 8-bit activation quantization
- SmoothQuant pre-processing

## Model Details

- **Original Model**: Qwen/Qwen3-8B
- **Quantization Method**: GPTQ + SmoothQuant (W8A8)
- **Compression Libraries**: [llm-compressor](https://github.com/vllm-project/llm-compressor)
- **Calibration Dataset**: open_platypus (512 samples)
- **Optimized For**: Inference with vLLM
"""

# Write README.md to model directory
readme_path = os.path.join(OUTPUT_DIR, "README.md")
with open(readme_path, "w") as f:
    f.write(model_card_content)
```

Finally, we'll upload the compressed model to Hugging Face Hub. This process may take some time depending on your internet connection speed, as the model files are large (approximately 9.5GB even after compression):

```python
# Login to Hugging Face with your token
login(token=HF_TOKEN)

# Initialize the Hugging Face API
api = HfApi()

# Create the repository (if it doesn't exist)
api.create_repo(
    repo_id=HF_MODEL_ID,
    exist_ok=True,        # Don't error if repo already exists
    private=False,        # Set to True if you want a private repository
)

# Upload the model files
print(f"Starting upload of {OUTPUT_DIR} to {HF_MODEL_ID}...")
api.upload_folder(
    folder_path=OUTPUT_DIR,
    repo_id=HF_MODEL_ID,
    repo_type="model",
    ignore_patterns=["*.tmp", "*.log", "__pycache__"],  # Files to ignore during upload
    commit_message="Upload compressed Qwen3-8B with W8A8 quantization",
)

print(f"Model successfully uploaded to: https://huggingface.co/{HF_MODEL_ID}")
```

## Wrapping Up: From 16GB to 9.5GB Without Breaking the Bank

We've just walked through a transformation that would have seemed impossible a few years ago—taking a sophisticated language model and cutting its resource requirements by approximately 40% while preserving its capabilities.

### What We Accomplished

Our compression workflow delivered results across multiple dimensions:

**Technical achievements:**

- Reduced model footprint from 16GB to 9.5GB using W8A8 quantization
- Maintained output quality through careful calibration with technical datasets
- Created a production-ready model compatible with modern inference engines

**Economic impact:**

- Enabled deployment on more affordable GPUs with less VRAM requirements
- Reduced cloud inference costs through lower resource requirements
- Made advanced language models accessible to teams with limited budgets

**Operational benefits:**

- Faster model loading and reduced memory pressure during inference
- Better resource utilization allowing for higher throughput or multi-model serving
- Simplified deployment pipeline through automated compression workflows

### The Bigger Picture

Model compression offers an alternative approach to AI deployment. Rather than accepting ever-increasing resource requirements, we can actively optimize models for real-world constraints. The combination of LLM-Compressor's sophisticated algorithms and Vast.ai's accessible infrastructure makes this optimization practical for any team.

### Coming Up Next

In Part 2, we'll put our compressed model to the test. We'll deploy both our optimized Qwen3-8B-W8A8 model and the original full-precision version on Vast.ai, then conduct detailed comparisons of their performance, cost, and output quality. You'll see exactly how much you can save without compromising on results—and learn when compression makes the most sense for your specific use case.


Compress LLMs like Qwen3-8B with LLM-Compressor and deploy on Vast.ai. Cut model size ~40% while preserving quality—save cost, speed up inference, and scale smartly.

Model Compression with LLM-Compressor and Deployment on Vast.ai (Part 1)


You wouldn't buy a car without taking it for a test drive, so why deploy a large language model (LLM) without benchmarking it first?

When you're running LLMs at scale, you need more than just theoretical performance data. You're going to want to know how they actually perform under real conditions – throughput, latency, and hardware efficiency all matter. Skipping this step can result in higher costs and wasted resources.

Benchmarking early helps you avoid inefficiencies down the line, and it shouldn't take hours to get started. That's where **vLLM** comes in.

### A Smarter Way to Serve and Benchmark LLMs

vLLM is an open-source library that optimizes the inference and serving of LLMs, and it [simplifies](https://blog.vllm.ai/2023/06/20/vllm.html) the benchmarking process with an efficient architecture built around the attention algorithm PagedAttention. This mechanism reduces memory waste and improves throughput by storing attention data non-contiguously.

When fast and efficient LLM inference is crucial, understanding your model's performance starts with benchmarking. With vLLM, you can spin up a high-performance inference server and start collecting those benchmarks in minutes.

To make things even easier, you don't need to spin up your own infrastructure from scratch. Vast.ai provides an ideal platform to leverage vLLM, offering high-performance GPUs at a fraction of the cost of traditional cloud providers. With Vast, it's easy to find the right hardware for your model, and with simple Docker integration, you can get up and running with vLLM quickly.

This guide demonstrates how to benchmark any LLM using vLLM on Vast.ai, in just a few minutes.

### Getting Started: Benchmarking LLMs with vLLM

This guide walks you through how to **serve a large language model using vLLM** and run a benchmark test to evaluate performance. It’s using a **H100** **GPU server from [Vast.ai](https://vast.ai/)** with the **"PyTorch" template**. In this example, we’ll use the model `meta-llama/Llama-3.1-8B-Instruct`, but you can use **any Hugging Face-supported model** compatible with vLLM.

**1\. Install required Python packages**

First, install the **vLLM** library along with **Pandas** and Hugging Face’s **datasets** library. For example, you can use pip in your Python environment:

```shell
pip install vllm pandas datasets
```

This installs vLLM and the other libraries for data handling.

**2\. Log in to Hugging Face**

To access the Meta Llama 3.1 model, you need a Hugging Face API token. Then run:

```shell
huggingface-cli login
```

This will prompt you to paste your Hugging Face **access token** (from your Hugging Face account settings). After entering the token, the CLI saves it so vLLM and other tools can download the model.

## **3\. Start the vLLM server**

Now start the vLLM server hosting the Llama 3.1 8B Instruct model. vLLM’s OpenAI-compatible server listens on localhost port 8000 by default. Use the `vllm serve` command with **nohup** to run it in the background and redirect output to a log file. For example:

```shell
nohup vllm serve meta-llama/Llama-3.1-8B-Instruct > vllm.log 2>&1 &
```

This command launches the server with the specified model and writes all output (including any Uvicorn logs) to `vllm.log`. The server will run in the background. (The vLLM docs give a similar example: `vllm serve <model-name>`) By default, this serves the model on [`http://localhost:8000`](http://localhost:8000).

## **4\. Test the server with a curl request**

Once the server is running, you can test it by sending a chat completion request to the API. For example, use `curl` to call the chat endpoint (`/v1/chat/completions`) with a simple prompt:

```shell
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
          "model": "meta-llama/Llama-3.1-8B-Instruct",
          "messages": [
              {"role": "system", "content": "You are a helpful assistant."},
              {"role": "user", "content": "What is the capital of France?"}
          ]
        }'
```

This sends a JSON payload to the server. The `model` field names our loaded model, and `messages` contains a chat-style conversation. The server will respond with a JSON completion. (The official vLLM docs show a similar curl example for chat completions)

You should see a JSON response in the terminal with the model’s answer.

## **5\. Clone the vLLM repository**

To access the benchmark script, clone the vLLM source repository:

```shell
git clone https://github.com/vllm-project/vllm.git
cd vllm/benchmarks
```

This pulls the latest vLLM code. Now you have the `benchmark_serving.py` script and related files in the `benchmarks` folder.

## **6\. Run the benchmark script**

Finally, run the built-in benchmark script to measure serving performance. The `benchmark_serving.py` tool sends many requests to the running server and reports throughput/latency. For example, you might run:

```shell
  python3 vllm/benchmarks/benchmark_serving.py \
    --backend vllm \
    --base-url http://127.0.0.1:8000 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --tokenizer meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name random \
    --random-input-len 128 \
    --random-output-len 128 \
    --num-prompts 80 \
    --max-concurrency 4 \
    --temperature 0.7

```

-   `--backend vllm`: Use the vLLM backend to send benchmark requests.

-   `--base-url http://127.0.0.1:8000`: Address of your running vLLM server.

-   `--model meta-llama/Llama-3.1-8B-Instruct`: Name of the model being benchmarked.

-   `--tokenizer meta-llama/Llama-3.1-8B-Instruct`: Tokenizer used for encoding/decoding; usually the same as the model.

-   `--dataset-name random`: Use synthetic/randomly generated prompts instead of a real dataset.

-   `--random-input-len 128`: Length (in tokens) of each generated input prompt.

-   `--random-output-len 128`: Desired length (in tokens) of the model's output.

-   `--num-prompts 80`: Total number of prompts to send during the test.

-   `--max-concurrency 4`: Number of prompts sent in parallel (simulates batch size).

-   `--temperature 0.7`: Controls randomness in generation; higher \= more creative.

This example command (modeled on published vLLM benchmarks) will run 80 chat requests against your server with concurrency 4\. After it finishes, the script prints stats like throughput and latency.

By following these steps and citing vLLM’s documentation and examples, you can serve `Llama-3.1-8B-Instruct` with vLLM and evaluate its performance. Adjust parameters (batch size, number of tokens, etc.) in the commands above as needed for your environment.

### Conclusion

And that's it\! A fast, easy-to-use LLM benchmarking workflow powered by vLLM. In just ten minutes, you can gather meaningful performance data that will help you make smarter decisions about which models (and hardware) are right for your specific project and workload.

Need powerful GPUs to run your tests? Vast.ai offers on-demand access to H100s, RTX 5090s, and other high-performance options at a fraction of the typical price – so you can save 5-6X on GPU compute. Get started today and benchmark on your terms\!


Benchmark LLMs in minutes using vLLM on Vast.ai. Serve models like Llama 3.1-8B, test performance, and optimize cost with fast, efficient, real-world insights.

How to Benchmark An LLM with vLLM in 10 Minutes




Are you looking to run powerful open-source LLMs like **Llama 4**, **Kimi K2**, or **Qwen3** without the hassle of managing complex infrastructure? Vast.ai makes it easy to train and deploy these models with a curated set of templates built for speed, flexibility, and scale. 

Our [templates](https://docs.vast.ai/templates) cover everything from browser-based interfaces like Oobabooga and Open WebUI to optimized backends like HuggingFace TGI and vLLM. 

Below, we break down the top recommended templates on **Vast.ai** and how you can use them to get your LLM project off the ground quickly and easily. 

### 1\. [Oobabooga (LLM WebUI)](https://docs.vast.ai/oobabooga-llm-webui)

[Oobabooga](https://github.com/oobabooga/text-generation-webui) provides a user-friendly web interface for interacting with open-source LLMs. Its UI  resembles the original ChatGPT style – ideal for those who prefer a graphical interface over command-line operations. 

With support for **multiple text generation backends in one UI/API**, you can switch between different models easily without restarting and maintain fine control over settings. Models like **Falcon**, **Llama**, and **Vicuna** can be loaded and explored with just a few clicks.

Our [Oobabooga template](https://docs.vast.ai/oobabooga-llm-webui) facilitates quick deployment, making it suitable for both beginners and experienced users.

### 2\. [HuggingFace TGI with Llama 3](https://docs.vast.ai/huggingface-tgi-with-llama3)

Ideal for serving **Llama 3**, this template is optimized for **high-performance text generation tasks** using HuggingFace's [Text Generation Inference (TGI) server](https://github.com/huggingface/text-generation-inference). It supports other popular open-source LLMs from HuggingFace, including **Falcon**, **StarCoder**, **BLOOM**, and **GPT-NeoX**, and it's particularly beneficial for applications requiring low-latency responses and scalability, such as chatbots or content generation tools.

For this [template](https://docs.vast.ai/huggingface-tgi-with-llama3), you'll need your own HuggingFace access token – and you'll also need to apply for permission to use Llama 3 on HuggingFace, since the model is hosted in a gated repository.

### 3\. [Open WebUI (Ollama)](https://docs.vast.ai/ollama-webui)

[Open WebUI](https://github.com/open-webui/open-webui) is an extensible, user-friendly platform for self-hosted AI deployments. It operates entirely offline and supports a range of open-source LLM backends like **Llama 4**, **Kimi K2**, **DeepSeek R1**, and **OpenAI-compatible APIs**. 

With **built-in support for retrieval-augmented generation** (RAG), Open WebUI seamlessly integrates document interactions and web search into the chat experience. You can even incorporate image generation capabilities as well as engage with multiple LLMs simultaneously in parallel.

Our [Ollama \+ WebUI template](https://docs.vast.ai/ollama-webui) will automatically set up Open WebUI as a web-based interface and expose a port for the Ollama API, making it easy to run and interact with LLMs directly from your instance.

### 4\. [vLLM](https://docs.vast.ai/vllm-llm-inference-and-serving)

Optimized for [serving open-source LLMs](https://vast.ai/article/serving-online-inference-with-vllm-api-on-vast) with high-throughput inference, [vLLM](https://github.com/vllm-project/vllm) uses a novel architecture that drastically reduces memory overhead, making it easier to serve large models like **Llama**, **Mistral**, and **Falcon** efficiently. Notably, it supports continuous batching for faster and more efficient multi-user inference at scale.

vLLM provides an OpenAI-compatible API server for easy integration into existing workflows, and it's particularly well suited for developers building **commercial or research applications that require fast and stable model responses.** The vLLM framework seamlessly supports most open-source models on HuggingFace, including the ones named above as well as **Mixtral**, **DeepSeek**, **LLaVA**, **BLOOM**, **GPT-NeoX**, **Qwen**, and more.

Our [vLLM template](https://docs.vast.ai/vllm-llm-inference-and-serving) contains everything needed for you to get started – all you have to do is specify the model you want to serve and the corresponding vLLM configuration.

### From Setup to Serving in Minutes

With Vast.ai's market-based cloud GPU rental platform, you can avoid the usual infrastructure and budget roadblocks as you spin up high-performance instances tailored to your workloads. 

Our [library](https://cloud.vast.ai/templates) of **ready-to-use templates** – covering everything from web-based interfaces to optimized inference engines – make it easy to train and deploy LLMs at scale. We take care of the heavy lifting, so you can spend less time configuring and more time building.

Why wait? **Get started with open-source LLMs on Vast.ai today\!**

Run open-source LLMs like Llama 4, Qwen3, and Kimi K2 with ease. Vast.ai templates let you deploy fast using Oobabooga, vLLM, HuggingFace TGI, or Open WebUI.

Open-Source LLMs You Can Train and Deploy on Vast.ai Right Now


The DeepSeek R1-0528 release introduces a significant improvement to the reasoning capabilities of language models by making it easier to access "thinking" mode without requiring complex prompt engineering or pre-pending thinking tokens. This enhanced model provides transparent, step-by-step reasoning that's particularly valuable for educational applications, complex problem solving, and scenarios where transparency in AI decision-making is crucial.

In this post, we'll explore how to deploy the **DeepSeek-R1-0528-Qwen3-8B** model using **vLLM** on Vast.ai's cloud GPU platform, leveraging the new **qwen3 reasoning parser** that simplifies access to the model's internal thinking process.

---

## The Open Source AI Revolution

DeepSeek R1-0528 represents more than just a technical advancement—it's a significant step toward democratizing advanced AI capabilities. As an **open source model**, it makes these reasoning and advanced AI capabilities more accessible.

Unlike closed-source models that require vendor dependence, usage costs, and data privacy concerns, DeepSeek R1-0528 enables **complete control** over your AI infrastructure. Organizations can now deploy advanced reasoning capabilities locally, maintaining data sovereignty while avoiding variable per-token pricing.

This democratization means organizations can access cutting-edge AI without massive infrastructure investments. Advanced reasoning capabilities are no longer exclusive to large tech companies—anyone with GPU access can deploy their own reasoning-capable AI system.

---

## What Makes DeepSeek R1-0528 Special?

The DeepSeek R1-0528 release addresses a key limitation in previous DeepSeek models: the complexity of accessing the model's internal reasoning process. Previously, users needed to manually prepend "thinking" tokens to model outputs or use complex prompt engineering to see step-by-step reasoning.

---

## Deploying DeepSeek R1-0528 on Vast.ai

### Step 1: Set Up Vast.ai Environment

First, install the Vast CLI and configure your API key:

```bash
pip install --upgrade vastai

export VAST_API_KEY="your_vast_api_key"
vastai set api-key $VAST_API_KEY
```

### Step 2: Find Suitable Hardware

The DeepSeek-R1-0528-Qwen3-8B model requires specific hardware capabilities:

-   **Minimum 24GB GPU RAM** for model weights and KV cache
-   **Single GPU configuration** (sufficient for the 8B parameter model)
-   **Static IP address** for stable API endpoint hosting
-   **Direct port access** for the vLLM OpenAI-compatible server
-   **60GB+ disk space** for model storage and dependencies

Search for suitable instances:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 24 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 60 \
rentable = true"
```

### Step 3: Deploy the Model

Deploy using the vLLM OpenAI-compatible server with the qwen3 reasoning parser:

```bash
export INSTANCE_ID="your_instance_id"

vastai create instance $INSTANCE_ID \
--image vllm/vllm-openai:latest \
--env '-p 8000:8000' \
--disk 60 \
--args --model deepseek-ai/DeepSeek-R1-0528-Qwen3-8B \
--max-model-len 4096 \
--reasoning-parser qwen3
```

**Key deployment parameters:**

-   `--reasoning-parser qwen3`: Enables the simplified reasoning parser
-   `--max-model-len 4096`: Accommodates longer reasoning sequences
-   `--tensor-parallel-size 1`: Single GPU configuration
-   `--gpu-memory-utilization 0.90`: Optimized memory usage

### Step 4: Get Instance Details and Test Connection

Navigate to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console and locate your instance. Click the IP address button to view the forwarded ports.

Test the connection with a simple curl command:

```bash
export VAST_IP_ADDRESS="your_ip_address"
export VAST_PORT="your_port"

curl -X POST http://$VAST_IP_ADDRESS:$VAST_PORT/v1/completions \
-H "Content-Type: application/json" \
-d '{"model": "deepseek-ai/DeepSeek-R1-0528-Qwen3-8B", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

---

## Testing the Model's Reasoning Capabilities

### Setup Python Client

```python
import time
from openai import OpenAI

# Replace with your actual Vast.ai instance details
VAST_IP_ADDRESS = "your_ip_address"
VAST_PORT = "your_port"

client = OpenAI(
    api_key="DUMMY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)
```

### Basic Functionality Test

```python
start = time.time()
test_resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{"role": "user", "content": "Hello"}]
)
elapsed = time.time() - start

print("Response:", test_resp.choices[0].message.content)
print(f"Elapsed time: {elapsed:.4f} s")
```

### Reasoning Capability Test

This is where DeepSeek R1-0528 truly shines. The model naturally provides step-by-step reasoning without complex prompt engineering:

```python
reasoning_prompt = """
Write the History of Paris in a paragraph, show your thinking
step by step first before paragraph and then write the history
in a paragraph
"""

start = time.time()
resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{"role": "user", "content": reasoning_prompt}]
)
elapsed = time.time() - start

print("Model output:\n", resp.choices[0].message.content)
print(f"Elapsed time: {elapsed:.3f} s")
```

**Expected output structure:**

1. **Thinking process**: The model shows its step-by-step reasoning
2. **Final answer**: A well-structured paragraph about Paris's history

### Mathematical Reasoning Test

```python
math_prompt = "If 8 × 7 = ?, show your work step by step."

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    messages=[{"role": "user", "content": math_prompt}]
)

print("Mathematical reasoning:\n", resp.choices[0].message.content)
```

The model will naturally break down the problem, show intermediate steps, and arrive at the solution with clear reasoning traces.

---

## Key Benefits of DeepSeek R1-0528

### 1. **Simplified Reasoning Access**

No need to prepend thinking tokens or use complex prompt engineering. The `qwen3` reasoning parser automatically handles the extraction and presentation of reasoning steps.

### 2. **Educational Value**

Perfect for educational applications where students need to understand the problem-solving process, not just the final answer.

### 3. **Transparency and Trust**

Users can see exactly how the model arrives at its conclusions, building trust in AI-powered decision-making systems.

### 4. **Cost-Effective Deployment**

Running on Vast.ai provides significant cost savings compared to major cloud providers while maintaining high performance.

### 5. **Easy Integration**

OpenAI-compatible API ensures seamless integration with existing applications and workflows.

---

## Use Cases

### Educational Applications

-   Step-by-step problem solving for mathematics, science, and logic
-   Teaching critical thinking and reasoning skills
-   Creating interactive learning experiences

### Business Intelligence

-   Transparent decision-making in automated systems
-   Audit trails for AI-powered recommendations
-   Complex analysis with clear reasoning paths

### Research and Development

-   Understanding model behavior and capabilities
-   Testing reasoning patterns across different domains
-   Developing more interpretable AI systems

---

## Conclusion

The DeepSeek R1-0528 release represents a significant step forward in making AI reasoning more accessible and transparent. By combining this powerful model with Vast.ai's cost-effective GPU infrastructure and the simplified qwen3 reasoning parser, developers can now easily deploy transparent, reasoning-capable AI systems.

Whether you're building educational tools, business intelligence systems, or research applications, DeepSeek R1-0528 on Vast.ai provides an excellent foundation for applications that require clear, step-by-step reasoning capabilities.

The elimination of complex prompt engineering for accessing reasoning modes makes this model particularly valuable for production applications where transparency and interpretability are key requirements.

---

## Useful Resources

-   [DeepSeek R1-0528 Model on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B)
-   [vLLM Documentation](https://docs.vllm.ai/)
-   [Vast.ai GPU Marketplace](https://vast.ai)
-   [OpenAI Python SDK](https://github.com/openai/openai-python)

---

_Ready to deploy transparent, reasoning-capable AI? Get started with DeepSeek R1-0528 on Vast.ai today! 🚀_


Deploy DeepSeek R1-0528 on Vast.ai to access transparent, step-by-step AI reasoning without prompt tricks—ideal for education, research, and business use.

DeepSeek R1-0528: Enhanced Reasoning with Simplified Thinking Mode on Vast.ai


We're back with a new update from the team at **Vast.ai**\! This month, we've rolled out fresh hardware options, new video-generation tools, and continued improvements behind the scenes.

The big news is that our GPU lineup now includes the **NVIDIA DGX B200** as well as the **RTX PRO 6000 WS**\! These powerful machines open the door to even more demanding workloads, bringing you the compute power required for everything from multi-modal inference to precision-driven simulations.

We also recently reached an exciting milestone: we now have **over one thousand RTX 5090 GPUs available on Vast** – 1,180 and counting.

Here are the details about what's new this month\!

### NVIDIA Cloud GPU Updates

Whether you're training foundation models, rendering high-resolution video, or running inference, our latest hardware additions give you extra flexibility:

-   **NVIDIA DGX B200** – A universal AI supercomputer built on the Blackwell architecture, equipped with **eight GPUs** and **1.4 TB** of GPU memory. With massive bandwidth and compute capacity, it's well suited to handle virtually any enterprise AI workload.

-   **RTX PRO 6000 WS** – A high-end Blackwell-based workstation GPU featuring **96 GB** of GDDR7 memory and up to **4000 TOPS** of AI compute. It can easily handle massive datasets and multi-billion-parameter models while supporting seamless multi-app workflows.

### New Guide and Templates

We've focused on video generation this month, adding a couple of new templates and a guide to help you go from idea to clip as seamlessly as possible:

-   [**Guide to Video Generation**](https://docs.vast.ai/video-generation) – A walkthrough for setting up and using ComfyUI on Vast.ai, so you can create advanced stable diffusion pipelines (ideal for video generation workflows) with this powerful node-based interface.

-   **LTX Video (ComfyUI)** – A ready-to-use template that pairs the LTX Video model with ComfyUI, optimized for consumer-grade GPUs.

-   **Open-Sora** – A template for this open-source video generation model with a streamlined, user-friendly platform that lowers the barrier to efficient, high-quality video content creation.

And that's a wrap for June 2025\!

### Our Commitment

Whether you're looking to launch a B200 instance, experiment with video generation using our latest templates, or scale up with our expanding lineup of RTX 5090 GPUs, we're here to support your work at every stage. Our goal at **Vast.ai** is to make powerful compute accessible, flexible, and easy to use – so you can focus on building, training, and creating.

Need help? Reach out to our customer support team anytime at [support@vast.ai](mailto:support@vast.ai), or connect with the community on our Discord server for tips, troubleshooting, and more.

### Change Log

#### New Templates & Guides:

-   [Guide to Video Generation](https://docs.vast.ai/video-generation)
-   LTX Video (Comfy UI) – Video-Gen model optimized for consumer-grade GPUs.
-   Open-Sora – Democratizing efficient video production for everyone.

#### New GPUs:

-   DGX B200
-   RTX PRO 6000 WS

Plus we now have over one thousand 5090 GPUs available.


New DGX B200 and RTX PRO 6000 WS GPUs, over 1000 RTX 5090s available, and new video generation templates.

June 2025 Product Update


Ten years ago today, our CEO and founder, Jacob Cannell, published ["The Brain as a Universal Learning Machine"](https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine) on LessWrong.com.

Drawing from neuroscience, machine learning, and systems theory, he proposed that **the human brain is not a patchwork of specialized, genetically programmed modules, but a general-purpose learning system** – or, as he put it, "a biological implementation of a _Universal Learning Machine_."

In hindsight, the timing was notable. Cannell published the piece just as deep learning was moving from niche research to broader relevance in artificial intelligence (AI). In the decade since then, rapid progress in scalable, trainable, data-driven models has validated the idea that learning, not pre-engineered structure, is the key to building intelligent systems.

So on the tenth anniversary of its publication, we'd like to revisit the key ideas of the article, reflect on what's aged well, and look ahead at what's still unfolding.

We’d encourage you to read the full article on [Less Wrong](https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine).


A look at the brain as a universal learning machine, and how it can be used to learn about the world.

A Decade Later: Revisiting the Brain as a Universal Learning Machine


Serving multiple machine learning models simultaneously is a key challenge in production AI systems. The increasing size of foundational models and constraints on GPU memory often mean that deploying separate models for each task results in high RAM usage, underutilized hardware, and latency problems caused by repeatedly swapping entire models in and out of GPU memory. This not only increases cost but also complicates scaling.

However, by leveraging Lorax, a framework for dynamically loading LoRA adapters—and Vast.ai's flexible GPU marketplace, developers can efficiently serve several specialized models on a single base model deployment. This results in significantly better hardware utilization, reduced infrastructure costs, and low-latency inference for diverse AI workloads, making this a game-changer for enterprises deploying their own AI infra.

In this blog post, we'll explore how to deploy and run multiple LoRA adapters on a shared base model using Lorax on Vast.ai's cloud platform. This setup enables hosting thousands of fine-tuned models simultaneously, loading task-specific adapters on demand, and seamlessly switching between tasks such as math problem solving and customer support classification with minimal overhead.

---

## What Are LoRA Adapters and Lorax?

- **LoRAs (Low-Rank Adaptation):** Lightweight, task-specific parameter adjustments that adapt a large base model to new tasks with far fewer parameters than full fine-tuning. LoRAs can be swapped in and out without needing to reload the entire large model, saving significant RAM and time.

- **Lorax (LoRA eXchange):** An efficient serving framework that enables running thousands of LoRA adapters on a single GPU by dynamically loading these adapters into a base model at inference time. Lorax maintains high throughput and low latency while drastically reducing the deployment cost of hosting multiple fine-tuned variants.

---

## Deploying Multiple Models Using Lorax and Vast.ai

The following example uses the `mistralai/Mistral-7B-v0.1` base model hosted on Vast.ai, along with two LoRA adapters:

- `predibase/gsm8k`: Designed for solving math problems like the GSM8K benchmark.
- `predibase/customer_support`: Specializes in customer service query classification.

We deploy the base model once and dynamically switch between these specialized LoRA adapters, enabling efficient multi-model serving on the same underlying GPU instance.

---

### Step 1: Set Up Vast.ai Environment

1. **Install Vast CLI and configure API key:**

```bash
pip install vastai==0.2.6

export VAST_API_KEY="your_vast_api_key"
vastai set api-key $VAST_API_KEY
```

2. **Search for a suitable GPU instance:**

The `mistralai/Mistral-7B-v0.1` model requires at least 16GB VRAM, but it's safer to select a larger instance (e.g., 32GB VRAM) for LoRA adapter loading.

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 32 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

3. **Deploy the instance:**

Make sure you have accepted the usage terms for `Mistral-7B-v0.1` on Huggingface before proceeding.

```bash
export INSTANCE_ID=your_instance_id

vastai create instance $INSTANCE_ID --image ghcr.io/predibase/lorax:main \
--env '-p 8080:80 --shm-size 1g -e HUGGING_FACE_HUB_TOKEN="your_hf_token"' \
--disk 80 --args --model-id mistralai/Mistral-7B-v0.1
```

This command runs the Lorax server that hosts the base model and dynamically loads adapters on demand.

4. **Get Instance IP Address and Port**

Now, we need to get our IP address and port to call our model. We first need to wait for our machine to download the image and the model and start serving. This will take a few minutes. The logs will show you when it's done.

Next, go to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console and find the instance you just created.

At the top of the instance, there is a button with an IP address in it. Click this and a panel will show up of the ip address and the forwarded ports.
You should see something like:

```
Open Ports
XX.XX.XXX.XX:YYYY -> 8000/tcp
```

You will need the IP address (XX.XX.XXX.XX) and the port (YYYY) for the next step.

---

### Step 2: Query LoRA Adapters via HTTP Requests

Lorax supports on-demand loading of LoRAs from the Huggingface Hub, minimizing memory overhead versus deploying multiple large models simultaneously.

First, configure environment variables to connect to your Vast instance and set your Huggingface token:

```python
HF_TOKEN = "your_hf_token"
VAST_IP_ADDRESS = "your_instance_ip"
VAST_PORT = "your_vast_port"
```

#### Example: Math Problem Solving with `predibase/gsm8k`

Prepare a prompt:

```python
question = "James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week?"
math_prompt = f"Please answer the following question: {question}\nAnswer"
```

Make a POST request to the `/generate` endpoint specifying the LoRA adapter ID:

```python
import requests

headers = {
    "Content-Type": "application/json",
    "Authorization": f"Bearer {HF_TOKEN}"
}
data = {
    "inputs": math_prompt,
    "parameters": {
        "max_new_tokens": 100,
        "adapter_id": "predibase/gsm8k",
        "adapter_source": "hub"
    }
}

url = f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/generate"
response = requests.post(url, headers=headers, json=data)
print("Response:", response.json()["generated_text"])
```

**Response:**

```
He runs 3*3=<<3*3=9>>9 sprints a week
So he runs 9*60=<<9*60=540>>540 meters a week
#### 540
```

#### Example: Customer Support Classification with `predibase/customer_support`

Prepare a customer support transcript prompt:

```python
transcript = "Hi I am having trouble with my account. It says I need to reset my password to log in but I already reset my password."

customer_support_prompt = f"""Consider the case of a customer contacting the support center.
The term task type refers to the reason for why the customer contacted support.

### The possible task types are: account issue, billing issue, product issue, none of the above

Summarize the issue/question/reason that drove the customer to contact support:

Transcript: {transcript}

Task Type:
"""
```

Make the API call:

```python
data = {
    "inputs": customer_support_prompt,
    "parameters": {
        "max_new_tokens": 100,
        "adapter_id": "predibase/customer_support",
        "adapter_source": "hub"
    }
}

response = requests.post(url, headers=headers, json=data)
print("Response:", response.json()["generated_text"])
```

**Response:**

```
account issue
```

---

### Step 3: Access Models Using the OpenAI SDK

Lorax provides compatibility with the OpenAI API, allowing easy integration into existing apps with minimal code changes.

1. **Install OpenAI SDK**

```bash
pip install --upgrade openai
```

2. **Call the math adapter:**

```python
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1",
    default_headers={"Authorization": f"Bearer {HF_TOKEN}"}
)

resp = client.completions.create(
    model="predibase/gsm8k",
    prompt=math_prompt,
    max_tokens=100,
    temperature=0.7
)

print("Response:", resp.choices[0].text)
```

Output:

```
James runs 3 x 3 = <<3*3=9>>9 sprints in a week.
He runs 9 x 60 = <<9*60=540>>540 meters in a week.
#### 540
```

3. **Call the customer support adapter:**

```python
resp = client.completions.create(
    model="predibase/customer_support",
    prompt=customer_support_prompt,
    max_tokens=100,
    temperature=0.7
)
print("Response:", resp.choices[0].text)
```

Output:

```
account issue
```

---

## Why This Approach Matters

Serving multiple specialized AI models traditionally involves deploying each fine-tuned model independently, consuming lots of GPU memory and resulting in higher cost and slower inference when switching tasks.

By contrast, Lorax’s approach uses a single shared base model and dynamically loads lightweight LoRA adapters tailored for different tasks. This yields several benefits:

- **Reduced RAM and GPU usage:** Only the small adapter layers are loaded on demand, not the entire large model.
- **Lower infrastructure cost:** A single GPU instance can serve thousands of specialized models.
- **Faster context switching:** Switching between tasks is near-instant as you don’t reload the full model.
- **Flexibility:** New adapters can be added or updated independently.
- **Ease of integration:** OpenAI-compatible APIs simplify adoption.

Combined with Vast.ai’s cost-effective, flexible GPU rental options, this method provides a scalable, production-ready solution for deploying multi-model AI services with a minimal footprint.

---

## Conclusion

Efficiently serving multiple machine learning models is vital for scaling AI-powered products. The combination of Lorax’s dynamic LoRA adapter serving and Vast.ai’s flexible GPU infrastructure offers a powerful, cost-effective, and scalable way to support many specialized models on a single deployment.

Whether solving math problems, classifying customer inquiries, or other tasks, this approach reduces overhead and unlocks practical multi-model serving for modern AI applications.

Try setting up your own multi-LoRA serving environment today to experience the benefits firsthand!

---

## Useful Resources

- [Lorax GitHub Repository](https://github.com/predibase/lorax)
- [Vast.ai GPU Marketplace](https://vast.ai)
- [Mistral-7B-v0.1 on Huggingface](https://huggingface.co/mistralai/Mistral-7B-v0.1)
- [OpenAI Python SDK](https://github.com/openai/openai-python)

---

_Happy modeling! 🚀_


Serve multiple ML models efficiently using Lorax and LoRA adapters on Vast.ai—cut costs, reduce latency, and maximize GPU use with dynamic adapter loading.

Efficiently Serving Multiple Machine Learning Models with Lorax and vLLM on Vast.ai


Nearly every startup faces the same two pressures: to move fast and make every dollar count.

Time and budget may be limited while demands remain high – especially for teams working with AI, machine learning, or other compute-heavy workloads. For startups, the pressure to deliver quickly and prove technical viability makes access to reliable infrastructure even more critical.

Traditional cloud providers often aren't built with startups in mind. Pricing may be steep or overly complex, resources can be constrained, and it's easy to end up locked into systems that don't scale with your needs. On the other hand, buying your own hardware adds upfront costs and delays, which can be difficult to justify for early-stage teams.

Vast.ai offers another path. **Our market-based cloud GPU rental platform gives startups the flexibility to run powerful workloads without the usual friction.**

Here's a closer look at why startups choose to build with Vast.ai.

### Why Vast.ai Makes Sense for Startups

Startups need infrastructure that keeps up with their pace, budget, and technical demands. The following are some of the reasons Vast.ai hits the mark.

#### 1\. Instant Access to GPUs

With Vast.ai, there are no waitlists or delays. Startups can instantly access powerful GPUs like **H100s**, **A100s**, and **RTX 5090s**, backed by a global network of providers that ensures better availability and built-in redundancy.

Plus, with machines available around the world, startups can deploy GPUs in regions closer to their major user concentrations, reducing network latency for performance-sensitive applications.

#### 2\. Cost-Effective Compute Power

Vast.ai offers significantly cheaper GPU rental compared to traditional cloud providers – **on average 5-6X lower in cost** – making high-performance compute far more accessible.

For bootstrapped startups and solo founders, Vast.ai levels the playing field, enabling smaller teams on a budget to compete in compute-intensive spaces, as well as handle temporary spikes in workload without having to invest in expensive infrastructure.

#### 3\. Flexible Startup-Friendly Billing

Vast.ai's **pay-as-you-go pricing** gives startups the freedom to scale compute resources up or down as needed, with no lock-in contracts. This way, **you only pay for what you use**, and you can quickly adapt to new demands as they arise.

On-demand and interruptible (bid auction) [rental options](https://docs.vast.ai/instances/rental-types) offer even more control – ideal for variable usage patterns and budget-conscious teams. On top of that, real-time cost tracking and detailed resource utilization dashboards help you monitor and optimize performance.

#### 4\. Enterprise-Grade Security

With Vast.ai, **startups can choose the level of trust and security that best fits their needs.** Providers [range](https://vast.ai/compliance) from verified hosts, suitable for general computing needs, to our vetted datacenter partners who maintain third-party compliance certifications (at minimum ISO 27001 certified) and sign comprehensive Data Processing Agreements (DPAs) with Vast.ai.

For sensitive or production workloads, our Secure Cloud tier ensures you’re connecting with partners in certified locations that are current with ISO 27001 and/or Tier 3/4 standards. (You can even filter for it. Look for the blue datacenter label\!)

We've also [completed](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) our SOC 2 Type I certification – part of our ongoing commitment to the highest standards of regulatory compliance.

#### 5\. Developer Control

Vast.ai provides **transparent instance specs** (GPU type, RAM, storage, and more), so startups can choose exactly what fits their workload and budget. Developers can bring their own Docker images, run their own stack, and configure environments to match their specific requirements.

With a low barrier to entry, even early-stage teams without dedicated DevOps support can get up and running quickly. Our selection of pre-configured [templates](https://docs.vast.ai/templates) are like recipes that help users easily set up instances; templates can define the Docker image, environment variables, and launch mode for each new instance.

#### 6\. Ideal for AI Workloads

Vast.ai is well suited for AI startups working on large-scale training, fine-tuning, and inference. Our customizable, on-demand [CPU/GPU clusters](https://vast.ai/clusters) ensure reliable throughput and flexible resource allocations for continuous or large-scale workloads.

With support for large language models (LLMs) and diffusion models, our platform is compatible with all major machine learning (ML) frameworks and offers the flexibility to run custom containers and tooling.

For teams that want to get started quickly, Vast.ai provides **ready-to-use GPU instances pre-configured for frameworks like TensorFlow and PyTorch**, reducing setup time and simplifying deployment.

#### 7\. Built to Experiment

With Vast.ai, **startups can prototype, test, and explore without any long-term commitments** or expensive upfront investment.

For instance, teams can run quick tests and train models on demand, access and evaluate cutting-edge GPUs like the H200 that may otherwise be out of reach, and even spin up high-performance environments on short notice for hackathons and demos.

### The Bottom Line

**Great ideas can come from anywhere, and they deserve the resources to grow.** By making GPU compute power more open and accessible, Vast.ai gives startups and early-stage teams a real shot at achieving their goals.

Built with transparency, accessibility, and security in mind – and aligned with the open-source ethos that many founders value – our platform supports the kind of flexibility and control that your team needs from day one.

Ready to build, test, and scale on your terms? Discover what's possible with Vast.ai\!


Vast.ai offers startups instant, low-cost access to powerful GPUs—perfect for AI, ML, and compute-heavy work—without lock-ins or upfront infrastructure costs.

7 Reasons Why Startups Use Vast.ai


The GeForce RTX 4090 and the A100 both sit at the high end of NVIDIA's GPU lineup, but they're built for very different worlds.

The **RTX 4090** is a powerful consumer-grade GPU, designed for ultra high-performance gaming, professional creative work, and even some entry-level AI workloads. It's surprisingly capable across a variety of tasks, especially for its price point.

The **A100**, on the other hand, isn't aimed at consumers at all. It's an enterprise-grade GPU built for data centers, research labs, and AI teams running large-scale model training, inference, and simulation workloads – designed to move serious data at serious speed.

In this post, we'll look at how these two machines compare with each other, and why you might choose one over the other based on your specific needs.

### NVIDIA GeForce RTX 4090: A Workhorse with Range

When it launched in September 2022 as the top-tier consumer GPU of its generation, the RTX 4090 quickly became the go-to choice for 4K gaming, 3D rendering, and AI-enhanced creative workflows.

Built on NVIDIA's **Ada Lovelace** architecture and powered by the **AD102 chip**, the RTX 4090 introduced significant upgrades in ray tracing and Tensor Core acceleration, and delivered notable performance gains with **DLSS 3** in a variety of games.

While it's no longer the flagship GPU of NVIDIA's GeForce line (that title now goes to the RTX 5090, which you can read about [here](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know)), the RTX 4090 definitely still holds its own. It remains one of the most capable and well-rounded GPUs available at its price point for consumers and technical teams alike.

#### Some of the RTX 4090's key specs and features include:

-   **24 GB of GDDR6X memory** with **1,008 GB/s bandwidth** on a **384-bit** interface.  

-   **16,384 CUDA cores** for high-parallel throughput.  

-   **128 third-gen RT cores** and **512 fourth-gen Tensor cores** for advanced ray tracing and AI workloads.  

-   **DLSS 3** support for real-time upscaling and performance boosts in many games.  

-   **450 watts** of thermal design power (TDP), requiring an **850-watt power supply** due to its high processing capabilities.

If you're looking for serious performance in a desktop form factor, the RTX 4090 continues to deliver. It may not be the newest card on the market, but it remains a workhorse that gets the job done across a wide range of applications.

As capable as the RTX 4090 is, it's not built for everything. If your workloads go beyond what a high-end consumer GPU can handle, the A100 just might bring the compute power you need.

### NVIDIA A100: A Powerhouse with Purpose

The A100 is built for scale. Based on NVIDIA's Ampere architecture and powered by the **GA100 chip**, it's designed to tackle massive workloads – like foundational AI model training – thanks to its raw throughput, high-bandwidth memory, and ability to scale across multi-GPU environments.

When the A100 launched in June 2021, it introduced **third-gen Tensor Cores**, support for **TF32** precision, and **Multi-Instance GPU** (MIG) capabilities – enabling flexible resource partitioning and efficient parallel processing across shared environments.

While some of its core specs might look modest compared to the RTX 4090, the A100 delivers superior performance in AI and data-intensive workloads because it's optimized where it counts: memory bandwidth, interconnects, and specialized compute.

For instance, the A100's memory clock is much lower than the RTX 4090's on paper (roughly 3 Gbps vs. 21 Gbps). However, the A100 uses **HBM2e memory** with a much wider **5,120-bit interface**. This design allows it to deliver around **2 TB/s** of bandwidth – double the RTX 4090 – despite the lower frequency. It's an approach that prioritizes efficiency and scale.

#### To recap, here are a few notable features of the A100:

-   **80 GB of HBM2e memory** with about **2 TB/s of bandwidth** (twice that of the RTX 4090).  

-   **5,120-bit memory bus**, an exceptionally wide interface enabling fast access to memory, reducing bottlenecks in AI and high-performance computing (HPC) workloads.  

-   **432 third-gen Tensor Cores**, with high-throughput **TF32** performance for AI and robust **FP64** support for HPC workloads.  

-   **Multi-Instance GPU (MIG)** support, so it can be partitioned into up to 7 isolated GPU instances – ideal for multi-tenant workloads and fine-grained resource allocation.  

-   **NVLink and NVSwitch** allow up to 16 A100 GPUs to be interconnected at up to **600 GB/s** of bi-directional bandwidth per GPU.

In short, the A100 delivers serious performance for large-scale AI and HPC workloads. But if you're comparing it head-to-head with the RTX 4090 on specs alone, here's how the two stack up.

### RTX 4090 vs. A100: Key Specs Compared

For reference, this table highlights some of the features and specs of the RTX 4090 and A100:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Specification
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 4090
            </th>
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                A100
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Ada Lovelace
            </td>
            <td class="border border-gray-300 px-4 py-2">Ampere</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">VRAM</td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                24 GB GDDR6X
            </td>
            <td class="border border-gray-300 px-4 py-2">80 GB HBM2e</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1313 MHz 21 Gbps
            </td>
            <td class="border border-gray-300 px-4 py-2">1512 MHz 3 Gbps</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Bus Width
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">384</td>
            <td class="border border-gray-300 px-4 py-2">5120</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Bandwidth
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1.01 TB/s
            </td>
            <td class="border border-gray-300 px-4 py-2">1.94 TB/s</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Streaming Multiprocessors (SMs)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">128</td>
            <td class="border border-gray-300 px-4 py-2">108</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">512</td>
            <td class="border border-gray-300 px-4 py-2">432</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                CUDA Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">16,384</td>
            <td class="border border-gray-300 px-4 py-2">6,912</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Ray Tracing Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">128</td>
            <td class="border border-gray-300 px-4 py-2">N/A</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Base Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2235 MHz
            </td>
            <td class="border border-gray-300 px-4 py-2">1065 MHz</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Boost Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2520 MHz
            </td>
            <td class="border border-gray-300 px-4 py-2">1410 MHz</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Multi-Instance GPUs
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                No MIG Support
            </td>
            <td class="border border-gray-300 px-4 py-2">
                Up to 7 MIGs @ 10GB each
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Thermal Design Power (TDP)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">450W</td>
            <td class="border border-gray-300 px-4 py-2">300W</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Recommended Power Supply
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">850W</td>
            <td class="border border-gray-300 px-4 py-2">700W</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Launch Date
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Sept. 20, 2022
            </td>
            <td class="border border-gray-300 px-4 py-2">June 28, 2021</td>
        </tr>
    </tbody>
</table>

As mentioned, specs are just one part of the equation. Ultimately, choosing the right GPU depends on your workload, environment, and priorities.

### Use Cases: Which GPU Should You Choose?

Here’s how the strengths of each GPU align with different types of use cases.

#### You should consider the RTX 4090 if you prefer:

-   **A powerful, desktop-friendly GPU** for high-end gaming, creative workflows, and light AI experimentation.
-   **Excellent performance-per-dollar in single-GPU setups**, particularly for developers and small teams.
-   **A widely available, less costly option** for rendering, video editing, or running small to medium AI models locally.

#### Whereas the A100 may be a better fit if you need:

-   **Scalable performance** for training large models, running high-throughput inference, or powering complex simulations.
-   **A data center-optimized GPU** with advanced memory architecture and precision modes tailored for AI and HPC workloads.
-   **Flexible deployment options**, including support for multi-instance workloads and high-speed interconnects in clustered environments.

\------------------------------

At **Vast.ai**, we understand there are other considerations, as well. Access, flexibility, and affordability are just as important, especially when you're experimenting, scaling, or moving fast.

That's why our market-based cloud GPU rental platform gives you affordable, on-demand access to RTX 4090s, A100s, and much more. With Vast.ai, you can instantly match the right hardware to your workload – without the upfront investment and overhead of managing your own infrastructure.

Try Vast today and discover what you can achieve with compute power that fits your needs and your budget\!


Compare NVIDIA’s RTX 4090 vs A100 GPUs to find out which is better for gaming, AI, HPC, or creative work—and how to access both on Vast.ai.

NVIDIA RTX 4090 vs. A100: Two Powerhouses, Two Purposes


Welcome back to another update from the team at **Vast.ai**\! Over the past two months, we've been hard at work refining our platform and rolling out features that make your experience on Vast faster, smoother, and more powerful. These updates are part of our ongoing effort to support your projects with scalable, flexible infrastructure you can rely on.

We're also pleased to have achieved [**SOC 2 Type I certification**](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) recently, a clear validation of the controls we've put in place to protect customer data and ensure the integrity of our infrastructure. The independent SOC 2 audit confirms that our security practices meet the highest standards of regulatory compliance – giving you added confidence as you build on our platform.

Another exciting update: [**H200 GPUs**](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights) are now available on Vast\! If you're pushing the limits of high-performance compute, these machines are ready to meet the challenge.

Let's take a look at what else is new\!

### NVIDIA Cloud GPU Updates

Whether it's expanded hardware options, new templates, or updated guides, our latest improvements give you a more powerful, customizable environment for every stage of your workflow.

#### Highlights of the latest enhancements:

* **New Templates Added**

  * [***Dia 1.6b TTS***](https://cloud.vast.ai/?ref_id=62897&template_id=6ba071acb5f590913c5eaf6c2dc3d0db) – Convert text to lifelike speech using the open-source text-to-speech model from Nari Labs (full guide [here](https://docs.vast.ai/tts-with-nari-labs-dia))..  
  * [***Tari Universe***](https://cloud.vast.ai/?ref_id=62897&template_id=338f9628e43e586a87d37d18c719291d) – Run crypto mining workloads on Vast via desktop application (note: cryptocurrency funding is required).  
  * [***Pinokio Desktop***](https://cloud.vast.ai/?ref_id=62897&template_id=06c4818322f002904b324165b9dba52f) – Deploy AI applications with verified scripts from trusted publishers, without manual setup.

* **Introducing Local Volumes (CLI-only)**

  * Local Volumes are **persistent Docker volumes** that keep your data intact across restarts, new instances, and failures – so your storage survives instance deletion.  
  * **Benefits:** Share one volume across multiple containers on the same machine, and spin up fresh instances without data-transfer overhead. 

* **Upgraded Instance Portal**

  * For Vast.ai instances using [recommended templates](https://cloud.vast.ai/templates/), instances now launch via **secure Cloudflare tunnels** by default.  
  * Create a user-friendly **custom tunnel URL** that points to a port on your instance.  
  * **Live logs** and **live stats** (CPU, Memory, and Disk) for your instance are now viewable.

* **Vast Teams Improvements**

  * **Smoother invites** (fast and easy team creation), **standalone teams** (no need to convert your individual account), and **context switching** (seamlessly switch between Individual and Team views).  
  * **Access Keys:** Share API Keys and view Team-wide Session Keys as owner.  
  * Various stability and usability improvements.

For more information and helpful links, check out the full change log for our April–May 2025 updates at the bottom of this page.

### Our Commitment

Whether you're trying out the H200, deploying AI applications with just a few clicks, launching a new template, or exploring our latest platform features, we're here to make your experience on **Vast.ai** as smooth and flexible as possible. However you use Vast, our goal remains the same: to deliver reliable, cost-effective compute whenever you need it.

Got questions? Help is always within reach\! Contact our customer support team at [support@vast.ai](mailto:support@vast.ai) anytime, or join our Discord server for additional resources and community support.

### Change Log

#### New Templates

* **Dia 1.6b TTS** – Transform text into ultra-realistic speech using Dia, the open-source text-to-speech model from Nari Labs.  
  * [Full Guide](https://docs.vast.ai/tts-with-nari-labs-dia)  
  * [Launch an Instance with the Dia Template](https://cloud.vast.ai/?ref_id=62897&template_id=6ba071acb5f590913c5eaf6c2dc3d0db)  
* **Tari Universe** – Desktop Crypto mining application you can run on Vast. IMPORTANT: You must fund your Vast account with cryptocurrency if you want to use the service for mining.  
  * [Launch an Instance with the Tari Universe Template](https://cloud.vast.ai/?ref_id=62897&template_id=338f9628e43e586a87d37d18c719291d)  
* **Pinokio Desktop** – Access an extensive library of verified scripts from trusted publishers and deploy AI applications effortlessly without manual setup.  
  * [Launch an Instance on Vast with the Pinokio Desktop Template](https://cloud.vast.ai/?ref_id=62897&template_id=06c4818322f002904b324165b9dba52f)

####  Introducing Local Volumes (CLI only – UI coming soon\!)

**What are Local Volumes?**  
Persistent Docker volumes to keep your data intact across restarts, new instances, and failures.

**Key client benefits:**  
• Persistent storage survives instance deletion.  
• Share one volume across multiple containers on the same machine.  
• Spin up fresh instances on the same machine without data-transfer overhead.

**Get started:**  
• Search for available Volumes via CLI:  vastai search volumes  
• General docs: [https://docs.vast.ai/volumes](https://docs.vast.ai/volumes) 

*Note: Currently, clients can only rent Local Volumes through CLI. UI support and Network Volumes are coming soon.*

#### Teams Improvements

We’ve improved the Vast Teams experience for user access management and collaboration:

* **Smoother Invites:** Get team members connected on Vast quickly.  
* **Standalone Teams:** Create Teams without converting your individual account.  
* **Context Switching:** Seamlessly switch between individual and Team views.  
* **Access Keys:** Share API Keys and view Team-wide Session Keys as owner.  
* **Team Basics:** Transfer credits to/from your team, transfer ownership, and other basics.  
* **Bug Fixes:** Various stability and usability improvements.

[Create a team](https://cloud.vast.ai/manage-members/) and let us know your feedback\!

#### Instance Portal upgrade

We've also rolled out a major upgrade to the **Instance Portal** that appears on Vast.ai instances using [recommended templates](https://cloud.vast.ai/templates/).

**What’s New:**

* **Secure Tunnel URLs:** Instances now launch via secure Cloudflare tunnels by default, eliminating the “not secure” screen.  
* **Custom Tunnel URLs:** Create a user-friendly URL that points to a port on your Instance. Optionally use a named tunnel to use a custom domain.  
* **Live Logs:** See a live stream of log updates on your Instance.  
* **Live Stats:** See live CPU, Memory, and Disk stats for your Instance.

Deploy a recommended template instance or check out the [updated Instance Portal guide](https://docs.vast.ai/instance-portal) to learn more\!

#### Opt in to the new Instances UI

We've rolled out an (optional) UI update as part of our ongoing UI refresh:

* Cleaner instance card design  
* Instance data fetching improvements  
* Various instance UI bug fixes

Opt-in: Go to [Settings \> Early Access](https://cloud.vast.ai/account/) and toggle it on. You can switch back anytime.

#### SOC2 Type I Certification complete.

#### H200s available now\!

Vast.ai adds H200 GPUs, SOC 2 Type I cert, new templates, persistent storage, and better teams & instance tools for faster, secure cloud compute.

April-May 2025 Product Updates


## Introduction

The recent advancements in AI for video generation have been very fast paced. Models now create much more realistic videos than before, with multiple different use cases. One such model is [HunyuanVideo](https://huggingface.co/tencent/HunyuanVideo), a model known for its impressive outputs—and tackle the practical challenges of running it efficiently on high-memory GPUs like the A100 or H100.

HunyuanVideo is Tencent's state-of-the-art text-to-video generation model that rivals or surpasses leading closed-source alternatives. As the largest open-source video generation model with over 13 billion parameters, HunyuanVideo represents a significant breakthrough in AI-powered video creation.

We also have a [notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/LznBGR/aec37016e507d00895435abda7f0a3bc6e29625a/files/hunyuan_video_vast.ipynb) to follow along once you deploy the Vast Instance.

### Key Innovations

- **Unified Architecture**: Employs a "Dual-stream to Single-stream" transformer design that effectively handles both image and video generation.
- **MLLM Text Encoder**: Utilizes a multimodal large language model for superior text-to-visual alignment compared to previous text-encoder architectures.
- **Efficient Compression**: Implements causal 3D VAE for efficient spatial-temporal compression to enable training at higher resolution.
- **Automatic Prompt Rewriting**: Features intelligent prompt enhancement to optimize user-given prompts tailored for the model's consumption.

### What You’ll Learn

In this guide, we will:

1. Set up a custom Docker template for HunyuanVideo on Vast.ai.
2. Launch the Docker template on a suitable GPU instance.
3. Download pretrained models and generate high-quality videos from text prompts.

Let’s get started with HunyuanVideo!

---

## Renting an Instance on Vast

### Create a Custom Template

Tencent maintains a custom Docker image for HunyuanVideo: `hunyuanvideo/hunyuanvideo:cuda_12`. To run the model on Vast, we'll create a custom template using this image.

Follow these steps:

1. Ensure you have a [Vast.ai](https://vast.ai/) account.
2. Navigate to the **Templates** page on the [Vast Console](https://cloud.vast.ai/templates/).
3. Find the existing template named **PyTorch (CuDNN Runtime)** and click **Edit**.
4. Rename the template to **HunyuanVideo**.
5. Replace the Docker image in the **Image Path:Tag** field with `hunyuanvideo/hunyuanvideo:cuda_12`.
6. Set the **On-start Script** to:
   ```
   git clone https://github.com/tencent/HunyuanVideo
   ```
   This ensures the HunyuanVideo repository is downloaded on instance startup.
7. Allocate sufficient disk space, for example, **80 GB**.
8. Click **Save & Use** to save your template and prepare to deploy it.

---

### Selecting an Instance

HunyuanVideo requires a GPU with **at least 80GB VRAM** to run smoothly. Available GPUs meeting this requirement include Nvidia’s A100 and H100 cards.

To select an instance:

1. Filter Vast instances based on:
   - **Instance Type**: A100 or H100
   - **Number of GPUs**: 1
   - **VRAM**: ≥ 80GB
2. Choose and rent a suitable instance.
3. Install the Vast TLS certificate in your browser, enabling secure access to your Jupyter notebook server: [Installing TLS Certificate Guide](https://docs.vast.ai/instances/jupyter#1SmCz).
4. Open the Jupyter server via https://cloud.vast.ai/instances/, clicking **Open** for your rented instance.
5. Upload this notebook into the directory `/workspace/HunyuanVideo/` on the server.

---

## Downloading the Model Weights

Before generating videos, we need to download pretrained model weights from Hugging Face. These include the video model weights and the text encoders.

Run the following commands inside your instance:

```bash
huggingface-cli download tencent/HunyuanVideo --local-dir ./ckpts
huggingface-cli download xtuner/llava-llama-3-8b-v1_1-transformers --local-dir ./ckpts/llava-llama-3-8b-v1_1-transformers
python hyvideo/utils/preprocess_text_encoder_tokenizer_utils.py --input_dir ckpts/llava-llama-3-8b-v1_1-transformers --output_dir ckpts/text_encoder
huggingface-cli download openai/clip-vit-large-patch14 --local-dir ./ckpts/text_encoder_2
```

For a more in-depth discussion of the checkpoints, refer to [Tencent’s checkpoint README](https://github.com/Tencent/HunyuanVideo/blob/main/ckpts/README.md).

---

## Generating Videos using HunyuanVideo

### Displaying Videos in Jupyter

To conveniently view generated videos within the Jupyter notebook, let’s define a helper function using `IPython.display.Video`:

```python
from IPython.display import Video

def show_video(video_path, width=640, height=360, embed=True):
    """
    Display a video in a Jupyter notebook.

    Parameters:
    -----------
    video_path : str
        Path to the video file (local file or URL)
    width : int, optional
        Width of the video player in pixels
    height : int, optional
        Height of the video player in pixels
    embed : bool, optional
        Whether to embed the video in the notebook (True)
        or just link to it (False)

    Returns:
    --------
    IPython.display.Video
        Video display object
    """
    return Video(video_path, width=width, height=height, embed=embed)
```

---

### Generating Your First Video: Cat on the Grass

Let’s generate a realistic video of a cat walking on the grass using the example prompt provided by Tencent.

Run the following command in your terminal:

```bash
python sample_video.py \
    --video-size 720 1280 \
    --video-length 129 \
    --infer-steps 50 \
    --prompt "A cat walks on the grass, realistic style." \
    --flow-reverse \
    --use-cpu-offload \
    --save-path ./results
```

This command will create a video approximately 129 frames long (roughly 4-5 seconds at standard frame rates) at 720x1280 resolution.

Once the generation finishes, locate the video file inside the `./results` directory.

To display it in your notebook, run:

```python
show_video("./results/[your_video_filename].mp4")
```

Replace `[your_video_filename].mp4` with the actual file name.

You should see a high-quality, realistically animated cat walking on grass, showcasing HunyuanVideo’s remarkable detail in animal motion and textures.

---

### Creating a Video of an Astronaut Walking on the Moon

Next, try generating a completely different scene:

```bash
python sample_video.py \
    --video-size 720 1280 \
    --video-length 129 \
    --infer-steps 50 \
    --prompt "An astronaut walks across the moon, realistic style." \
    --flow-reverse \
    --use-cpu-offload \
    --save-path ./results
```

After the video is generated, display it in your notebook the same way:

```python
show_video("./results/[your_astronaut_video].mp4")
```

The resulting video will demonstrate the model’s versatility at rendering vastly different environments and characters, from furry animals to astronauts in space suits.

---

## Conclusion and Next Steps

You've now successfully generated your first videos using HunyuanVideo on a Vast-powered cloud instance. This powerful model unlocks creative possibilities in AI-driven video generation with:

- **Flexible Resolution Options**: Try different `--video-size` values (e.g., 1280×720, 960×960) to optimize for your needs.
- **Adjustable Quality Settings**: Modify `--infer-steps` (default 50) to trade off video generation quality and speed.
- **Creative Control**: Experiment with `--embedded-cfg-scale` (default 6.0) to balance prompt fidelity and creative variance.
- **Deterministic Outputs**: Set `--seed` to reproduce favorite generated videos reliably.

Deploying large models like HunyuanVideo can be resource-intensive but is now accessible with cloud platforms like Vast, giving you access to top-tier GPUs without upfront hardware costs.

With these foundations, you’re ready to explore more sophisticated prompt engineering, fine-tune generation parameters, or even integrate HunyuanVideo into multimedia projects!

---

**Resources:**

- [HunyuanVideo GitHub Repository](https://github.com/tencent/HunyuanVideo)
- [Vast.ai Documentation](https://docs.vast.ai/)
- [HunyuanVideo Checkpoints README](https://github.com/Tencent/HunyuanVideo/blob/main/ckpts/README.md)

Learn how to run Tencent’s HunyuanVideo text-to-video model on Vast.ai using A100/H100 GPUs to generate realistic AI-powered videos from prompts.

Text to Video Generation Using HunyuanVideo on Vast


[RolmOCR](https://huggingface.co/reducto/RolmOCR) from Reducto is a powerful, open-source document OCR solution that delivers superior performance while requiring fewer resources than comparable models. Compared to its predecessor, [allenai/olmOCR-7B-0225-preview](https://huggingface.co/allenai/olmOCR-7B-0225-preview), this new model is built on a better base model, Qwen2.5-VL-7B instead of Qwen2.0-VL-7b. This model isn't trained to take in metadata information and with some 15% rotation of the documents, which enables it to be more flexible under different conditions. Overall, it excels at OCR on PDF's, documents, forms, and invoices. Because it is open source, companies can build on top of this model for their own proprietary pipelines while not sending data to other providers or model hosting services.

Vast.ai offers a GPU marketplace where you can rent compute power at lower costs than major cloud providers, with the flexibility to select specific hardware configurations optimized for specific models and GPU RAM needs, while keeping all of your company's data private.

In this guide, we'll demonstrate how to extract structured pricing data from invoice images using this new model. To follow along, we also have a notebook version you can find [here](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/nEbMzk/c6b76123dbef385aa3576dcda376eb2db876fc28/files/reducto_rolmocr_vast.ipynb)

## Deploying RolmOCR on Vast

### Install Vast

First, we'll install and set up the Vast API.

You can get your API key from the [Account Page](https://cloud.vast.ai/account/) in the Vast Console and set it below in `VAST_API_KEY`.

```bash
pip install vastai==0.2.6
```

```bash
export VAST_API_KEY="" # Your API key here
vastai set api-key $VAST_API_KEY
```

### Search for an Instance

Next, we'll search for an instance to host the model. While `reducto/RolmOCR` requires at least 16GB VRAM, selecting an instance with 60GB VRAM will accommodate larger documents and enable a wider context window.

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 60 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

### Deploy the Instance

Using the instance ID from the search, deploy the model to the instance.

Note: we set `VLLM_USE_V1=1` to use the v1 engine for vLLM, which `reducto/RolmOCR` requires.

```bash
export INSTANCE_ID= # insert instance ID here
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e VLLM_USE_V1=1' --disk 80 --args --model reducto/RolmOCR
```

## Using the OpenAI API to Call `reducto/RolmOCR`

### Download Dependencies

While waiting for the instance to finish setting up and starting to serve the model, we'll install the required Python packages locally.

```bash
pip install --upgrade openai datasets pydantic
```

### Download Sample Invoice Data

Now, we'll use the `datasets` library to load a subset of invoice data from the `katanaml-org/invoices-donut-data-v1` dataset on Hugging Face. This dataset contains 500 annotated invoice images with structured metadata for training document extraction models.

```python
from datasets import load_dataset

# Stream the dataset
streamed_dataset = load_dataset("katanaml-org/invoices-donut-data-v1", split="train", streaming=True)

# Take the first 3 samples
subset = list(streamed_dataset.take(3))
```

Next, we'll create an `encode_pil_image` function to convert images from the dataset into a base64 encoded string format suitable for the OpenAI API.

```python
import base64
import io
from PIL import Image

def encode_pil_image(pil_image):
    # Resize image while maintaining aspect ratio
    max_size = 1024  # Maximum dimension
    ratio = min(max_size / pil_image.width, max_size / pil_image.height)
    new_size = (int(pil_image.width * ratio), int(pil_image.height * ratio))
    resized_image = pil_image.resize(new_size, Image.Resampling.LANCZOS)

    # Convert to bytes with JPEG format and reduced quality for smaller size
    img_byte_arr = io.BytesIO()
    resized_image.save(img_byte_arr, format='JPEG', quality=85)
    img_byte_arr = img_byte_arr.getvalue()
    return base64.b64encode(img_byte_arr).decode("utf-8")
```

### Enforcing Structured Data Extraction

We'll create an `Invoice` Pydantic model for use to store the invoice number and the amount.

```python
from pydantic import BaseModel

class Invoice(BaseModel):
    invoice_number: str
    invoice_amount: str

json_schema = Invoice.model_json_schema()
```

## Configuring the API Client

Here we're creating a function to interface with the RolmOCR endpoint via the OpenAI API.

Set your instance's IP address and port from the [Instances tab](https://cloud.vast.ai/instances/) of the Vast AI Console.

You can see that we're leveraging the json schema fromt he above `Invoice` Model to pull have our model generate structured data.

```python
from openai import OpenAI

VAST_IP_ADDRESS = ""  # Insert your instance IP address
VAST_PORT = ""        # Insert your instance port
base_url = f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"

client = OpenAI(api_key="", base_url=base_url)

model = "reducto/RolmOCR"

def ocr_page_with_rolm(img_base64):
    response = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/png;base64,{img_base64}"},
                    },
                    {
                        "type": "text",
                        "text": "Return the invoice number and total amount for each invoice as a json: {invoice_number : str, invoice_amount: str}",
                    },
                ],
            }
        ],
        extra_body={"guided_json": json_schema},
        temperature=0.2,
        max_tokens=500
    )
    return response.choices[0].message.content
```

### Extract and Display Invoice Data

Iterate over the sample subset of invoices, display the image, extract the invoice number and amount with RolmOCR, and compare the extracted data to the ground truth.

```python
import matplotlib.pyplot as plt
import json

invoices = []
ground_truth = []
for sample in subset:
    # Display invoice image
    plt.figure(figsize=(10, 14))
    plt.imshow(sample['image'])
    plt.axis('off')
    plt.show()

    # Encode and process with OCR
    img_base64 = encode_pil_image(sample['image'])
    result = ocr_page_with_rolm(img_base64)
    result_dict = json.loads(result)
    invoices.append(result_dict)

    # Extract ground truth fields
    ground_truth_i = json.loads(sample["ground_truth"])
    ground_truth_dict = {
        "invoice_number": ground_truth_i["gt_parse"]["header"]["invoice_no"],
        "invoice_amount": ground_truth_i["gt_parse"]["summary"]["total_gross_worth"]
    }
    ground_truth.append(ground_truth_dict)

    print("Ground Truth")
    print(ground_truth_dict)

    print("Extracted Info")
    print(result_dict)
```

### Example Outputs

#### Invoice 1

![Invoice 1](/uploads/reducto_rolmocr_output_1.png)

```
Ground Truth
{'invoice_number': '40378170', 'invoice_amount': '$8,25'}
Extracted Info
{'invoice_number': '40378170', 'invoice_amount': '$8.25'}
```

#### Invoice 2

![Invoice 2](/uploads/reducto_rolmocr_output_2.png)

```
Ground Truth
{'invoice_number': '61356291', 'invoice_amount': '$ 212,09'}
Extracted Info
{'invoice_number': '61356291', 'invoice_amount': '$212.09'}
```

#### Invoice 3

![Invoice 3](/uploads/reducto_rolmocr_output_3.png)

```
Ground Truth
{'invoice_number': '49565075', 'invoice_amount': '$96,73'}
Extracted Info
{'invoice_number': '49565075', 'invoice_amount': '$96,73'}
```

As observed, the output from `reducto/RolmOCR` closely matches the expected ground truth, demonstrating its accuracy in extracting key invoice information. The differences between the extracted and ground truth in these cases is around commas versus decimal points, which can be easily solved for with simple downstream methods.

## Conclusion

In this walkthrough, we demonstrated how to deploy and use RolmOCR on Vast to extract structured data from invoice images efficiently.

RolmOCR delivers high accuracy for critical fields such as invoice numbers and amounts, while staying more robust with image-based proxessing and reduced RAM usage. When combined with Vast.ai's flexible and cost-effective GPU marketplace, this forms a powerful solution for scalable, privacy-focused document processing workflows.

This approach can be extended to extract a wide range of structured data from diverse document formats, unlocking advanced automation and insights for businesses of all sizes.


Learn how to deploy Reducto’s RolmOCR on Vast.ai to extract structured invoice data using affordable GPUs and accurate open-source document OCR.

Running RolmOCR on Vast


NVIDIA's GeForce RTX 5090D was supposed to be a clever workaround – a GPU specially created to comply with U.S. export regulations while still delivering advanced gaming performance to Chinese consumers.

But recent reports suggest that even this modified version may not make it to market.

### NVIDIA RTX 5090D: A China-Specific Solution

The RTX 5090D was designed specifically for the Chinese market in response to the U.S. government's increasingly strict export rules. These rules target high-performance chips like the NVIDIA H100 and H20, and AMD's MI300 series, banning them from being sold in China if they exceed certain memory and I/O bandwidth thresholds.

To get around these limits, NVIDIA modified the RTX 5090D to lower its AI throughput while keeping specs that matter to gamers intact – such as core count, memory size, and clock speeds. That may not have been sufficient, however.

### The Bandwidth Problem

Intended to limit China's ability to develop advanced AI systems, U.S. export regulations on chips impose strict caps on memory and I/O bandwidth: **1,400 GB/s and 1,100 GB/s, respectively – _or_ a combined total over 1,700 GB/s**.

If the RTX 5090D exceeds any of these thresholds, it could still be restricted, despite being designed with reduced AI capabilities and not typically suited for large-scale enterprise workloads. The problem is, it very well might fall into the restricted category based on total bandwidth.

NVIDIA appears to be [taking preemptive action](https://wccftech.com/nvidia-reportedly-prepares-for-a-ban-on-the-geforce-rtx-5090d-in-china/) – asking add-in board (AIB) partners to halt shipments and prep work, and even putting a pause on related orders by cooling module manufacturers.

So far, there's been no official statement from NVIDIA. But after getting burned when the RTX 4090 was unexpectedly swept up in a prior wave of export controls, NVIDIA is reportedly acting early this time with this "precautionary" move amid ongoing regulatory uncertainty between the U.S. and China.

### What This Means for the Market

Whether this sales halt is temporary or permanent remains to be seen, but the wider implications are potentially far-reaching. The RTX 5090D was the only flagship-class Blackwell GPU intended for Chinese gamers. If it's taken off the table, that leaves a gap in the market – and an opportunity for rivals like AMD, which is reportedly preparing a China-exclusive gaming GPU.

Furthermore, these ongoing developments reflect a growing reality: that geopolitical decisions are impacting GPU availability worldwide.

While Chinese consumers lose access to another high-end option, global suppliers are keeping a close eye on what happens next. Constraints in one region can ripple across the supply chain, affecting availability, pricing, and competitive positioning elsewhere, as well.

### High-End Compute On Demand

While geopolitical tensions disrupt supply in some regions, developers, researchers, and creators still need reliable access to powerful hardware. Vast.ai offers a flexible and cost-effective alternative.

Our cloud GPU rental platform delivers high-performance compute – H100s, H200s, RTX 5090s, and more – at a fraction of the typical cost. With on-demand pricing and spot instance options, **Vast users save 5–6X on GPU compute** compared to traditional cloud providers.

Explore your options and get started today\! With Vast, you don't have to wait to access the power you want.


NVIDIA's RTX 5090D, designed specifically for the Chinese market to bypass US export restrictions, now faces uncertainty as it may still exceed bandwidth thresholds. NVIDIA has reportedly asked partners to halt shipments amid regulatory concerns, potentially leaving a gap for competitors in China's high-end GPU market.

RTX 5090D Faces Uncertainty in China Amid Export Crackdown


NVIDIA has [confirmed](https://nvidianews.nvidia.com/news/nvidia-blackwell-rtx-pro-workstations-servers-agentic-ai) the launch of the RTX Pro Blackwell series – a revolutionary generation of workstation and server GPUs. Designed to meet the needs of professional designers, data scientists, developers, and content creators, the lineup features a cutting-edge **RTX Pro 6000 Blackwell** GPU for workstations, plus other desktop and laptop variants and a data center version.

After early signs of the card [appeared in shipping logs](https://vast.ai/article/nvidia-rtx-pro-6000-x-rumors-blackwell-gpu-with-96gb-vram-spotted-in-shipping-log) a couple of months ago, we now have full details straight from the source – and this GPU is no minor upgrade. Built on NVIDIA's Blackwell architecture and featuring an incredible **96GB of GDDR7 memory** on a **512-bit bus**, the RTX Pro 6000 is set to redefine what's possible on a single professional workstation.

Here's what we now know about the RTX Pro 6000 Blackwell GPU.

### NVIDIA RTX Pro 6000: Specs That Mean Business

At the heart of the RTX Pro 6000 is the **GB202 die**, aligning with previous high-end Blackwell GPUs, like the RTX 5090\. But while the RTX 5090 is geared toward gamers and other enthusiasts, the RTX Pro 6000 goes all in for professional use.

We've certainly seen professional-grade GPUs with massive VRAM configurations before – there's the RTX 6000 Ada with 48GB of memory, for instance. However, the RTX Pro 6000 offers **double that amount**, with the faster speeds and efficiency of GDDR7 ECC memory. This makes it one of the most powerful workstation GPUs ever released.

And that's not all. On top of **24,064 CUDA cores** and **1,792 GB/s of memory bandwidth**, here's what else you get with this beast of a machine:

- **Streaming multiprocessors** with new neural shaders, delivering up to **1.5X** faster throughput for AI-augmented graphics workflows.
- **Fourth-gen RT cores** offering up to **2X** faster ray tracing performance for photorealism and complex 3D environments, with optimizations for **RTX Mega Geometry**.
- **Fifth-gen Tensor cores** capable of **4,000 AI TOPS**, supporting **FP4** precision and **DLSS 4 Multi Frame Generation** for cutting-edge AI graphics and accelerated model prototyping.
- **Ninth-gen NVENC** **& sixth-gen NVDEC**, improving encoding and decoding speed and quality for pro video work.
- **PCIe Gen 5 interface**, doubling bandwidth for faster CPU-GPU data transfer and smoother performance on heavy data workloads.
- **DisplayPort 2.1** support for ultra-high refresh rates – up to **4K at 480Hz** and **8K at 165Hz** – enabling multi-monitor setups with high color accuracy.
- **Multi-Instance GPU (MIG)**, allowing secure partitioning into up to **four GPU instances** for better workload isolation and resource allocation.

Not surprisingly, the RTX Pro 6000 does require a substantial **600 watts** of power, albeit only slightly more than the 575 watts of the RTX 5090\. But with that power comes the performance needed for agentic AI, immersive XR, large-scale simulation, and autonomous systems development – workloads where speed, memory, and throughput are non-negotiable.

(For more details and specs, check out the table at the bottom of this page\!)

So the only question that remains is: when can you get your hands on one?

### Availability and Rollout Timeline

The RTX Pro 6000 Workstation Edition is [expected](https://nvidianews.nvidia.com/news/nvidia-blackwell-rtx-pro-workstations-servers-agentic-ai) to **start shipping in May 2025** from manufacturers like BOXX, Dell, HP, Lambda, and Lenovo.

The server version of the RTX Pro 6000 is also on the way, with systems from Cisco, Dell, HP, Lenovo, and Supermicro expected to launch "soon," according to NVIDIA. Laptop variants, featuring **24GB of VRAM**, are expected to ship later in the year, from Dell, HP, Lenovo, and Razer.

NVIDIA hasn't announced official pricing yet, but estimates place the RTX Pro 6000 around $8500 – quite the investment. Fortunately, for those who need top-tier professional-grade GPU performance without the upfront cost, there are more accessible options.

### Access GPU Power Without the Overhead

Here at Vast.ai, we're excited to expand our GPU rental offerings as the **RTX Pro 6000 series** rolls out this year. In the meantime, you can tap into powerful GPUs like the **H100**, **H200**, and other high-performance options, all available on-demand through our platform, today.

With Vast, you can enjoy flexible, cost-effective access to serious compute power – with no long-term commitments and no hardware overhead – that's ready when you are.

### NVIDIA RTX Pro 6000: Key Specs

| Specification                    | RTX Pro 6000                                         |
| :------------------------------- | :--------------------------------------------------- |
| **Architecture**                 | Blackwell                                            |
| **GPU**                          | GB202                                                |
| **GPU Memory**                   | 96GB GDDR7 ECC                                       |
| **Memory Bandwidth**             | 1792 GB/s                                            |
| **Memory Interface**             | 512-bit                                              |
| **CUDA Cores**                   | 24,064                                               |
| **5th-Gen Tensor Cores**         | 752                                                  |
| **4th-Gen RT Cores**             | 188                                                  |
| **L2 Cache**                     | 128 MB                                               |
| **AI TOPS**                      | 4000                                                 |
| **Single-Precision Performance** | 125 TFLOPS                                           |
| **RT Core Performance**          | 380 TFLOPS                                           |
| **System Interface**             | PCIe 5.0 x16                                         |
| **Display Connectors**           | 4x DisplayPort 2.1b                                  |
| **Video Engines**                | 4x NVENC (9th Gen); 4x NVDEC (6th Gen)               |
| **MIG Support**                  | Up to 4x 24GB; or 2x 48GB; or 1x 96GB                |
| **Graphics APIs**                | DirectX 12, Shader Model 6.6, OpenGL 4.6, Vulkan 1.3 |
| **Compute APIs**                 | CUDA 12.8, OpenCL 3.0, DirectCompute                 |
| **Thermal Solution**             | Double-flow-through                                  |
| **Form Factor**                  | 5.4” H x 12” L, dual slot, extended height           |
| **Total Board Power**            | 600 watts                                            |


NVIDIA has confirmed the launch of the RTX Pro Blackwell series – a revolutionary generation of workstation and server GPUs. Designed to meet the needs of professional designers, data scientists, developers, and content creators, the lineup features a cutting-edge RTX Pro 6000 Blackwell GPU for workstations, plus other desktop and laptop variants and a data center version.

NVIDIA RTX Pro 6000 Blackwell: Redefining the Workstation GPU


## Introduction

Meta's Llama 4 is a breakthrough AI model combining state-of-the-art multimodal capabilities with the computational efficiency of mixture-of-experts (MoE) architecture. These models can process both text and images through an early fusion design that seamlessly integrates different modalities. Perhaps most impressively, they support context windows of up to 10 million tokens - dramatically expanding what's possible with large language models.

The Llama 4 family includes:

- **Llama 4 Scout**: 17B active parameters with 16 experts - efficient enough to run on modest GPU setups.

- **Llama 4 Maverick**: 17B active parameters but with 128 experts - offering enhanced capabilities while maintaining efficiency.

- **Llama 4 Behemoth**: 288B active parameters with 16 experts - Meta's most powerful model (not yet publicly released).

What makes these models special:

- **Mixture-of-Experts (MoE)**: Unlike traditional models that activate all parameters for every token, MoE models selectively activate only a fraction - dramatically improving computational efficiency.
- **Early Fusion**: Vision and text processing are integrated directly into the model backbone, enabling true multimodal reasoning.
- **10M Token Context Window**: Process entire books, codebases, or document collections in a single prompt - enabling comprehensive analysis impossible with earlier models.

In this guide, we'll deploy Llama 4 models on Vast.ai using three practical hardware configurations:

1. Llama 4 Scout on 8× H100 GPUs
2. Llama 4 Scout on 4× H100 GPUs
3. Llama 4 Maverick on 8× H200 GPUs

You'll learn exactly how to set up each configuration, deploy the models with the vLLM server, and interact with them through an OpenAI-compatible API. By the end, you'll have hands-on experience running some of the most advanced AI models available today.

## Installing Vast

Let's start by installing the Vast.ai command-line tools and setting up our API key:

```bash
pip install --upgrade vastai
```

```bash
# Here we will set our api key
export VAST_API_KEY="" #Your key here
vastai set api-key $VAST_API_KEY
```

## Running Llama 4 Scout on 8× H100s

### Choosing the Right Hardware

First, we need to search for GPUs on Vast.ai to run the Llama 4 Scout model on 8× H100 GPUs with a 200k token context window. This model requires specific hardware capabilities to run efficiently with vLLM's optimizations. Here are our requirements:

1. 8× H100 GPUs (80GB of VRAM each) to accommodate:

   - Llama 4 Scout model weights (17B active parameters with 16 experts)
   - KV Cache for handling the 200k token context window

2. A static IP address for:

   - Stable API endpoint hosting
   - Consistent client connections

3. At least one direct port that we can forward for:

   - vLLM's OpenAI-compatible API server
   - External access to the model endpoint
   - Secure request routing

4. At least 600GB of disk space to hold the model and other dependencies

Let's search for machines that meet these requirements:

```bash
vastai search offers " \
gpu_name = H100_NVL \
num_gpus = 8 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 600 \
rentable = true"
```

### Deploying the Server via Vast

We'll choose a machine from our search results and copy and paste the id below to set `INSTANCE_ID`.

We use `vastai create instance` to create an instance that:

1. Uses `vllm/vllm-openai:latest` docker image. This gives us an OpenAI-compatible server.
2. Forwards port `8000` to the outside of the container, which is the default OpenAI server port.
3. Uses `--model meta-llama/Llama-4-Scout-17B-16E-Instruct` to serve the Llama 4 Scout model.
4. Uses Llama 4-specific parameters:
   - `--tensor-parallel-size 8` to distribute inference across multiple GPUs.
   - `--max-model-len 200000` to set a 200k token context window.
   - `--override-generation-config='{"attn_temperature_tuning": true}'` for optimal text generation quality.
5. Uses `--disk 600` to ensure that we have 600GB of disk space for model weights and dependencies.

Note: Ensure that you fill in your huggingface token `HUGGING_FACE_HUB_TOKEN` to access the model. You'll need to accept the Llama 4 license terms on the model's huggingface page: https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct

```bash
export INSTANCE_ID=19319856 #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGING_FACE_HUB_TOKEN' --disk 600 --args --model meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 8 --max-model-len 200000 --override-generation-config='{"attn_temperature_tuning": true}'
```

### Get Instance IP Address and Port

Now, we need to get our IP address and port to call our model. We first need to wait for our machine to download the image and the model and start serving. This will take a few minutes. The logs will show you when it's done.

Next, go to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console and find the instance you just created.

At the top of the instance, there is a button with an IP address in it. Click this and a panel will show up of the ip address and the forwarded ports.
You should see something like:

```
Open Ports
XX.XX.XXX.XX:YYYY -> 8000/tcp
```

You will need the IP address (XX.XX.XXX.XX) and the port (YYYY) for the next step.

### Call our Model

#### Install OpenAI

To call our model we will install the OpenAI SDK.

```bash
pip install --upgrade openai
```

#### Download our Data

To show the power of our large context window, we will download The Great Gatsby from Project Gutenberg and ask Llama 4 to summarize it.

Here we download the text for the book:

```python
import requests

# Download the text file to a variable
url = "https://www.gutenberg.org/ebooks/64317.txt.utf-8"
text = requests.get(url).text

# Print the first 100 characters to confirm it worked
print(text[:200])
```

### Call Our Model

We will then use the OpenAI SDK. To do this, we need to set our `VAST_IP_ADDRESS` and `VAST_PORT` that we found above.

```python
from openai import OpenAI

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client with your server URL
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an actual API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

prompt = "please summarize this book in two sentences:" + text

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    messages=[
        {"role": "user", "content": prompt}
    ],

)
print(response.choices[0].message.content)
```

### Model Response

We see that our model was able to ingest a large amount of text and summarize our book.

```
Here is a two-sentence summary of The Great Gatsby:

Set in the roaring twenties, The Great Gatsby is a classic novel by F. Scott Fitzgerald that revolves around the mysterious millionaire Jay Gatsby and his obsession with winning back his lost love, Daisy Buchanan. Through the eyes of narrator Nick Carraway, the novel explores themes of wealth, class, love, and the corrupting influence of materialism in the excesses of the Jazz Age
```

## Running Llama 4 Scout on 4× H100s

After successfully deploying Scout on 8× H100s, we'll now try a more cost-effective setup using 4× H100s. This will reduce our context window capacity but allow us to save on GPU costs.

We'll start by searching for an instance that meets our specifications:

```bash
vastai search offers " \
num_gpus = 4 \
gpu_name = H100_SXM \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 600 \
rentable = true"
```

We'll choose a machine from our search and create our instance with 4 GPUs and a 100k token context window using `--tensor-parallel-size 4` and `--max-model-len 100000`:

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGING_FACE_HUB_TOKEN' --disk 600 --args --model meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 4 --max-model-len 100000 --override-generation-config='{"attn_temperature_tuning": true}'
```

Once our instance is running, we'll set the new `VAST_IP_ADDRESS` and `VAST_PORT` and call our instance like we did earlier:

```python
from openai import OpenAI

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client with your server URL
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an actual API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

prompt = "please summarize this book in two sentences:" + text

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    messages=[
        {"role": "user", "content": prompt}
    ],

)
print(response.choices[0].message.content)
```

### Model Response

```
Here is a two-sentence summary of The Great Gatsby:

Set in the roaring twenties, The Great Gatsby is a classic novel by F. Scott Fitzgerald that revolves around the mysterious millionaire Jay Gatsby and his obsession with winning back his lost love, Daisy Buchanan. Through the eyes of narrator Nick Carraway, the novel explores themes of love, greed, class, and the corrupting influence of wealth, ultimately leading to a tragic confrontation that exposes the dark underbelly of the American Dream.
```

## Running Llama 4 Maverick on 8× H200s

Finally, we'll upgrade to Llama 4's larger Maverick model with a 100k token context window on 8× H200 GPUs - a more powerful configuration that delivers enhanced capabilities.

To start, we'll search for an H200 machine with 8 GPUs:

```bash
vastai search offers " \
num_gpus = 8 \
gpu_name = H200 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 1000 \
rentable = true"
```

We'll find a machine from our search and create an instance with Llama 4 Maverick using 8 GPUs and a 100k token context window by setting `--disk 1000`, `--model meta-llama/Llama-4-Maverick-17B-128E-Instruct`, `--tensor-parallel-size 8` and `--max-model-len 100000`:

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGING_FACE_HUB_TOKEN' --disk 1000 --args --model meta-llama/Llama-4-Maverick-17B-128E-Instruct --tensor-parallel-size 8 --max-model-len 100000 --override-generation-config='{"attn_temperature_tuning": true}'
```

Finally, we'll set the new `VAST_IP_ADDRESS` and `VAST_PORT`, change our model to `"meta-llama/Llama-4-Maverick-17B-128E-Instruct"`, and call our instance:

```python
from openai import OpenAI

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client with your server URL
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an actual API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

prompt = "please summarize this book in two sentences:" + text

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct",
    messages=[
        {"role": "user", "content": prompt}
    ],

)
print(response.choices[0].message.content)
```

### Model Response

```
Here is a two-sentence summary of The Great Gatsby:

The novel is narrated by Nick Carraway, who becomes fascinated with his wealthy neighbor Jay Gatsby and becomes entangled in Gatsby's quest to win back his lost love, Daisy Buchanan, through a complex web of relationships and tragic events. Ultimately, Gatsby's dream is destroyed, and he is murdered by George Wilson, the husband of Myrtle Wilson, who was having an affair with Tom Buchanan, Daisy's husband, highlighting the corrupting influence of wealth and the elusiveness of the American Dream.
```

## Conclusion

In this post, we've demonstrated how to deploy Meta's Llama 4 models on Vast.ai using three specific hardware configurations: Llama 4 Scout on 8× H100 GPUs with a 200k token context window, Scout on 4× H100 GPUs with a 100k context window, and Llama 4 Maverick on 8× H200 GPUs with a 100k context window. We've shown that with the right GPU setup, you can easily deploy and interact with these models through an OpenAI-compatible API.

Now that you have a basic deployment working, here are some next steps you might consider:

- **Expand the context window**: While we demonstrated context windows of 100k-200k tokens, Llama 4 supports up to 10M tokens. Try increasing `--max-model-len` to experiment with longer documents, books, or codebases. Note: You will also need to increase the number of GPUs.

- **Explore multimodal capabilities**: The tutorial focused on text processing, but Llama 4 is natively multimodal. Try adapting the deployment to accept image inputs using vLLM's multimodal capabilities and the early fusion architecture.

By leveraging Vast.ai's flexible GPU infrastructure, you can cost-effectively experiment with these cutting-edge models and build applications that take advantage of their sophisticated reasoning capabilities.


Learn how to deploy and interact with Meta's Llama 4 models on Vast.ai using three specific hardware configurations: Llama 4 Scout on 8× H100 GPUs with a 200k token context window, Scout on 4× H100 GPUs with a 100k context window, and Llama 4 Maverick on 8× H200 GPUs with a 100k context window.

Running Llama 4 Models on Vast.ai


The H100 and H200 GPUs represent the top end of NVIDIA's current lineup for AI and high-performance computing (HPC). Both are designed for serious, large-scale workloads – from training foundation models to running complex scientific simulations.

At a glance, the two GPUs may appear similar, but for those working across enterprise environments or pushing the boundaries of exascale computing, the differences between them can have a major impact on performance and scalability.

Today, we're taking a look at how the H100 and H200 stack up against each other to help you decide which of these two heavyweights may be the better fit for your needs.

### NVIDIA H100: The Proven Performer

The H100 GPU made waves when it launched in March 2023\. Quickly adopted by big names like OpenAI, Meta, and Stability AI, it became the engine behind some of the most advanced generative AI systems in production today.

Built on NVIDIA's innovative Hopper architecture, the H100 was engineered for scale, handling everything from trillion-parameter language models to high-throughput inference. It introduced **fourth-gen Tensor Cores** and a dedicated **Transformer Engine**, designed to accelerate deep learning workloads with speed and FP8 precision.

The H100 boasts up to a **30X boost in performance** for large language models (LLMs) over its predecessor, the A100 GPU, according to NVIDIA. This massive improvement helped cement the H100 as the new standard for foundational model training, and to this day, it's still one of the very best GPUs on the market.

#### Some of the H100's more notable features include:

-   **80 GB of HBM3 memory**, with some models offering up to 94 GB.

-   **3.35 TB/s of memory bandwidth** to handle data-intensive workloads.

-   **Fourth-gen Tensor Cores** with superior throughput performance, along with an advanced Transformer Engine and FP64 and FP8 capabilities to reduce memory usage.

-   **PCIe Gen5 and fourth-gen NVLink** (with 900 GB/s of GPU-to-GPU interconnect), as well as NDR Quantum-2 InfiniBand networking for efficient scalability.

-   **Multi-Instance GPU (MIG) support** for secure partitioning into up to 7 isolated right-size instances that maximize quality of service for multi-tenant environments and parallel workloads.

If your projects involve massive datasets, large-scale simulations, or advanced AI training for foundational models, the H100 has proven itself as a reliable, high-performance solution. But with the arrival of the H200 in late 2024, NVIDIA has taken things a step further, introducing upgrades that push the envelope even more for memory-intensive and high-throughput workloads.

We ran a side-by-side comparison of inference performance on the **DeepSeek R1 Distill Llama 8B** model using `vLLM` with both NVIDIA H100 and H200 GPUs on Vast.ai.

#### **Benchmark Configuration**

-   **Model:** `deepseek-ai/DeepSeek-R1-Distill-Llama-8B`
-   **Requests:** 5,000
-   **Batch:** 512
-   **Input Tokens:** 16
-   **Output Tokens:** 128
-   **Backend:** `vLLM`

---

### **Results**

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Metric
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H100
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H200
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Benchmark Duration
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                45.28 s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                40.02 s
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Request Throughput
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                110.42 req/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                124.93 req/s
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Output Throughput
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                13,650 tok/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                15,435 tok/s
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Total Throughput
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                15,416 tok/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                17,434 tok/s
            </td>
        </tr>
    </tbody>
</table>

### NVIDIA H200: The Supercharged Successor

The H200 builds directly on the foundation laid by the H100, bringing meaningful improvements that target one of the biggest bottlenecks in AI and HPC workloads: memory.

While it shares the same Hopper architecture as the H100 and has similar core compute specs, the H200 significantly increases both memory capacity and bandwidth, two factors that become critical when operating at enterprise scale or pushing into exascale territory.

Equipped with **141 GB of HBM3e memory** and **4.8 TB/s of bandwidth**, the H200 nearly doubles the memory of the H100 and delivers about **1.4X faster** data access. Early benchmarks reflect that improvement. In MLPerf testing using Llama 2 70B, the H200 achieved over 31,000 tokens per second – about **45% faster than the H100**.

<img
    src="/uploads/h100vsh200_2.png"
    alt="Image credit: NVIDIA"
    style={{ width: "100%" }}
/>

The H200 retains all of the architectural strengths of the H100, including **fourth-gen NVLink**, **PCIe Gen5**, **MIG support**, and full **Tensor Core and Transformer Engine capabilities**. But for developers and organizations looking to squeeze out more performance without overhauling their infrastructure, the H200 offers a straightforward upgrade path with [measurable gains](https://resources.nvidia.com/en-us-data-center-overview/hpc-datasheet-sc23-h200).

#### To recap, here are a few advantages of the H200:

-   **141 GB of HBM3e memory at 4.8 TB/s** – nearly double the capacity of the H100, with 1.4X more memory bandwidth.

-   **Up to 1.9X the inference performance** of the H100 when handling LLMs, with more efficient processing per token and per inference cycle.

-   **50% reduced energy use and total cost of ownership** (TCO) due to improved thermal management and energy efficiency.

-   **Drop-in compatibility with existing Hopper-based systems**, enabling accelerated performance gains in H100-optimized infrastructure with minimal or no additional changes.

In short, the H200 represents a meaningful leap beyond the H100 in memory capacity, bandwidth, and throughput, bringing a new level of performance to AI and HPC workloads. That comes at a premium price, of course. But if you're weighing the tradeoffs between the two GPUs in terms of specs alone, let's take a side-by-side look at how they compare.

### H100 vs. H200: Key Specs Compared

For reference, here's a quick outline of the various features and specs of the NVIDIA H100 and H200:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Specification
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H100
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H200
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                GPU Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">Hopper</td>
            <td class="border border-gray-300 px-4 py-2 text-center">Hopper</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                GPU Memory
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                80 GB HBM3
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                141 GB HBM3e
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                GPU Memory Bandwidth
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3.35 TB/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                4.8 TB/s
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP64 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                33.5 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                33.5 TFLOPS
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">FP32</td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                67 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                67 TFLOPS
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                TF32 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                989 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                989 TFLOPS
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP16 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1,979 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1,979 TFLOPS
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP8 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TFLOPS
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                INT8 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TOPS
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Decoders
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                7 NVDEC, 7 NVJPEG
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                7 NVDEC, 7 NVJPEG
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                L2 Cache
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">50 MB</td>
            <td class="border border-gray-300 px-4 py-2 text-center">50 MB</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Max. Thermal Design Power
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 700 W
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 1000 W
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Multi-Instance GPUs
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 7 MIGs @ 10GB each
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 7 MIGs @ 18GB each
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Form Factor
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">SXM</td>
            <td class="border border-gray-300 px-4 py-2 text-center">SXM</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Interconnect
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                PCIe 5.0 x16
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                PCIe 5.0 x16
            </td>
        </tr>
    </tbody>
</table>

Ultimately, choosing the right GPU largely depends on your workload demands and performance priorities.

### Use Cases: Which GPU Should You Choose?

While both the H100 and H200 excel in AI and HPC environments, they serve slightly different needs.

#### You should consider the H100 if you prefer:

-   **A proven, high-performance GPU** for training medium to large models and handling intensive AI or HPC workloads.
-   **Use cases involving compute-bound workloads**, where total memory capacity is not as critical.
-   **A time-tested solution** already in widespread production use, balancing performance with lower power and cost.

#### Whereas the H200 may be a better fit if you need:

-   **Greater memory capacity and bandwidth** for running _even larger_ models, handling memory-bound workloads, or scaling LLM inference on the enterprise level.
-   **Higher throughput**, especially for tasks involving extended context windows, multi-modal AI, or very large batch sizes.
-   **A forward-looking upgrade path** with improved energy efficiency and lower TCO, thanks to higher performance per watt.

\------------------------------

At **Vast.ai**, we understand that choosing a GPU isn't only about specs. Often, the decision comes down to access and affordability, as well – especially when we're talking about some of the most powerful and expensive GPUs in existence. (Speaking of, check out our previous posts comparing the [H100 vs. A100](https://vast.ai/article/H100-vs-A100-Comparing-two-Powerhouse-GPUs), and the [H100 vs. L40S](https://vast.ai/article/nvidia-h100-vs-l40s-power-meets-versatility).)

That's why our cloud GPU rental platform is built to offer high-performance compute at a fraction of the typical cost. Vast gives you flexible, cost-effective access to H100s, H200s, and more.

With options for on-demand pricing and budget-friendly spot instances, you can scale up as needed without overspending. On average, **Vast users save 5- 6X on GPU compute** compared to traditional cloud providers.

Explore our platform today and get the power you need, when you need it – with no upfront investment required.


The H100 and H200 GPUs represent the top end of NVIDIA's current lineup for AI and high-performance computing (HPC). Both are designed for serious, large-scale workloads – from training foundation models to running complex scientific simulations.

NVIDIA H100 vs. H200: Two Hopper-based Heavyweights


Deep Cogito models are hybrid reasoning Large Language Models (LLMs) uniquely designed to operate in two distinct modes within a single deployment: standard direct answering and detailed step-by-step reasoning. Unlike traditional models that require separate deployments for different reasoning capabilities, Deep Cogito can seamlessly toggle between these modes using simple prompt engineering.

This guide demonstrates how to deploy `deepcogito/cogito-v1-preview-llama-8B` on Vast.ai and leverage its dual reasoning capabilities through vLLM's OpenAI-compatible API.

## Deploying on Vast.ai

First, install the Vast.ai SDK and configure your API key:

```bash
pip install --upgrade vastai
export VAST_API_KEY="YOUR_KEY_HERE"
vastai set api-key $VAST_API_KEY
```

Search for a suitable instance that meets the model's requirements:

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 60 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

Deploy the model using vLLM's OpenAI-compatible API:

```bash
vastai create instance YOUR_INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 80 --args --model deepcogito/cogito-v1-preview-llama-8B
```

## Using Deep Cogito with OpenAI SDK

Install the OpenAI SDK:

```bash
pip install --upgrade openai
```

### Enabling Thinking Mode

Deep Cogito provides two methods to toggle thinking mode:

1. **Hugging Face Transformers Library Method**:
   When using the Hugging Face Transformers library directly, you can enable thinking mode using the `enable_thinking=True` flag with the tokenizer's `apply_chat_template` method. This activates the model's reasoning process, displaying its thought process within `<thinking>` tags before providing the final answer.

2. **vLLM OpenAI-compatible API Method**:
   When using vLLM's OpenAI-compatible API (as in our Vast.ai deployment), use a system prompt instead. The official prompt is:
   ```
   "Enable deep thinking subroutine."
   ```
   
   For the 8B model specifically, we recommend using this enhanced version for more consistent results:
   ```
   "Enable deep thinking subroutine. \n\n YOU MUST USE THE THINKING SUBROUTINE"
   ```

Here's an implementation example using the OpenAI SDK:

```python
from openai import OpenAI

# Vast.ai instance configuration
VAST_IP_ADDRESS = "YOUR_VAST_IP"
VAST_PORT = "YOUR_VAST_PORT"

# Initialize the OpenAI client
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

# Function for reasoning mode
def chat_with_model_reasoning(prompt):
    response = client.chat.completions.create(
        model="deepcogito/cogito-v1-preview-llama-8B",
        messages=[
            {"role": "system", "content": "Enable deep thinking subroutine. \n\n YOU MUST USE THE THINKING SUBROUTINE"},
            {"role": "user", "content": prompt}
        ],
    )
    return response.choices[0].message.content

# Function for standard mode
def chat_with_model(prompt):
    response = client.chat.completions.create(
        model="deepcogito/cogito-v1-preview-llama-8B",
        messages=[
            {"role": "user", "content": prompt}
        ],
    )
    return response.choices[0].message.content
```

## Response Comparison

The following examples demonstrate how the model's responses differ between thinking mode and standard mode when asked to explain Large Language Models:

### With Thinking Mode:

```
<think>
Okay, I need to give a short introduction to Large Language Models (LLMs). Let me start by recalling what I know about them. LLMs are a type of artificial intelligence that uses neural networks to process and generate human-like text. They're trained on massive amounts of text data from the internet, books, and other sources. This training allows them to understand and generate contextually relevant responses.

Wait, I should probably mention that the training process involves machine learning, specifically deep learning. They use transformer architectures, which were introduced by Google in 2017. Transformers revolutionized NLP by allowing the model to process input in parallel, which makes them more efficient than previous sequential models.

But maybe I'm getting a bit too technical. The user might just want a basic overview. Let me focus on the key aspects. LLMs are known for their ability to understand natural language and generate coherent text. They can engage in conversations, answer questions, write stories, translate languages, and even help with coding. But they also have limitations, like sometimes providing incorrect or biased information.

I should also mention some popular examples, like GPT-3, BERT, and the more recent ones like GPT-4. These models vary in size and capability. For instance, GPT-3 has 175 billion parameters, making it one of the largest models. GPT-4 is even bigger and claimed to have more capabilities, though it's not publicly available yet.

Also, LLMs are used in various applications, such as chatbots, virtual assistants, and content generation tools. They can help with tasks like summarization, question answering, and sentiment analysis. But it's important to note that they're not perfect and can have issues like hallucinations (making up information) or biases inherited from the training data.

Hmm, I might need to simplify this a bit. Maybe start with a basic definition and then highlight their key features and applications. Avoid too much technical jargon unless necessary.

Wait, I should also mention the training data aspect. LLMs learn from enormous datasets, which means they can pick up on language patterns, idioms, and even cultural references. But this also means they can reflect biases present in the data, which is a concern.

In summary, LLMs are advanced AI models that can understand and generate human-like text using deep learning techniques. They excel at language tasks but have limitations. They're widely used in various applications but also require careful consideration of their ethical implications.
</think>

Large Language Models (LLMs) are advanced AI systems that can process and understand language in ways similar to human intelligence. These models use neural networks and are trained on vast amounts of text data from the internet, books, and other sources.

Here are the key things you should know about LLMs:

1. **How they work**: LLMs use a technology called transformer architecture, which allows them to process language in parallel and maintain context in conversations. This means they can understand what you're saying and respond appropriately.

2. **Capabilities**: LLMs are incredibly versatile, able to:
   - Engage in conversations
   - Answer questions
   - Write stories and articles
   - Help with coding
   - Translate languages
   - Summarize text
   - Create content

3. **Examples**: Popular LLMs include GPT-3, BERT, and ChatGPT. These models vary in size and capability, but they all share the ability to understand and generate human-like text.

4. **Important limitations**: While powerful, LLMs aren't perfect. They can:
   - Make up information (hallucinations)
   - Reflect biases from their training data
   - Sometimes provide incorrect answers

5. **Applications**: LLMs are used in:
   - Chatbots
   - Virtual assistants
   - Content generation tools
   - Language translation services
   - Text analysis and processing

As these models continue to evolve, they're becoming increasingly important in our digital world, though we need to remain mindful of their limitations and potential biases.
```

### Without Thinking Mode:

```
Large Language Models (LLMs) are AI systems that use neural networks to process and generate human-like text. They're pre-trained on massive datasets of text and can understand context, generate coherent responses, and perform various language tasks like translation, summarization, and question-answering.

The most well-known LLMs include GPT (Generative Pre-trained Transformer) models created by OpenAI, BERT (Bidirectional Encoder Representations from Transformers) created by Google, and others. They've become increasingly useful for applications like chatbots, content generation, and language understanding tasks.

Key characteristics of LLMs include their ability to understand context, generate human-like text, and adapt to different tasks, though they can also sometimes produce incorrect or biased information.
```

## Conclusion

Deep Cogito on Vast.ai offers a cost-effective solution for accessing a powerful reasoning LLM. The system prompt technique enables visibility into the model's reasoning process, making it particularly valuable for complex tasks such as coding, mathematical problems, and detailed analysis.

While the `enable_thinking` flag isn't available through vLLM's API, the system prompt approach can be highly effective with proper configuration. Users should experiment with prompt variations to achieve optimal results, as factors such as emphasis and repetition can significantly impact the consistency of the thinking mode, especially with the 8B model.

For production environments where understanding the model's reasoning process is crucial, this approach provides a robust and economical solution. We recommend testing different prompt variations to find the optimal configuration for your specific use case, as the effectiveness of thinking mode can vary depending on the complexity of the task and the chosen model size.


Learn how to deploy and interact with Deep Cogito’s hybrid reasoning LLM (cogito-v1-preview-llama-8B) on Vast.ai. This guide covers both standard and deep thinking modes via vLLM’s OpenAI-compatible API.

Running Deep Cogito on Vast.ai


## Introduction

In multi-speaker audio recordings like meetings, podcasts, or interviews, knowing who spoke when is crucial for many applications. This process, known as Speaker Diarization, partitions an audio stream into segments according to speaker identity.

[PyAnnote Audio](https://github.com/pyannote/pyannote-audio), an open-source toolkit built on PyTorch, offers state-of-the-art models for speaker diarization that are both accurate and accessible. Running these models on Vast.ai provides a cost-effective solution for processing large audio datasets without investing in expensive hardware. This combination gives developers and researchers the tools they need to build sophisticated audio processing pipelines at a fraction of the cost of traditional cloud providers.

### Why Speaker Diarization Matters

Speaker Diarization provides several key benefits for audio processing pipelines:

1. **Speaker Identification**: It identifies different speakers in a conversation, meeting, or any multi-speaker audio recording.

2. **Improved Transcription**: When combined with speech-to-text systems, diarization allows for speaker-attributed transcripts, making it clear who said what.

3. **Processing Efficiency**: By segmenting audio by speaker and removing non-speech portions, diarization can significantly reduce the computational load for downstream tasks like speech recognition, allowing these systems to process only relevant speech segments rather than the entire audio file.

4. **Audio Indexing**: Makes audio content searchable by speaker, allowing users to find all segments where a specific person speaks.

### What This Guide Covers

In this guide, we will:

-   Set up the Pyannote Audio Speaker Diarization pipeline
-   Process audio files to detect different speakers and their speaking turns
-   Calculate speaking time for each identified speaker
-   Identify regions with overlapping speech
-   Extract and save speaker-specific segments from the input audio
-   Play and verify the diarization results

The output will be a collection of audio files separated by speaker, making them ready for further processing in speech-to-text pipelines or speaker-specific analysis.

### Why VAST.ai

VAST.ai offers a marketplace approach to GPU rentals that provides significant advantages for audio processing tasks. Unlike traditional cloud providers, VAST.ai allows you to:

-   Rent precisely the GPU capacity needed for your workload
-   Access GPUs at more affordable rates
-   Access more types of GPU SKUs, particularly lower RAM/more affordable GPUs
-   Avoid long-term commitments for experimental projects

For speaker diarization specifically, VAST.ai is particularly well-suited as these models typically require GPU acceleration but don't demand extensive resources. Users can rent just the right amount of computing power for this specific task without overpaying for unused capacity, making it an economical choice for audio processing workflows that might otherwise be cost-prohibitive on traditional cloud platforms.

## Download This Notebook

To follow along with this tutorial, you can download the complete Jupyter notebook:

[Notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/q7MzLr/ee586f7e9bfbcdb2fb48f059b7bf64caf92fa3bd/files/pyannote_diarization_vast.ipynb)

Having the notebook will allow you to execute the code blocks as you read through this guide, making it easier to understand and experiment with the diarization implementation.

## Choosing an Instance

For running the Pyannote Speaker Diarization model on VAST.ai, you'll need a relatively modest GPU setup. The pyannote/speaker-diarization-3.1 model runs in pure PyTorch and is designed to be efficient. Here are the recommended specifications:

-   GPU: A low-end GPU like an RTX 3060 or 4060 would be sufficient.
-   VRAM: 6-8GB of VRAM should be adequate as the Pyannote diarization pipeline is relatively efficient.
-   RAM: 8-16GB system RAM is recommended for processing audio files.
-   Storage: At least 10GB for the model, dependencies, and your audio files.
-   CUDA: Make sure the instance has CUDA installed (version 11.0+ recommended).
-   Python: Python 3.8+ with PyTorch installed.

## Selecting an Instance

Follow these steps to set up your environment:

1. Ensure that you have a Vast.ai account
2. Go to the Vast Templates in the Console https://cloud.vast.ai/templates/
3. Select the `PyTorch (CuDNN Runtime)` Template
4. Filter for an instance with:
    - 1 GPU
    - 6-8GB of VRAM
    - 8-16GB system RAM
    - 10GB of storage
5. Select an instance and click rent
6. Install the Vast TLS certificate in your browser to access the notebook server https://docs.vast.ai/instances/jupyter#1SmCz
7. Go to your Instances https://cloud.vast.ai/instances/ and click "Open" to access the jupyter server on your instance.
8. Upload the notebook to the server or create a new notebook

## Installing Dependencies

Let's start by installing the necessary Python packages:

```python
%%bash
pip install pyannote.audio
pip install pydub
pip install librosa
pip install datasets
```

We also need to install FFmpeg for audio processing:

```python
%%bash
apt-get update && apt-get install -y ffmpeg
```

## Setting Up Your Hugging Face Token

Pyannote models are hosted on Hugging Face, so you'll need to set up authentication:

```python
# Make sure you've accepted the user conditions at:
# https://huggingface.co/pyannote/speaker-diarization-3.1
# https://huggingface.co/pyannote/segmentation-3.0

HF_TOKEN = ""  # Add your token here
```

Ensure that you have accepted the terms for the models at the URLs above. The models are free to use, but you must agree to their terms of service.

## Downloading Test Data

For this tutorial, we'll use a sample file from the AMI Meeting Corpus dataset, which is a collection of 100 hours of meeting recordings. This dataset is perfect for testing speaker diarization as it contains natural multi-speaker conversations.

```python
from datasets import load_dataset
import os
import soundfile as sf

# Create a directory to save the files
os.makedirs("ami_samples", exist_ok=True)

# Load the dataset with the correct split
dataset = load_dataset("diarizers-community/ami", "ihm", split="train", streaming=True)

# Load just one sample
n_samples = 1
samples = list(dataset.take(n_samples))

for i, sample in enumerate(samples):
    audio = sample["audio"]
    audio_array = audio["array"]
    sampling_rate = audio["sampling_rate"]

    # Calculate duration in seconds
    duration = len(audio_array) / sampling_rate

    # Use soundfile to save the audio
    output_path = f"ami_samples/sample_{i}.wav"
    sf.write(output_path, audio_array, sampling_rate)

    print(f"Saved {output_path} - Speaker: {sample['speakers']} - Duration: {duration:.2f} seconds")
```

## Speaker Diarization

First, let's set up the Speaker Diarization pipeline:

```python
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
        "pyannote/speaker-diarization-3.1",
        use_auth_token=HF_TOKEN
)

# Move pipeline to appropriate device
pipeline = pipeline.to(device)
```

Next, we'll process our audio file to identify different speakers and their speaking turns:

```python
# Process the audio file
audio_file = "./ami_samples/sample_0.wav"
print(f"Processing {audio_file} on {device}")
output = pipeline(audio_file)

print("Voice activity segments:")

for segment, _, speaker in output.itertracks(yield_label=True):
    result = f"{segment.start:.2f} --> {segment.end:.2f} (duration: {segment.duration:.2f}s) Speaker: {speaker}"
    print(result)
```

The output will look something like this:

```
Processing ./ami_samples/sample_0.wav on cuda
Voice activity segments:
18.36 --> 18.42 (duration: 0.07s) Speaker: SPEAKER_03
23.01 --> 25.63 (duration: 2.62s) Speaker: SPEAKER_03
27.08 --> 27.64 (duration: 0.56s) Speaker: SPEAKER_05
...
```

## Additional Analytics

Now that we have processed our file, let's explore some useful features of the Pyannote SDK:

### Speaker Time

Here we calculate the total speaking time for each speaker:

```python
for speaker in output.labels():
    speaking_time = output.label_duration(speaker)
    print(f"Speaker {speaker} total speaking time: {speaking_time:.2f}s")
```

This will output something like:

```
Speaker SPEAKER_00 total speaking time: 558.98s
Speaker SPEAKER_01 total speaking time: 18.98s
Speaker SPEAKER_02 total speaking time: 22.88s
Speaker SPEAKER_03 total speaking time: 469.68s
Speaker SPEAKER_04 total speaking time: 698.02s
Speaker SPEAKER_05 total speaking time: 190.70s
Speaker SPEAKER_06 total speaking time: 5.74s
...
```

### Speaker Overlap

Pyannote can identify regions where multiple speakers are talking simultaneously:

```python
overlap = output.get_overlap()
print(f"Overlapping speech regions: {overlap}")
```

This will output the segments where more than one speaker is speaking.

```
Overlapping speech regions:
[[ 00:00:27.672 -->  00:00:27.689]
 [ 00:00:38.337 -->  00:00:38.860]
 [ 00:00:40.395 -->  00:00:40.463]
```

### Filter by Speaker

We can also filter the diarization output to focus on a specific speaker:

```python
speaker = "SPEAKER_06"
speaker_turns = output.label_timeline(speaker)
print(f"Speaker {speaker} speaks at:")
for speaker_turn in speaker_turns:
    print(speaker_turn)
```

This will show the segments where this speaker is speaking.

```
Speaker SPEAKER_06 speaks at:
[ 00:03:45.767 -->  00:03:45.852]
[ 00:03:55.386 -->  00:03:55.521]
[ 00:05:07.257 -->  00:05:07.274]
...
```

## Extracting Speaker Segments

To verify our results and prepare the audio for further processing, let's split the original audio into segments by speaker:

```python
import shutil
from pydub import AudioSegment

def split_audio_by_segments(audio_path, diarization_output, output_dir="output_segments"):
    """
    Split an audio file into multiple files based on diarization output

    Parameters:
    -----------
    audio_path: str
        Path to the input audio file
    diarization_output: Annotation
        Pyannote diarization output
    output_dir: str
        Directory to save the output segments
    """
    # Clear the output directory if it exists
    if os.path.exists(output_dir):
        shutil.rmtree(output_dir)

    # Create output directory
    os.makedirs(output_dir, exist_ok=True)

    # Load the audio file
    audio = AudioSegment.from_file(audio_path)

    # Extract each segment with speaker information
    for i, (segment, _, speaker) in enumerate(diarization_output.itertracks(yield_label=True)):
        # Convert seconds to milliseconds
        start_ms = int(segment.start * 1000)
        end_ms = int(segment.end * 1000)

        # Extract segment
        segment_audio = audio[start_ms:end_ms]

        # Generate output filename with speaker information
        filename = os.path.basename(audio_path)
        name, ext = os.path.splitext(filename)
        output_path = os.path.join(output_dir, f"{name}_segment_{i+1:04d}_{start_ms:08d}ms-{end_ms:08d}ms_speaker_{speaker}{ext}")

        # Export segment
        segment_audio.export(output_path, format=ext.replace('.', ''))
        print(f"Saved segment {i+1} to {output_path} (Speaker: {speaker})")

# Apply the function to our audio file
split_audio_by_segments(audio_file, output)
```

## Inspecting Results

To verify our results, we'll create a function to play audio files in our Jupyter environment:

```python
import librosa
from IPython.display import Audio, display

def play_audio(file_path, sr=None):
    """
    Play an audio file in a Jupyter notebook.
    """
    # Load the audio file
    y, sr = librosa.load(file_path, sr=sr)

    # Display an audio widget to play the sound
    audio_widget = Audio(data=y, rate=sr)
    display(audio_widget)
```

Now, let's listen to a few clips to verify that the speakers were correctly identified and isolated.

```python
import os
audio_dir = "./output_segments/"

audio_files = os.listdir(audio_dir)
audio_files.sort()

n_offset = 21
n_clips = 5

for fname in audio_files[n_offset:n_clips + n_offset]:
    print(f"File: {fname}")

    # Extract speaker info if present in filename
    if "_speaker_" in fname:
        speaker_part = fname.split("_speaker_")[1].split(".")[0]
        print(f"Speaker: {speaker_part}")

    play_audio(audio_dir + fname)
```

Around file 21 or so, we see a 14-second clip of `SPEAKER_00` speaking:

`File: sample_0_segment_0025_00055364ms-00070045ms_speaker_SPEAKER_00.wav Speaker: SPEAKER_00`

A few files later we see a 10-second clip of `SPEAKER_00` speaking again:

`File: sample_0_segment_0029_00071530ms-00081840ms_speaker_SPEAKER_00.wav Speaker: SPEAKER_00`

Note: The filenames may change. Each time we run the model there may be a different number of clips. The diarization model sometimes captures `<1` second of audio and labels it as a speaker.

The 10-second clip is audio of `SPEAKER_00` about the meeting agenda. It is a file of just `SPEAKER_00` talking.

The 14-second clip has two other speakers speaking. This seems like an error at first, but we will notice two other files that capture the additional speakers:

`sample_0_segment_0026_00059228ms-00060325ms_speaker_SPEAKER_02.wav`
`sample_0_segment_0027_00061793ms-00062924ms_speaker_SPEAKER_02.wav`

With further processing, we could remove these from the `SPEAKER_00` file if necessary for our application.

We can also use the Pyannote overlap function to verify that we did have overlapping speech at this time in the audio file.

## Verifying Speaker Overlap

Finally, let's check for regions where multiple speakers are talking simultaneously:

```python
overlap = output.get_overlap()
for overlap_ts in overlap:
    print(f"Overlapping speech regions: {overlap_ts}")
```

Here we see two overlapping segments that match up with the overlapping files we found above.

`sample_0_segment_0026_00059228ms-00060325ms_speaker_SPEAKER_02.wav`
`sample_0_segment_0027_00061793ms-00062924ms_speaker_SPEAKER_02.wav`

```
Overlapping speech regions: [ 00:00:59.228 -->  00:01:00.325]
Overlapping speech regions: [ 00:01:01.793 -->  00:01:02.924]
```

## Conclusion

This tutorial demonstrates how Pyannote's speaker diarization on VAST.ai provides a powerful solution for identifying "who spoke when" in multi-speaker recordings. The implementation offers several advantages:

-   **Accuracy and Efficiency**: Pyannote accurately identifies different speakers and their speaking times, even detecting overlapping speech
-   **Practical Applications**: Enables speaker-attributed transcription, conversation analytics, and content indexing for meetings, podcasts, and interviews
-   **Cost-Effective Processing**: VAST.ai's affordable GPU rentals make processing large audio datasets accessible without expensive hardware investments

The Pyannote diarization model provides impressive results out of the box, and running it on VAST.ai makes it accessible and affordable for a wide range of applications. Whether you're building a meeting transcription service, analyzing call center interactions, or researching conversation dynamics, this approach gives you a solid starting point.

Look out for more content from Vast about other audio processing tasks!


Run speaker diarization with Pyannote on VAST.ai to detect who spoke when in audio files—cost-effective, accurate, and easy to integrate with your pipeline.

Speaker Diarization with Pyannote on VAST


We're pleased to announce that Vast.ai has successfully completed the SOC 2 Type I audit, a significant milestone as we continue to deliver secure and scalable GPU compute to a global user base.

Security remains central to everything we do here at Vast.ai. We've spent the past 6+ years delivering reliable service worldwide while meeting the [highest standards](https://vast.ai/article/security-and-compliance-at-vast-ai) of regulatory compliance. 

### Vast.ai’s Secure Cloud Offering

Our [Trust Center](https://trust.vast.ai/) includes additional information about our controls, compliance stance, FAQs and Subprocessors.

For customers with heightened security and compliance requirements, our [**Secure Cloud**](https://cloud.vast.ai/?secure_cloud=true) offering provides access to GPUs hosted exclusively by our certified datacenter partners. This environment already delivers an added layer of assurance – now further backed by an independent SOC 2 audit on our platform infrastructure.

Achieving **SOC 2 Type I certification** is a clear validation of the controls we've built to protect customer data and ensure the integrity of our infrastructure.

Here's a brief overview of SOC 2 Type I and what it entails.

### What Is SOC 2 Type I?

Put simply, SOC 2 (System and Organization Controls 2\) compliance assesses an organization's security measures. The audit determines whether internal controls are appropriately designed to meet [trust services criteria](https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-2) – covering areas like security, confidentiality, privacy, and processing integrity – in order to safeguard customer data.

Developed by the American Institute of Certified Public Accountants (AICPA), the SOC 2 Type I report is essentially a detailed snapshot evaluating the design and implementation of these controls at a specific point in time. Its goal is to ensure that security measures meet rigorous industry standards.

### Why This Matters

Our SOC 2 Type I certification affirms to our customers and partners that we've put strong, well-designed controls in place to protect their data and maintain operational reliability. We take data security seriously and follow industry-recognized best practices.

The Type I audit is only the first step in a broader compliance roadmap. We are currently preparing for the SOC 2 Type II audit, which evaluates how effectively internal controls perform over an extended period. That assessment will validate that we maintain our security practices consistently, both in design and in day-to-day execution.

We're also advancing our compliance efforts across other standards, to help us better serve organizations with strict regulatory requirements.

### Our Ongoing Commitment

Earning this SOC 2 Type I certification reflects our commitment to protecting the privacy and integrity of the data entrusted to us by developers, researchers, and organizations worldwide. As a distributed peer-to-peer platform powering global compute, we take our responsibility seriously. This certification is one step in our ongoing effort to provide users with infrastructure they can trust. 

For more about security and compliance at Vast.ai as well as our datacenter partners, please see our previous post [here](https://vast.ai/article/security-and-compliance-at-vast-ai). To request a copy of our SOC 2 report, feel free to contact us anytime at [compliance@vast.ai](mailto:compliance@vast.ai).

A huge thank-you to our team at Vast.ai and the audit partners who helped us reach this milestone\! 

We'll keep raising the bar – and keep you updated as we do.



Vast.ai has achieved SOC 2 Type I certification, reinforcing its commitment to data security and trusted infrastructure. Learn what this means for users and what’s next on their compliance roadmap.

Vast.ai Achieves SOC 2 Type I Certification


Google has recently released Gemma 3, their latest open-source language model family. The model, designed to run efficiently on a single GPU, brings state-of-the-art capabilities including advanced multimodal understanding, multilingual support, and an impressive 128K token context window. In this post, we'll explore the key features of Gemma 3 and demonstrate how to deploy it on Vast.ai to leverage one of its new features, comparing images.

## What makes Gemma 3 Special

Gemma 3 is available in four model sizes: 1B, 4B, 12B, and 27B parameters. Each size comes in both instruction-tuned (IT) and pre-trained (PT) variants, allowing developers to choose the optimal model for their specific applications.

What makes Gemma 3 particularly accessible is its efficient architecture - all models are optimized to run on a single GPU, with the smaller variants capable of running on consumer-grade GPUs through Vast.ai's platform.

Gemma 3 offers multiple state-of-the-art capabilities:

### 1. Enhanced Context Window

-   Supports up to 128K tokens context window
-   Improved long-range understanding and coherence
-   Better handling of complex, multi-turn conversations

### 2. Multi-Modal Capabilities

-   Native image understanding and analysis (available in 4B, 12B, and 27B models only)
-   Support for both text and image inputs in the same prompt
-   Advanced ability to analyze images, text, and short videos

### 3. Improved Performance

-   State-of-the-art performance for its size
-   Enhanced reasoning capabilities
-   Better code generation and understanding

### 4. Multilingual Support

-   Out-of-the-box support for over 35 languages
-   Pretrained support for over 140 languages

### 5. Additional Features

-   Function calling support for automated workflows
-   Official quantized versions for faster performance
-   Optimized for various hardware platforms (NVIDIA GPUs, Google Cloud TPUs, AMD GPUs)

## Using Gemma 3 For Image Comparison On Vast

Gemma 3 supports multimodal inputs, including an impressive image comparison capability. In this guide, we'll demonstrate this feature using images from the Open Image Preferences dataset, available [here](https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1-binarized).

### Image Selection

For this post, we've selected two contrasting astronaut images from the Open Image Preferences dataset:

<div style={{ display: "flex", gap: "20px", marginBottom: "2rem" }}>
    <div style={{ flex: 1 }}>
        <img
            src="/uploads/image-1.jpg"
            alt="Astronaut 1"
            style={{ width: "100%" }}
        />
        <p style={{ textAlign: "center", fontWeight: "bold" }}>Astronaut 1</p>
    </div>
    <div style={{ flex: 1 }}>
        <img
            src="/uploads/image-2.jpg"
            alt="Astronaut 2"
            style={{ width: "100%" }}
        />
        <p style={{ textAlign: "center", fontWeight: "bold" }}>Astronaut 2</p>
    </div>
</div>

### Launch an Instance

First, let's set up a Vast.ai instance with the necessary configurations to run Gemma 3:

```bash
vastai create instance $INSTANCE_ID \
    --image vllm/vllm-openai:latest \
    --disk 40 \
    --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=hf-token' \
    --args --model google/gemma-3-4b-it --gpu-memory-utilization 0.9 --max-model-len 4096 --limit-mm-per-prompt "image=2"
```

### Call Gemma 3 Using the OpenAI SDK

Next, we'll write a script to call Gemma 3 on our instance. vLLM exposes an OpenAI-compatible API endpoint, allowing us to interact with Gemma 3 using the OpenAI SDK:

```python
from openai import OpenAI
import base64

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client
client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

# Load and convert local images to base64
def image_to_base64(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

# Convert images to base64
image1_base64 = image_to_base64("/path/to/image-1.jpg")
image2_base64 = image_to_base64("/path/to/image-2.jpg")

response = client.chat.completions.create(
    model="google/gemma-3-4b-it",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Compare these two images"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image1_base64}"
                    }
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image2_base64}"
                    }
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)
```

Here is our output from Gemma 3:

```
Okay, let's compare these two images of astronauts. Here's a breakdown of the key differences and similarities:

**Similarities:**

* **Subject:** Both images feature a spacesuit and the astronaut's helmet.
* **Style:**  Both are highly stylized and appear to be digitally rendered images, likely created with a focus on aesthetic appeal and a sense of wonder. They evoke a futuristic and somewhat surreal feeling.
* **Helmet:** The overall shape and design of the helmet are fairly consistent between the two – a classic, bulbous, and slightly vintage-looking design.



**Differences:**

1. **Background:** This is the most significant difference.
   * **Image 1:** Features a vibrant, complex, and detailed space scene with a blue and gold starry sky, nebulas, and a visible horizon line suggesting a planet or moon. The background is rich in color and texture.
   * **Image 2:** Shows a much simpler and muted background. It’s a dusky blue sky with scattered stars and a faint suggestion of a horizon with mountains. There's no nebula or dramatic planetary features.

2. **Suit Detail & Appearance:**
    * **Image 1:** The spacesuit is more elaborate and brightly colored. It has intricate gold detailing on the arms and chest and a much more modern and polished look. It also appears to be a little more reflective.
    * **Image 2:** The suit is weathered and looks older. There's a visible rust and decay on the metal parts, suggesting it's been used and exposed to the elements. It’s a more rugged and worn appearance.

3. **Pose and Composition:**
   * **Image 1:** The astronaut is standing and facing slightly towards the camera, but the composition is broader, showing more of the suit and the expansive space.
   * **Image 2:** The astronaut is positioned at a three-quarter angle, focusing primarily on the helmet and upper body. The composition is more intimate and tightly framed.

4. **Lighting & Tone:**
   * **Image 1:**  The lighting is dramatic and highlights the spacesuit with warm, glowing lights. It's a brighter and more vibrant image overall.
   * **Image 2:** The lighting is softer and cooler, creating a more subdued and melancholic atmosphere.

**In essence:**

* **Image 1** is a grand, visually stunning depiction of an astronaut experiencing the majesty of space.
* **Image 2** is a more intimate and evocative portrayal, suggesting a sense of isolation, exploration, and the passage of time.



Would you like me to elaborate on any specific aspect of the comparison, such as the artistic style or the possible symbolism of the images?
```

### Image Comparison Analysis

The model provides detailed, structured analysis of the images, demonstrating its sophisticated visual understanding capabilities. The analysis includes:

1. **Comprehensive Comparison**: The model systematically breaks down both similarities and differences between the images
2. **Structured Output**: Results are organized into clear categories (Similarities, Differences, In essence)
3. **Detailed Observations**: The model captures both obvious and subtle details, from overall composition to specific visual elements
4. **Contextual Understanding**: The analysis includes interpretation of artistic style, mood, and potential symbolism
5. **Interactive Engagement**: The model offers to elaborate on specific aspects of the comparison

This level of analysis showcases Gemma 3's advanced multimodal capabilities, particularly its ability to:

-   Process and compare multiple images simultaneously
-   Provide nuanced visual analysis
-   Generate structured, coherent responses
-   Maintain context throughout the analysis

## Next Steps

The image comparison capabilities we've demonstrated can be applied to several real-world use cases:

-   **Visual Quality Control**: Compare product images for manufacturing defects
-   **Design Evolution**: Analyze changes between design iterations
-   **Medical Imaging**: Compare diagnostic images for changes or anomalies

### Beyond Image Analysis: Additional Capabilities

While image comparison showcases Gemma 3's visual understanding, the model's versatility extends far beyond visual tasks. Here are several other powerful applications you can build:

-   **Multilingual Chatbots**: Create AI assistants that can communicate in 140+ languages
-   **Document Analysis**: Leverage the 128K token context window for processing extensive documents
-   **Educational Tools**: Build interactive learning systems with multilingual support
-   **Visual Search**: Combine text and image understanding for sophisticated search applications

These applications can be deployed using the same Vast.ai infrastructure we've explored, allowing for efficient development and scaling.

## Conclusion

Gemma 3 represents a significant advancement in open-source AI models, offering state-of-the-art performance while being designed to run efficiently on a single GPU. Vast.ai's infrastructure provides an ideal platform for deploying Gemma 3 in production environments, making it accessible for both development and scaling applications.


Google’s open-source Gemma 3 model brings multimodal reasoning, 128K context, and multilingual support—all runnable on a single GPU. Deploy it on Vast.ai to compare images or build advanced AI apps.

Running Google's Gemma 3 on Vast.ai


Analyzing lengthy documents like SEC filings has traditionally required complex chunking strategies and careful management of context. However, with Mistral AI's [Mistral-Small-3.1-24B-Instruct-2503](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503) model and its 128k token context window, we can now process entire documents at once. This post demonstrates how to leverage this capability using Vast.ai's cloud GPU platform.

The Mistral Small 3.1 model represents a significant advancement in open-source language models. With 24 billion parameters, it combines state-of-the-art vision capabilities with an extended context window that makes it particularly suited for document analysis tasks.

Vast.ai's marketplace provides GPU configurations needed for optimal performance at cost-effective prices.

In this post, we will:
1. Set up a Vast.ai instance with the right GPU specifications
2. Deploy the model using vLLM for optimal serving performance
3. Utilize the model's native chat completions API
4. Leverage the long context window to analyze full public company annual reports (10-K filings)

This setup demonstrates how Mistral Small 3.1 with extended context windows can transform document analysis workflows, making it possible to process and understand complex documents in their entirety.

## Deploy Mistral Small 3.1 on Vast

Let's start by getting our environment set up. First, we need to install the Vast AI client and configure our access.

### Install Vast

Head over to the [Vast.ai Account Page](https://cloud.vast.ai/account/) to grab your API key. Once you have it, we'll use it to set up the client:

```bash
#In an environment of your choice
pip install --upgrade vastai
```

With the client installed, we can configure our API key:

```bash
# Here we will set our api key
export VAST_API_KEY="VAST_API_KEY" #Your key here
vastai set api-key $VAST_API_KEY
```

### Choosing the Right Hardware

Now we are going to search for GPUs on Vast.ai to run the Mistral-Small-3.1-24B-Instruct-2503 model. This model requires specific hardware capabilities to run efficiently with vLLM's optimizations. Here are our requirements:

1. A minimum of 60GB GPU RAM to accommodate:
   - Mistral model weights (24B Parameters)
   - KV Cache for handling the 128k token context window

2. A single GPU configuration, as Mistral-Small-3.1-24B-Instruct-2503 can be efficiently served on one GPU:
   Note: Multi-GPU configurations are supported if higher throughput is needed.

3. A static IP address for:
   - Stable API endpoint hosting
   - Consistent client connections
   - Reliable Langchain integration

4. At least one direct port that we can forward for:
   - vLLM's OpenAI-compatible API server
   - External access to the model endpoint
   - Secure request routing

5. At least 80GB of disk space to hold the model and other dependencies

We'll use `vast ai search offers` to find the right instance.

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 60 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

### Deploying the Server via Vast

Select a machine from the output above and copy its ID into the `INSTANCE_ID` variable below.

We will use `vastai create instance` to create an instance that:
1. Uses `vllm/vllm-openai:latest` docker image. This gives us an OpenAI-compatible server.
2. Forwards port `8000` to the outside of the container, which is the default OpenAI server port.
3. Forwards `--model mistralai/Mistral-Small-3.1-24B-Instruct-2503` to serve the Mistral model.
4. Uses Mistral-specific parameters:
   - `--tokenizer_mode mistral` to use the Mistral tokenizer.
   - `--config_format mistral` for the correct model config.
   - `--load_format mistral` to properly load the model weights.
5. Uses `--gpu-memory-utilization 0.95` to maximize context window.
6. Ensures that we have 80 GB of disk space.

Note: Ensure that you fill in your huggingface token `HUGGING_FACE_HUB_TOKEN` to access the model. You'll need to accept the terms on the model's huggingface page https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503.

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGING_FACE_HUB_TOKEN' --disk 80 --args --model mistralai/Mistral-Small-3.1-24B-Instruct-2503   --tokenizer_mode mistral   --config_format mistral   --load_format mistral   --gpu-memory-utilization 0.95
```

### Verify Setup and Get Instance IP Address and Port

Now, we need to verify that our setup is working. We first need to wait for our machine to download the image and the model and start serving. This will take a few minutes. The logs will show you when it's done. 

Next, go to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console and find the instance you just created.

At the top of the instance, there is a button with an IP address in it. Click this and a panel will show up of the IP address and the forwarded ports. 
You should see something like: 
```
Open Ports
XX.XX.XXX.XX:YYYY -> 8000/tcp
``` 
Copy and paste the IP address (XX.XX.XXX.XX) into `VAST_IP_ADDRESS` and the port (YYYY) into `VAST_PORT` as inputs to the curl command below.

This curl command sends an OpenAI compatible request to your `vLLM` server. You should see the response if everything is set up correctly. 

Note: It may take a few minutes for the OpenAI server to initialize.

```bash
export VAST_IP_ADDRESS="VAST_IP_ADDRESS"
export VAST_PORT="VAST_PORT"
curl -X POST http://$VAST_IP_ADDRESS:$VAST_PORT/v1/completions -H "Content-Type: application/json"  -d '{"model" : "mistralai/Mistral-Small-3.1-24B-Instruct-2503", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

## Set Up Our Model

### Setup Dependencies

First, we'll install the OpenAI SDK to communicate with our model on the Vast server.

```bash
pip install --upgrade openai
```

### Set up Our Model

Next, let's configure our model using the IP address and port from the Vast.ai Console. We'll create a simple function to interact with the model, using Mistral's recommended temperature of 0.15.

```python
from openai import OpenAI

# Your Vast.ai instance details
VAST_IP_ADDRESS = "VAST_IP_ADDRESS"
VAST_PORT = "VAST_PORT"

# Initialize the client with your server URL
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an actual API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

# Simple function to chat with the model
def chat_with_model(prompt):
    response = client.chat.completions.create(
        model="mistralai/Mistral-Small-3.1-24B-Instruct-2503",
        messages=[
            {"role": "user", "content": prompt}
        ],
        temperature=0.15
    )
    return response.choices[0].message.content
```

#### Test our Model

Before we go on to long document analysis, let's test our function to make sure everything is working correctly.

```python
response = chat_with_model("what is the capital of France?")
print(response)
```

## Analyze Macy's Annual Reports to Track Store Numbers

The retail landscape has undergone significant changes, with major department stores closing across the US. Using our model's long context window, we'll analyze Macy's annual reports (10-K filings) to track their store count over the years.

### Download Filings

First, we need to download our data. We've gathered a list of Macy's annual report URLs from the SEC's EDGAR database (https://www.sec.gov/edgar/search/). Let's create a `macys_filings` list to store these URLs:

```python
import requests
from bs4 import BeautifulSoup
import re

# List of Macy's 10-K filing URLs
macys_filings = [
    "https://www.sec.gov/Archives/edgar/data/794367/000162828024012734/m-20240203.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000162828023009154/m-20230128.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000156459022011726/m-10k_20220129.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000156459021016119/m-10k_20210130.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436720000040/m-0201202010xk.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436719000038/m-0202201910xk.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436718000036/m-0203201810k.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436717000041/m-0128201710k.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436716000221/m-0130201610k.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436715000073/m-0131201510k.htm"
]
```

Next, we'll create a function that downloads and cleans the text from each filing, removing HTML formatting and other unnecessary elements:

```python
# Simple SEC headers
headers = {
    "User-Agent": "Your Name (your.email@example.com)",
    "Accept-Encoding": "gzip, deflate",
    "Host": "www.sec.gov"
}

def download_filing_text(url):
    """Download a filing and extract its text."""
    try:
        # Download the filing
        response = requests.get(url, headers=headers, timeout=30)
        response.raise_for_status()
        
        # Parse the HTML and extract text
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # Remove script and style elements
        for script in soup(["script", "style"]):
            script.decompose()
        
        # Get all text with minimal processing
        text = soup.get_text(separator=' ')
        
        # Basic cleaning - collapse whitespace
        text = re.sub(r'\s+', ' ', text).strip()
        
        return text
        
    except Exception as e:
        print(f"Error downloading filing: {str(e)}")
        return ""
```

Now that we have our download function ready, let's use it to process all ten years of Macy's reports. We'll loop through each URL, download the content, and store the cleaned text:

```python
# Process all filings
filings_text = []

for i, url in enumerate(macys_filings):
    
    # Download and process filing
    print(f"Downloading {url} filing...")
    text = download_filing_text(url)
    
    if text:  # If we got text back (not empty string)
        print(f"✓ Successfully processed filing ({len(text)} characters)")
        filings_text.append(text)
    else:
        print(f"✗ Failed to process filing")

# Print summary
print("\n=== PROCESSING SUMMARY ===")
print(f"Successfully processed: {len(filings_text)}/{len(macys_filings)} filings")
```

### Extract Store Counts from Filings

Now comes the exciting part - analyzing the filings to track Macy's retail footprint over time. We'll use our model's long context window to examine each document in its entirety, asking a simple but powerful question: "How many stores did Macy's operate this fiscal year?" This approach demonstrates the advantage of processing complete documents at once - we don't need to worry about missing context or searching through different sections manually.

For each filing, we'll store the model's response in an `all_responses` list, building a comprehensive picture of Macy's store count evolution:

```python
all_responses = []
for i, text in enumerate(filings_text):
    print(f"\nProcessing file {i+1}/{len(filings_text)}")
    
    try:
        prompt = f"How many stores did Macy's operate this fiscal year? DATA: {text}"
        response = chat_with_model(prompt)
        all_responses.append(f"File {i+1} analysis: {response}")
        print(f"✓ File {i+1} processed successfully")

        print(response)
        
    except Exception as e:
        print(f"✗ Error processing file {i+1}: {str(e)}")
        all_responses.append(f"File {i+1} error: {str(e)}")
        
# Print all responses
print("\n=== ANALYSIS OF ALL FILES ===")
for response in all_responses:
    print(f"\n{response}\n")
```

The model's responses show detailed information extraction - accurately identifying store counts while providing precise dates (e.g., "February 3, 2024"), section references (e.g., "Item 1. Business. General"), and geographical context (e.g., "43 states, the District of Columbia, Puerto Rico and Guam") from each filing:

```
=== ANALYSIS OF ALL FILES ===

File 1 analysis: As of February 3, 2024, Macy's operated 718 store locations.


File 2 analysis: As of January 28, 2023, Macy's operated 722 store locations. This information is explicitly stated in the document under the section "Item 1. Business. General" where it mentions:

"As of January 28, 2023, the Company operated 722 store locations in 43 states, the District of Columbia, Puerto Rico and Guam."


File 3 analysis: As of January 29, 2022, Macy's operated 725 store locations. This information is explicitly stated in the "Item 2. Properties" section of the 10-K filing:

"As of January 29, 2022, the operations of the Company included 725 store locations in 43 states, the District of Columbia, Puerto Rico and Guam, comprising a total of approximately 112 million square feet."


File 4 analysis: Macy's, Inc. operated 727 store locations in 43 states, the District of Columbia, Puerto Rico, and Guam as of January 30, 2021.


File 5 analysis: As of February 1, 2020, Macy's operated 775 store locations in 43 states, the District of Columbia, Puerto Rico, and Guam.


File 6 analysis: As of February 2, 2019, Macy's operated a total of **867 stores**. This information is detailed in the 10-K filing under the section discussing the company's properties and store count activity.


File 7 analysis: As of February 3, 2018, Macy's operated approximately **852 stores** in 44 states, the District of Columbia, Puerto Rico, and Guam.


File 8 analysis: As of January 28, 2017, Macy's operated a total of **829 stores**. This number includes stores across various brands such as Macy's, Bloomingdale's, and Bluemercury, located in 45 states, the District of Columbia, Guam, and Puerto Rico.


File 9 analysis: As of January 30, 2016, Macy's operated a total of **870 stores**. This included Macy's, Macy's Backstage, Bloomingdale's, Bloomingdale's Outlet, and Bluemercury stores across 45 states, the District of Columbia, Guam, and Puerto Rico.


File 10 analysis: As of January 31, 2015, Macy's operated a total of 823 stores.
```

### Summarize Store Count Findings

Now that we have the store counts for each year, we will have Mistral summarize the results for us.

```python
prompt = "Give me a list of macys store numbers from 2015-2024 DATA: " + " ".join(all_responses)
response = chat_with_model(prompt)
```

```python
print(response)
```

We now have a concise summary of the number of stores that Macy's operated in each year.

```
Based on the provided data, here is the list of Macy's store numbers from 2015 to 2024:

- **2015:** 823 stores
- **2016:** 870 stores
- **2017:** 829 stores
- **2018:** 852 stores
- **2019:** 867 stores
- **2020:** 775 stores
- **2021:** 727 stores
- **2022:** 725 stores
- **2023:** 722 stores
- **2024:** 718 stores

This list reflects the number of store locations Macy's operated as of the specified dates each year.
```

## Conclusion

In this post, we demonstrated how Mistral Small 3.1's 128k token context window transforms SEC filing analysis. We:

- Deployed Mistral's 24B parameter model on a cost-effective Vast.ai GPU instance
- Processed complete 10-K filings without splitting them into smaller chunks
- Extracted store count data from various sections of Macy's annual reports
- Maintained the full document context throughout the analysis process

With this foundation in place, we can tackle more sophisticated document analysis tasks. While tracking store counts offers a simple demonstration, the real value lies in analyzing complex relationships across entire documents. Financial analysts can:

- Extract specific metrics while understanding their broader context
- Connect information from management discussions with financial data
- Assess risk factors in relation to company performance
- Identify patterns across different sections of a single document

This capability extends beyond financial analysis. The same approach works for legal contracts, academic papers, technical documentation, and any other domain where processing long documents as complete units provides better insights.

By combining Mistral's context window capabilities with Vast.ai's accessible GPU infrastructure, comprehensive document analysis becomes more practical and accessible for a wider range of users and use cases.


Deploy Mistral Small 3.1 on Vast.ai to analyze SEC filings with a 128k token context window, extracting insights like Macy’s store count trends from 2015-2024.

SEC Filing Analysis Using Mistral Small 3.1's Long Context Window


DeepSeek may be getting a [lot of hype](https://vast.ai/article/deepseek-r1-open-source-disruptor-or-overhyped-upstart) recently, but there's a new rival on the scene. Alibaba Cloud's Qwen series of AI models is setting a new standard for intelligence in the open-source world.

Earlier this year, Alibaba Cloud launched **Qwen 2.5-Max** – and it reportedly [outperforms](https://www.reuters.com/technology/artificial-intelligence/alibaba-releases-ai-model-it-claims-surpasses-deepseek-v3-2025-01-29/) other foundation models like DeepSeek-V3, GPT-4o, and Llama-3.1-405B "almost across the board," according to the company. Not only that, but it also [powered](https://www.scmp.com/tech/big-tech/article/3298233/alibabas-qwen-powers-top-10-open-source-models-china-ai-know-how-goes-beyond-deepseek) the top 10 open-source LLMs on Hugging Face's rankings last month. All of the top-ranked models on the Open LLM Leaderboard were trained and developed on the updated open-source versions of Qwen.

So what is Qwen, and what can you do with it? Let's take a closer look\!

### Qwen 2.5: Advancements and Features

The Qwen model series includes Qwen (an LLM), Qwen-VL (a vision-language model), Qwen-Audio, Qwen-Coder, and Qwen-Math. The latest Qwen 2.5 models have been pre-trained on a large-scale dataset of up to 18 trillion tokens, resulting in a substantial knowledge base as well as much-improved capabilities in coding and math.

In terms of performance metrics, Qwen 2.5 is quite impressive. It [achieved](https://www.alibabacloud.com/en/solutions/generative-ai/qwen) a score of 85+ on the Massive Multitask Language Understanding (MMLU) benchmark, a HumanEval score of 85+ in coding, and a MATH benchmark score above 80\.

Beyond these core competencies, Qwen 2.5 has highly advanced abilities in following instructions and understanding and generating structured data. It can handle diverse system prompts, demonstrating an adaptability that's well suited for role-based interactions and condition-setting for chatbots.

The models can even interact with software on PCs and mobile devices. One user posted a video online [showing](https://techcrunch.com/2025/01/27/alibabas-qwen-team-releases-ai-models-that-can-control-pcs-and-phones/) Qwen 2.5-VL opening the Booking.com app for Android and booking a flight from Chongqing to Beijing\!

### Specialized Models in the Qwen Family

Each open-source model in the Qwen series is designed for a specific domain. The following are the main options in the Qwen 2.5 lineup:

-   **Qwen 2.5-VL** – This large vision-language model processes images, text, and bounding boxes to recognize and analyze content. It can read text in Chinese and English, compare visuals, create stories, solve math problems, and answer questions.

-   **Qwen 2.5-Audio** – Designed to process audio and text, this audio language model accepts a variety of audio formats (including speech, music, and natural sounds) and generates text responses.

-   **Qwen 2.5-Max** – Pre-trained on over 20 trillion tokens and post-trained with curated Supervised Fine-Tuning (SFT), this high-performing Mixture-of-Expert (MoE) model has surpassed DeepSeek V3, Llama 3.1, and others in benchmarks like Arena-Hard, LiveBench, LiveCodeBench, and GPQA-Diamond.

-   **Qwen 2.5-Coder** – This coding model supports up to 128K tokens of context, covers 92 programming languages, and delivers competitive performance against larger language models in code generation, multi-language coding, completion, and repair.

-   **Qwen 2.5-Math** – Pre-trained and fine-tuned on synthesized data, this mathematical LLM supports English and Chinese queries and excels in Chain-of-Thought (CoT), Program-of-Thought (PoT), and Tool-Integrated Reasoning (TIR) while outperforming most 70B math models.

Furthermore, Qwen APIs enable users to develop generative AI applications for tasks like writing, image generation, and audio analysis, boosting efficiency and transforming the customer experience for businesses.

### Powering Qwen with Vast.ai

Given how competitive the Qwen models are with larger, more established AI systems – and the fact that they're open source – they offer an exciting opportunity for AI researchers, developers, and enthusiasts to experiment and innovate at minimal cost.

However, deploying the models on high-performance hardware is crucial for optimal performance. That's where Vast.ai comes in. Our platform makes this possible with **on-demand [RTX 5090 GPU](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know) rental**, for instance, providing the power needed to run Qwen models efficiently without the overhead of expensive infrastructure.

With many other GPU options in our marketplace, you can choose the right balance of performance and cost for your specific needs. Get started with Vast today and experience the full potential of Qwen with high-performance GPUs at your fingertips\!


Alibaba Cloud’s Qwen 2.5 AI models outperform rivals like DeepSeek and GPT-4o, excelling in coding, math, and vision. Available open-source, powered by Vast.ai GPUs.

Alibaba's Qwen: An Open-Source AI Model that Surpasses DeepSeek?


Mukesh Ambani's Reliance Group is [making waves](https://www.bloomberg.com/news/articles/2025-01-23/billionaire-mukesh-ambani-plans-world-s-biggest-data-center-in-india-s-gujarat) with its plans to build what may become **the** **world's largest data center by capacity** in Jamnagar, India. The facility, expected to reach three gigawatts, would dwarf today's largest data centers, which are under one gigawatt each (or will just barely exceed one gigawatt once fully operational, as in the case of the Sines data center in Europe).

Reliance isn't alone in its bid to drastically expand AI infrastructure – companies around the world are racing to meet surging demand in this sector. Tech giants like Microsoft, Alphabet, and Amazon are investing billions of dollars into data centers to expand their AI capabilities.

The Stargate Project is another entity endeavoring to boost AI infrastructure in the United States. OpenAI, SoftBank, and Oracle all recently [pledged](https://techcrunch.com/2025/01/21/openai-teams-up-with-softbank-and-oracle-on-50b-data-center-project/) to invest $100 billion to $500 billion as part of the Stargate effort.

So what makes Reliance interesting? The company has some big ambitions...

### Reliance's AI Strategy

Ambani, the 67-year-old billionaire chairman and managing director of Reliance Group, hopes to offer AI inference to businesses and developers at significantly lower costs than competitors. In order to achieve this goal, Reliance is investing heavily in NVIDIA's AI semiconductors.

Undercutting prices isn't a new approach for Ambani; his aggressive pricing in other industries, like telecommunications, disrupted the market and forced competitors to adapt. It's not unreasonable to think that Reliance could do the same in the AI industry.

"By leveraging our expertise in infrastructure, networking, operations, software, and data and by collaborating with our global partners, our goal is to create the world's lowest AI inferencing cost, right here in India," Ambani [said](https://www.bloomberg.com/news/articles/2025-01-23/billionaire-mukesh-ambani-plans-world-s-biggest-data-center-in-india-s-gujarat) last year. "This will make AI applications in India more affordable than anywhere else, making AI accessible to all."

### Strengthening India's AI Infrastructure

One of Reliance's main objectives with this massive data center is to reduce India's reliance on overseas AI infrastructure. As AI adoption accelerates across the country, driven by a booming startup ecosystem and increasing enterprise demand, the need for localized AI compute power is greater than ever.

Currently, India's total data center capacity is under one gigawatt. If completed as planned, the Jamnagar facility alone could triple that amount by 2027\. Reliance also reportedly aims to power the new data center as much as possible with renewable energy – aligning with global sustainability trends and potentially positioning India as a more self-sufficient AI hub.

Beyond meeting domestic demand, this expansion raises questions about India's role in the global AI race. With governments and tech giants worldwide investing heavily in AI infrastructure, Reliance's project reflects India's potential to compete on the world stage as a provider of AI compute power.

### What's Next for AI Infrastructure?

Reliance's plan for the facility in Jamnagar is just one of many data center projects in the works today. According to McKinsey & Co, global data center capacity could [more than triple](https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/ai-power-expanding-data-center-capacity-to-meet-growing-demand) by 2030, reaching an estimated 219 gigawatts annually. Yet even with heavy investment, a significant shortfall will likely remain. **The U.S. alone is projected to face a 15-gigawatt gap by 2030**, requiring twice the data center capacity that's been built since 2000\.

If Reliance successfully builds its three-gigawatt facility, it could help address some of this global demand. However, ensuring that infrastructure keeps pace with AI's rapid advancement will remain a challenge worldwide.

In the meantime, access to high-performance compute remains a priority for developers, researchers, and businesses. While tech giants build massive facilities, **cloud GPU rental is an option that provides immediate, scalable access to AI compute** without the need for expensive hardware or infrastructure investments.

If you're looking for flexible and cost-effective AI compute, Vast.ai offers on-demand cloud GPU rental. We invite you to explore our platform and get the AI compute you need – and start scaling your workloads today\!


Reliance plans to build the world’s largest data center in India, aiming to lower AI costs and boost local infrastructure with NVIDIA chips and renewable energy.

India's AI Leap: World's Biggest Data Center on the Horizon?


A new NVIDIA Blackwell GPU with 96GB of VRAM? If rumors are to be believed, that could be exactly what's coming. 

A recent shipping manifest [revealed](https://wccftech.com/nvidia-rtx-pro-6000-x-blackwell-leak-96-gb-gddr7-600w/) an entry for the "**RTX PRO 6000 X Blackwell**," evidently being sent to India "for testing purposes." At least two different shippers have sent the GPUs for testing.

This isn't your typical consumer-grade gaming card, however. The as-yet-unannounced Blackwell-based GPU appears to be aimed at AI and data-intensive workloads. Here's what we know so far about this new entry in NVIDIA's professional lineup.

### NVIDIA RTX Pro 6000 X: What We Know So Far

With the impressive [RTX 5090](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know) expected to remain at the top of the GeForce 50-series lineup, the professional segment is where we'll likely see ultra-high-end specs that push the limits on VRAM – and this latest shipping log discovery suggests that will be the case. 

According to reports, as mentioned, the RTX Pro 6000 X features a staggering **96GB of GDDR7 memory** on a 512-bit memory bus. That's significantly higher than even the most powerful gaming cards. For example, the RTX 5090 has 32GB of GDDR7 memory – which is more than enough for most purposes – but higher VRAM is obviously beneficial for workloads that require handling massive datasets, such as foundation AI model training and large-scale simulations.

The RTX Pro 6000 X also features a **GB202 die** and **600 watts** of total board power (TBP). This aligns with previous high-end Blackwell architecture designs, like the RTX 5090, which also has the GB202 die and comes in at 575 watts of TBP.

Interestingly, the discovery of the RTX Pro 6000 X on the shipping manifest follows an [earlier sighting](https://wccftech.com/nvidia-rtx-4090-with-96gb-vram-reportedly-exists/) of a 96GB RTX 4090 GPU – perhaps an engineering sample. While NVIDIA hasn't made any announcements, these repeated sightings nonetheless suggest a broader push for high-VRAM GPUs.

### Comparison to Existing RTX Workstation GPUs

We've seen professional-grade GPUs with massive VRAM configurations before, such as the RTX 6000 Ada with 48GB of memory, as well as the A100/A800 series. However, the RTX Pro 6000 X's 96GB of GDDR7 memory would make it one of the highest-capacity workstation GPUs to date. 

Assuming we see the RTX Pro 6000 X emerge, it would set a new benchmark for professional GPUs, doubling the memory of the RTX 6000 Ada while offering the faster speeds and efficiency of GDDR7 memory.

Plus, unlike the A100 and A800 series, which are designed primarily for data centers and enterprise AI workloads, the RTX Pro 6000 X could serve as a more flexible solution. Its ultra-high VRAM would allow professionals to work with larger datasets without necessarily relying on multi-GPU setups. 

### Looking Ahead

NVIDIA hasn't officially announced the RTX Pro 6000 X, so all information should be taken with a grain of salt, but the repeated mentions of this new GPU in shipping logs suggest that an announcement may be imminent. 

For Vast.ai users, this could mean more powerful GPU rental options in the near future. Stay tuned for more developments\! In the meantime, check out our current selection of high-performance GPUs available on Vast.ai for flexible and cost-effective access to cutting-edge hardware today.  

NVIDIA’s rumored RTX Pro 6000 X packs 96GB of GDDR7 VRAM, a GB202 die, and 600W TBP—ideal for AI, data science, and high-end workstation workloads.

NVIDIA RTX Pro 6000 X Rumors: Blackwell GPU with 96GB VRAM Spotted in Shipping Log

Pyannote Audio and NVIDIA’s Sortformer enhance Whisper’s speech recognition by adding speaker diarization. We’ll run them on VAST.ai to create speaker-attributed transcripts.

Implementing Speech-to-Text with Speaker Diarization: Comparing Pyannote and Sortformer on VAST.ai

NVIDIA’s RTX 5090 delivers cutting-edge AI and gaming performance, but launch issues, high costs, and stock shortages make it hard to get. Try it now on Vast.ai!

NVIDIA GeForce RTX 5090 Specs: Everything You Need to Know


February may be a short month, but we've made the most of it with plenty of updates!

Vast.ai is now offering NVIDIA RTX 5000-series GPUs for rent, including the powerful **RTX 5090**, bringing even more high-performance options to our platform. Plus, we've partnered with **SkyPilot** in an integration that allows you to rent Vast machines through SkyPilot's infrastructure and price optimization framework for even greater flexibility.

Of course, keeping everything running smoothly means staying on top of any technical hiccups. As always, we've been hard at work refining our platform and fixing bugs. Here are some of the latest bug fixes we've rolled out since our last update:

-   Fixed instance-to-instance copy issue.
-   Made various virtual machine (VM) enhancements.
-   Corrected an issue where machine is erroneously marked as "unavailable."
-   Fixed earnings display issue and template category display issue.

### NVIDIA Cloud GPU Updates

At Vast.ai, every update is aimed at making your experience smoother, faster, and more efficient. From expanding our GPU offerings to adding templates and updating guides, our latest improvements are designed to give you even more control and streamline your workflow.

#### Highlights of the latest enhancements:

-   NVIDIA RTX 5000-series GPUs are now available, including the [RTX 5090](https://cloud.vast.ai/?gpu_option=RTX+5090).
-   SkyPilot Integration: Rent Vast machines through SkyPilot’s infrastructure and price optimization framework.
-   Added [vLLM Template](https://cloud.vast.ai/?ref_id=62897&template_id=f4c3ccaf87393c5266748b3b42ba8567) and [Guide](https://docs.vast.ai/vllm-llm-inference-and-serving): Fast and easy-to-use library for LLM inference and serving.
-   Added [Unreal Pixel Streaming Template](https://cloud.vast.ai/?ref_id=62897&template_id=c1c84fd5a4d945fc16012e7ca02d1423): Stream high-quality, interactive 3D content directly to web browsers without requiring end users to install anything.
-   Added [Ollama + WebUI Guide](https://docs.vast.ai/ollama-webui) for running DeepSeek R1.
-   Added [Guide for Creating a Custom Template](https://docs.vast.ai/creating-a-custom-template) from Vast Base Image.
-   Updated Hosting Guide.

Please check out the full change log for our February 2025 updates at the bottom of this page.

### Our Commitment

Whether you're looking to try out the RTX 5090, take advantage of SkyPilot integration, explore our new and updated templates and guides, or just enjoy the many features we continue to refine and expand, we're here to help you get the most out of Vast.ai! We're always happy to support both new and longtime users with affordable, on-demand GPU and virtual machine rental.

Our easy-to-use platform is backed by a responsive customer support team and an active Discord community. Need assistance? Email us at [support@vast.ai](mailto:support@vast.ai) or join our Discord server for helpful resources today!

### Change Log

#### Features:

-   NVIDIA RTX 5000 Series GPUs: Rent Blackwell chips on Vast, including the [RTX 5090](https://cloud.vast.ai/?gpu_option=RTX+5090).
-   SkyPilot Integration: Rent Vast machines through SkyPilot’s infrastructure and price optimization framework.

#### Templates & Guides:

-   Added [vLLM Template](https://cloud.vast.ai/?ref_id=62897&template_id=f4c3ccaf87393c5266748b3b42ba8567) and [Guide](https://docs.vast.ai/vllm-llm-inference-and-serving): Fast and easy-to-use library for LLM inference and serving.
-   Added [Unreal Pixel Streaming Template](https://cloud.vast.ai/?ref_id=62897&template_id=c1c84fd5a4d945fc16012e7ca02d1423): Stream high-quality, interactive 3D content directly to web browsers without requiring end users to install anything.
-   Added [Ollama + WebUI Guide](https://docs.vast.ai/ollama-webui) for running DeepSeek R1.
-   Added [Guide for Creating a Custom Template](https://docs.vast.ai/creating-a-custom-template) from Vast Base Image.
-   Updated VM Desktop Template.
-   Updated Open WebUI Template: Now includes pre-built llama.cpp.
-   Updated ComfyUI & ComfyUI + FLUX.1 Templates.
-   Updated SD Swarm & SD Forge Templates.
-   Updated Fooocus Template.
-   Updated Ubuntu 22.04 VM Template: Updated drivers to CUDA 12.8, preinstalled Docker container toolkit, added Tailscale support.
-   Updated Kohya’s GUI.
-   Updated Oobabooga Template.
-   Updated Hosting Guide.

#### Bug Fixes:

-   Various virtual machine (VM) enhancements.
-   Instance-to-instance copy issue.
-   Machine marked as "unavailable" issue.
-   Earnings display issue.
-   Template category display issue.
-   Session key issue.

#### Miscellaneous Tasks:

-   Added API keys section into CLI page for CLI login command.
-   Added length validation to API keys.  


Rent NVIDIA RTX 5090 GPUs on Vast.ai! Now integrated with SkyPilot for better pricing. New templates, guides & fixes to improve your experience.

February 2025 Product Update


## Introduction

In the world of audio processing and speech recognition, identifying when someone is speaking versus when there's silence or background noise is a critical first step. This process, known as Voice Activity Detection (VAD), serves as the foundation for many speech-related applications, from transcription services to voice assistants. While conceptually simple, implementing an efficient and accurate VAD system can significantly improve downstream tasks and reduce computational costs.

[PyAnnote Audio](https://github.com/pyannote/pyannote-audio), an open-source toolkit built on PyTorch, offers state-of-the-art models for VAD that are both accurate and accessible. Running these models on Vast.ai provides a cost-effective solution for processing large audio datasets without investing in expensive hardware. This combination gives developers and researchers the tools they need to build sophisticated audio processing pipelines at a fraction of the cost of traditional cloud providers.

### Why Voice Activity Detection Matters

VAD provides several key benefits for speech processing pipelines:

1. **Reduced Computation Load**: By filtering out non-speech segments before running speech-to-text (STT) models, we significantly reduce the computational resources needed for transcription.

2. **Improved Accuracy**: Many STT models perform better when processing only speech segments rather than trying to interpret silence or background noise.

3. **Efficient Storage**: Extracting only the speech segments can reduce storage requirements for large audio datasets.

4. **Better User Experience**: For applications like voice assistants or transcription services, VAD helps eliminate unnecessary processing of silence.

### What This Guide Covers

In this guide, we will:

-   Set up the Pyannote Audio VAD pipeline
-   Process audio files to detect speech segments
-   Extract and save only the speech portions of the input audio
-   Visualize the results

The output will be a collection of audio files containing only the detected speech segments from the original recording, making them ready for further processing in speech-to-text pipelines.

### Why VAST.ai

VAST.ai offers a marketplace approach to GPU rentals that provides significant advantages for audio processing tasks. Unlike traditional cloud providers, VAST.ai allows you to:

-   Rent precisely the GPU capacity needed for your workload
-   Access GPU's at more affordable rates
-   Access more types of SKU's of GPU's, particularly lower RAM/more affordable GPUs
-   Avoid long-term commitments for experimental projects

For VAD specifically, VAST.ai offers an ideal balance of performance and cost-effectiveness, as these models benefit from GPU acceleration without requiring the most expensive hardware tiers.

## Download This Notebook

To follow along with this tutorial, you can download the complete Jupyter notebook:

[Notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/6q8akx/6b8dc846a00d27decad5a60a6159466813f6d445/files/pyannote_vad_vast.ipynb)

Having the notebook will allow you to execute the code blocks as you read through this guide, making it easier to understand and experiment with the VAD implementation.

## Choosing an Instance

For running the Pyannote Voice Activity Detection model on VAST.ai, you'll need a relatively modest GPU setup since VAD models are computationally efficient compared to larger AI tasks. Here are the recommended specifications:

-   GPU: A mid-range GPU like an RTX 3060 or 4060 would be sufficient.
-   VRAM: 6-8GB of VRAM should be adequate as the Pyannote VAD model is relatively small.
-   RAM: 8-16GB system RAM is recommended for processing audio files.
-   Storage: At least 10GB for the model, dependencies, and your audio files.
-   CUDA: Make sure the instance has CUDA installed (version 11.0+ recommended).
-   Python: Python 3.8+ with PyTorch installed.

## Selecting an Instance

Follow these steps to set up your environment:

1. Ensure that you have a Vast.ai account
2. Go to the Vast Templates in the Console https://cloud.vast.ai/templates/
3. Select the `PyTorch (CuDNN Runtime)` Template
4. Filter for an instance with:
    - 1 GPU
    - 6-8GB of VRAM
    - 8-16GB system RAM
    - 10GB of storage
5. Select an instance and click rent
6. Install the Vast TLS certificate in your browser to access the notebook server https://docs.vast.ai/instances/jupyter#1SmCz
7. Go to your Instances https://cloud.vast.ai/instances/ and click "Open" to access the jupyter server on your instance.
8. Upload the notebook to the server or create a new notebook

## Installing Dependencies

Let's start by installing the necessary Python packages:

```python
%%bash
pip install pyannote.audio
pip install pydub
pip install librosa
pip install yt-dlp
```

We also need to install FFmpeg for audio processing:

```python
%%bash
apt-get update && apt-get install -y ffmpeg
```

## Setting Up Your Hugging Face Token

Pyannote models are hosted on Hugging Face, so you'll need to set up authentication:

```python
# Make sure you've accepted the user conditions at:
# https://hf.co/pyannote/voice-activity-detection
# https://hf.co/pyannote/segmentation

HF_TOKEN = ""  # Add your token here
```

Ensure that you have accepted the terms for the models at the URLs above. The models are free to use, but you must agree to their terms of service.

## Generating Test Data

For this tutorial, we'll download a sample audio file from Vast.ai's YouTube channel. You can also use your own audio file if you prefer:

```bash
yt-dlp -f "bestaudio" --extract-audio --audio-format wav -o "test.wav" https://www.youtube.com/watch?v=542xENIxKFU
```

## Voice Activity Detection

First, let's set up the VAD pipeline:

```python
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
        "pyannote/voice-activity-detection",
        use_auth_token=HF_TOKEN
    )

# Move pipeline to appropriate device
pipeline = pipeline.to(device)
```

Next, we'll process our audio file to identify speech segments:

```python
# Process the audio file
audio_file = "test.wav"
output = pipeline(audio_file)

print(f"Processing {audio_file} on {device}")
print("Voice activity segments:")

# Get all speech segments
speech_segments = list(output.get_timeline().support())

for i, speech in enumerate(speech_segments):
    # active speech between speech.start and speech.end
    print(f"Segment {i+1}: Speech from {speech.start:.2f}s to {speech.end:.2f}s (duration: {speech.duration:.2f}s)")
```

The output will look like this when using the default audio file:

```
Processing test.wav on cuda
Voice activity segments:
Segment 1: Speech from 6.78s to 51.62s (duration: 44.84s)
Segment 2: Speech from 53.56s to 54.27s (duration: 0.71s)
Segment 3: Speech from 55.55s to 84.76s (duration: 29.21s)
Segment 4: Speech from 86.53s to 89.03s (duration: 2.50s)
...
```

## Saving Speech Segments

Now we'll create a function to extract the identified speech segments from our audio file:

```python
import os
import shutil
from pydub import AudioSegment


def split_audio_by_segments(audio_path, segments, output_dir="output_segments"):
    """
    Split an audio file into multiple files based on speech segments

    Parameters:
    -----------
    audio_path: str
        Path to the input audio file
    segments: list
        List of speech segments (with start and end attributes)
    output_dir: str
        Directory to save the output segments
    """
    # Clear the output directory if it exists
    if os.path.exists(output_dir):
        shutil.rmtree(output_dir)

    # Create output directory
    os.makedirs(output_dir, exist_ok=True)

    # Load the audio file
    audio = AudioSegment.from_file(audio_path)

    # Extract each segment
    for i, segment in enumerate(segments):
        # Convert seconds to milliseconds
        start_ms = int(segment.start * 1000)
        end_ms = int(segment.end * 1000)

        # Extract segment
        segment_audio = audio[start_ms:end_ms]

        # Generate output filename
        filename = os.path.basename(audio_path)
        name, ext = os.path.splitext(filename)
        output_path = os.path.join(output_dir, f"{name}_segment_{i+1:04d}_{start_ms:08d}ms-{end_ms:08d}ms{ext}")

        # Export segment
        segment_audio.export(output_path, format=ext.replace('.', ''))
        print(f"Saved segment {i+1} to {output_path}")
```

Let's apply this function to extract our speech segments:

```python
split_audio_by_segments(audio_file, speech_segments)
```

This will be the output when using the default audio file:

```
Saved segment 1 to output_segments/test_segment_0001_00006780ms-00051617ms.wav
Saved segment 2 to output_segments/test_segment_0002_00053558ms-00054267ms.wav
Saved segment 3 to output_segments/test_segment_0003_00055549ms-00084760ms.wav
Saved segment 4 to output_segments/test_segment_0004_00086532ms-00089029ms.wav
...
```

## Inspecting Results

To verify our results, we'll create a function to play audio files in our Jupyter environment:

```python
import librosa
from IPython.display import Audio, display

def play_audio(file_path, sr=None):
    """
    Play an audio file in a Jupyter notebook.

    Parameters:
    -----------
    file_path : str
        Path to the audio file to play
    sr : int, optional
        Sample rate to load the audio with. If None, uses the file's native sample rate.

    Returns:
    --------
    Audio widget that can be played in the notebook

    Example:
    --------
    >>> play_audio('path/to/audio.wav')
    """
    # Load the audio file
    y, sr = librosa.load(file_path, sr=sr)


    # Return an audio widget to play the sound
    audio_widget = Audio(data=y, rate=sr)
    display(audio_widget)
```

First, we'll play the original audio file. Listen to the first minute or so to get an idea of what it sounds like before VAD.

```python
play_audio(audio_file)
```

Next, we'll listen to the first three clips to verify that we have isolated the speech in our test file:

```python
import os
audio_dir = "./output_segments/"

audio_files = os.listdir(audio_dir)
audio_files.sort()

n_clips = 3

for fname in audio_files[0:n_clips]:
    play_audio(audio_dir + fname)
```

You'll see that the audio lengths match up with the `speech_segments` output:

```
Segment 1: Speech from 6.78s to 51.62s (duration: 44.84s)
Segment 2: Speech from 53.56s to 54.27s (duration: 0.71s)
Segment 3: Speech from 55.55s to 84.76s (duration: 29.21s)
...
```

Listening to the output files, we can see that we have effectively isolated the speech.

## Conclusion

With this implementation, you now have a working Voice Activity Detection system that can identify and extract speech segments from audio files. This forms an excellent foundation for more advanced audio processing tasks like speech recognition, speaker diarization, or audio content analysis.

Look out for more content from Vast about all these other types of tasks!


Run state-of-the-art Voice Activity Detection (VAD) with PyAnnote Audio on Vast.ai for cost-effective speech processing. Extract speech segments easily!

Voice Activity Detection (VAD) with Pyannote on VAST


## Introduction

With the release of o1 family of models, reasoning has taken the AI world by storm. Instead of immediately answering a prompt, reasoning is where a model is trained to first output it's "thinking" for its answer. This increases performance on the most difficult tasks today, and since it is trained into the model, it is better than asking it to record its "chain of thought" via a prompt.

R1 takes this even further. We now have Open Source models for this type of task, which means:

1. We can retain the "thinking" token outputs which we couldn't do beforehand. This lets us create datasets for finetuning.
2. We can easily fine-tune these models on our own tasks
3. We can run them on more affordable compute to drastically reduce the cost!

This implementation combines three key components: A distilled DeepSeek model for reasoning transparency, a Vast Template for vLLM's optimized inference server for efficient deployment, and Langchain for parsing reasoning tokens versus normal output tokens. Together, they create a production-ready system that can handle both the technical demands of serving a large language model and the practical needs of processing its unique output format.

Vast.ai provides an ideal platform for this setup, offering the necessary GPU resources at a fraction of traditional cloud costs. Its marketplace model and simple Docker integration make it particularly well-suited for deploying and scaling language models, allowing developers to focus on building applications rather than managing infrastructure. The Templates feature from Vast allows for a repeatable deployment with little additional configuration.

This guide demonstrates how to deploy [deepseek/DeepSeek-R1-Distill-Qwen-32B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B) with Vast's templates and integrate them with Langchain for advanced processing capabilities. We'll show you how to leverage vLLM's optimized inference server and create custom parsers to handle DeepSeek's distinctive output format. Feel free to follow along in this [companion notebook](https://nbviewer.org/urls/bitbucket.org/!api/2.0/snippets/jsbcannell/xq8zG5/2cf7d8758235fa5176d9aee8b8d3885128c0109c/files/serving_deepseek_on_vast.ipynb)

## Setting Up the Environment

Before we can deploy our model, we need to set up our Vast.ai environment. First, install the Vast SDK:

```python
%%bash
pip install --upgrade vastai
```

Set up your Vast API key (available from your [Account Page](https://cloud.vast.ai/account/)):

```python
%%bash
# Here we will set our api key
export VAST_API_KEY="<your-key-here>"
vastai set api-key $VAST_API_KEY
```

### Choosing the Right Hardware

Now we are going to search for GPUs on Vast.ai to run the DeepSeek-R1-Distill-Qwen-32B model. This model requires specific hardware capabilities to run efficiently with vLLM's optimizations. Here are our requirements:

1. A minimum of 80GB GPU RAM to accommodate:

    - DeepSeek model weights (32B Parameters)
    - KV Cache for handling of extra long output token lengths

2. A single GPU configuration, as DeepSeek-R1-Distill-Qwen-32B can be efficiently served on one GPU:
   Note: Multi-GPU configurations are supported if higher throughput is needed.

3. A static IP address for:

    - Stable API endpoint hosting
    - Consistent client connections
    - Reliable Langchain integration

4. At least one direct port that we can forward for:

    - vLLM's OpenAI-compatible API server
    - External access to the model endpoint
    - Secure request routing

5. At least 120GB of disk space to hold the model and other things we might like to download

Here's how to search for suitable instances on Vast.ai:

```python
%%bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 80 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 120 \
rentable = true"
```

### Deploying the Server via Vast Template

We'll use vLLM's OpenAI-compatible server to deploy the DeepSeek model. This setup provides an OpenAI-compatible API endpoint that works seamlessly with existing tools and libraries.

We will do this with a template that:

1. Uses `vllm/vllm-openai:latest` docker image. This gives us an OpenAI-compatible server.
2. Forwards port `8000` to the outside of the container, which is the default OpenAI server port
3. Forwards `--model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --max-model-len 8192  --enforce-eager` on to the default entrypoint (the server itself)
4. Uses `--tensor-parallel-size 1` by default.
5. Uses `--gpu-memory-utilization 0.90` by default
6. Ensures that we have 120 GB of Disk space

```python
%%bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --disk 120 --template_hash eda062b3e0c9c36f09d9d9a294405ded
```

### Verify Setup

After deployment, verify that your server is running correctly with a simple curl test:

```python
%%bash
export VAST_IP_ADDRESS="<your-ip-here>"
export VAST_PORT="<your-port-here>"
curl -X POST http://$VAST_IP_ADDRESS:$VAST_PORT/v1/completions \
     -H "Content-Type: application/json" \
     -d '{
           "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
           "prompt": "Hello, how are you?",
           "max_tokens": 50
         }'
```

Note: You can find your instance's IP address and port in the [Instances tab](https://cloud.vast.ai/instances/) of the Vast AI Console.

## Implementing Custom Output Parsing

### Install Required Dependencies

```python
%%bash
pip install --upgrade langchain langchain-openai openai
```

### Creating a DeepSeek Output Parser

DeepSeek models output responses in a unique format with separate thinking and response sections. The thinking section, wrapped in `<think>` tags, contains the model's reasoning process, while everything after the closing tag represents the final response. This separation is valuable for understanding how the model reaches its conclusions.

Let's create a custom parser that can handle this format:

```python
from typing import Optional, Tuple
from langchain.schema import BaseOutputParser

class R1OutputParser(BaseOutputParser[Tuple[Optional[str], str]]):
    """Parser for DeepSeek R1 model output that includes thinking and response sections."""

    def parse(self, text: str) -> Tuple[Optional[str], str]:
        """Parse the model output into thinking and response sections.

        Args:
            text: Raw text output from the model

        Returns:
            Tuple containing (thinking_text, response_text)
            - thinking_text will be None if no thinking section is found
        """
        if "</think>" in text:
            # Split on </think> tag
            parts = text.split("</think>")
            # Extract thinking text (remove <think> tag)
            thinking_text = parts[0].replace("<think>", "").strip()
            # Get response text
            response_text = parts[1].strip()
            return thinking_text, response_text

        # If no thinking tags found, return None for thinking and full text as response
        return None, text.strip()

    @property
    def _type(self) -> str:
        """Return type key for serialization."""
        return "r1_output_parser"
```

Our parser does several important things:

1. It looks for the `</think>` tag to identify the boundary between thinking and response
2. It extracts and cleans up the thinking section by removing the `<think>` tags
3. It handles cases where no thinking section is present by returning None
4. It ensures both sections are properly stripped of extra whitespace

### Setting Up the Model Chain

Now that we have our parser, we need to set up a processing chain that connects our model to the parser. This chain will:

-   Send requests to our deployed model
-   Process the responses through our custom parser
-   Return the separated thinking and response sections

Here's how we set it up:

```python
from langchain_openai import ChatOpenAI
from langchain.prompts import ChatPromptTemplate
from langchain.schema.runnable import RunnablePassthrough

# Initialize the model with your Vast instance details
VAST_IP_ADDRESS="<your-ip-here>"
VAST_PORT="<your-port-here>"

openai_api_key = "EMPTY"
openai_api_base = f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"

model = ChatOpenAI(
    base_url=openai_api_base,
    api_key=openai_api_key,
    model_name="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
    max_tokens=8000,
    temperature=0.7
)

# Create prompt template
prompt = ChatPromptTemplate.from_messages([
    ("user", "{input}")
])

# Create parser
parser = R1OutputParser()

# Create chain
chain = (
    {"input": RunnablePassthrough()}
    | prompt
    | model
    | parser
)
```

Let's break down what each component does:

-   The `ChatOpenAI` instance connects to our deployed model using the OpenAI-compatible API
-   The `ChatPromptTemplate` formats our input messages
-   The `RunnablePassthrough` ensures our inputs flow through the chain correctly
-   Finally, our custom parser processes the model's output

### Example Usage

Let's test our setup with a prompt that can demonstrate the model's reasoning capabilities:

```python
prompt_text = "Explain quantum computing to a curious 10-year-old who loves video games."

thinking, response = chain.invoke(prompt_text)
print("\nTHINKING:\n")
print(thinking)
print("\nRESPONSE:\n")
print(response)
```

#### Output:

```
THINKING:

Okay, so I need to explain quantum computing to a 10-year-old who loves video games. Hmm, let's break this down. First, I should think about what a 10-year-old knows. They understand basic concepts like computers, games, maybe some video game terms like pixels or bits. They also enjoy stories or analogies that relate to their interests, so using video games as a context makes sense.

Quantum computing is a complex topic, so I need to simplify it. I remember that classical computers use bits, which are 0s and 1s. Quantum computers use qubits, which can be both 0 and 1 at the same time. That's a key difference. Maybe I can compare it to something in a video game. Maybe like a character that can be in two places at once, or a power-up that gives multiple abilities simultaneously.

I should also mention superposition and entanglement. Superposition is when a qubit can be in multiple states at once, and entanglement is when qubits are connected, so the state of one affects the others. Maybe relate that to something like a power-up that affects multiple things at once or a team of characters working together.

Another important point is that quantum computers can solve certain problems much faster than classical computers. Maybe compare it to a game where you can unlock a level or defeat a boss much quicker with a special power-up. But I should also note that not all problems are better solved by quantum computers, just specific ones.

Putting it all together, the explanation should start with what a classical computer does, then introduce qubits as special bits, explain their unique properties using video game analogies, and then talk about the advantages and limitations. I should keep the language simple, use terms they're familiar with, and make it engaging.

Let me think of an example. Maybe a game where you have to find a key in a maze. A classical computer would check each path one by one, while a quantum computer could check all paths at the same time, making it faster. Or maybe a power-up that lets you see all possible paths at once.

I also want to make it clear that it's not just faster, but it's a different way of computing. Maybe use the idea of magic or special abilities in a game that allow for unique capabilities.

I should avoid technical jargon and keep sentences short. Maybe use a narrative style, like telling a story about a game where quantum computing helps the player win. That could make it more relatable and fun.

So, to sum up, I'll start by comparing classical computers to video game rules, then introduce qubits as special bits that can do multiple things at once, explain superposition and entanglement with game-related examples, and discuss the speed advantages and specific applications. I'll wrap it up by connecting it back to video games, showing how quantum computing could make games more complex or fun in the future.

I think that's a solid approach. Now, I'll structure this into a clear, concise explanation using these points and analogies.

RESPONSE:

Quantum computing is like having a supercharged video game character! Imagine you're playing a game where you need to find a hidden treasure in a maze. A regular computer (like a normal game character) would check each path one by one, which could take a long time. But a quantum computer is like a special character that can check all paths at the same time, making it super fast to find the treasure!

In video games, you sometimes get power-ups that give you special abilities. Quantum computers use something called qubits, which are like those power-ups. Unlike regular bits in a computer that can only be 0 or 1, qubits can be both at the same time. This is called superposition. It's like having a character that can be in two places at once!

Also, qubits can work together in a magical way called entanglement. If one qubit changes, the others change too, no matter how far apart they are. It's like having a team of characters where each member's actions affect the others instantly.

Quantum computers are especially good at solving tricky puzzles or problems that would take a regular computer forever. But they're not better at everything—just certain tasks. So, in the game, your special character would shine in specific levels, making them super fun and challenging!

One day, quantum computing might help create even more exciting games, with puzzles that seem impossible to solve without those special powers. So, think of quantum computing as the ultimate power-up for solving complex problems in the blink of an eye!
```

This example shows how the model:

1. First works through its reasoning in the thinking section
2. Then provides a clear, final response
3. Keeps these two aspects separate for better analysis and control

## Key Features

-   **Separated Thinking and Response**: DeepSeek's unique output format provides insight into the model's reasoning process
-   **Custom Output Parsing**: Langchain integration enables structured parsing of the model's unique output format

## Conclusion

This guide has demonstrated how to deploy and serve the DeepSeek-R1-Distill-Qwen-32B model on Vast.ai. We've shown how to:

-   Set up a cost-effective GPU instance with the right specifications
-   Easilly deploy the model using Vast's Templates with vLLM
-   Create a custom Langchain parser to handle DeepSeek's unique thinking/response format
-   Integrate everything through an OpenAI-compatible API

With this setup on Vast.ai's GPU marketplace, you can now build powerful AI applications with DeepSeek while minimizing both infrastructure complexity and compute costs. The platform's combination of flexible GPU options and simple Docker-based deployment makes it an ideal foundation for serving and scaling language models without the expensive cost model of traditional cloud providers.


Deploy DeepSeek-R1-Distill-Qwen-32B on Vast.ai for cost-effective AI reasoning. Use vLLM, Langchain, and custom parsing to extract structured model insights efficiently.

Serving DeepSeek Models on Vast.ai with vLLM and Langchain!


## Introduction

Rerankers are powerful tools that excel at determining the relevance between pairs of text - whether you're matching search queries to documents, evaluating LLM outputs against prompts, or finding similar content in a database. Unlike simple keyword matching or embedding similarity, these specialized models perform a detailed comparison between inputs, capturing nuanced relationships that simpler methods might miss. They're particularly valuable in RAG (Retrieval Augmented Generation) systems, recommendation engines, and content filtering pipelines, where they can significantly improve quality while requiring minimal computational overhead.

vLLM has recently expanded its capabilities to include reranker model serving, offering compatibility with both OpenAI and Cohere APIs. In this guide, we'll focus on deploying the `BAAI/bge-reranker-base` model - a powerful yet efficient reranker designed for semantic similarity scoring.

Vast.ai provides a marketplace for renting GPU compute power, offering a cost-effective alternative to major cloud providers. Vast has GPU SKU's that you cannot normally find on clouds, including 4000 series GPU's which are cheaper, with a tradeoff of less RAM. Re-ranker models are often quite small so we can take advantage of these extra affordable prices.

In this guide, we will:

1. Set up a Vast.ai instance with the right GPU specifications for serving rerankers
2. Deploy the model using vLLM's optimized inference server
3. Demonstrate two ways to interact with the reranker:
    - Using the Cohere-compatible API for batch reranking
    - Using the OpenAI-compatible API for cross-encoder scoring

This setup provides a production-ready environment for serving reranker models, with the ability to handle both batch reranking requests and individual similarity scoring tasks.

## Setting Up Vast.ai

First, we'll install the Vast.ai API:

```bash
pip install --upgrade vastai
```

Next, we'll set our API key (found on the [Account Page](https://cloud.vast.ai/account/)):

```bash
export VAST_API_KEY="your-key-here"
vastai set api-key $VAST_API_KEY
```

## Choosing Hardware

The `BAAI/bge-reranker-base` model has modest requirements compared to larger language models. Here's what we need:

1. 16GB GPU RAM for:
    - Model weights (~278M parameters)
    - Batch processing overhead
    - System operations
2. Single GPU with Turing architecture or newer
3. Static IP for stable API endpoint
4. At least one direct port for the API server

Note that you could potentially go down to 8GB of GPU RAM for this model for more affordable usages.

We'll use the `vastai search offers` command to find instances that meet our requirements:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 16 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
rentable = true"
```

## Deploying the Server

Next, we'll copy the `ID` from our chosen instance into the `instance-id` field below along with our `HUGGINGFACE_TOKEN` to rent an instance and deploy `BAAI/bge-reranker-base` using vLLM's OpenAI-compatible server:

```bash
export INSTANCE_ID=<instance-id>
vastai create instance $INSTANCE_ID \
    --image vllm/vllm-openai:latest \
    --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGINGFACE_TOKEN' \
    --disk 40 \
    --args --model BAAI/bge-reranker-base
```

This setup:

-   Uses vLLM's optimized inference server
-   Exposes port 8000 inside the container (Vast.ai will forward this to a different external port)
-   Downloads and serves the `BAAI/bge-reranker-base` model

## Verify the Setup

Before proceeding, verify that your instance is running correctly:

1. Go to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console
2. Wait for the instance to download the image and model (this may take a few minutes)
3. Find your instance's IP address and port from the "Open Ports" panel - you'll see something like `XX.XX.XXX.XX:YYYY -> 8000/tcp` where YYYY is the external port that forwards to the container's port 8000
4. Use these values in `VAST_IP_ADDRESS` and `VAST_PORT` for the test request below

```bash
export VAST_IP_ADDRESS="your-ip-here"
export VAST_PORT="your-port-here"

curl -X POST http://$VAST_IP_ADDRESS:$VAST_PORT/rerank \
    -H "Content-Type: application/json" \
    -d '{
    "model": "BAAI/bge-reranker-base",
    "query": "What is deep learning?",
    "documents": [
        "Deep learning is a type of machine learning"
    ]
    }'
```

## Using the Reranker

vLLM provides two ways to rerank documents using the same underlying model:

1. **OpenAI API Compatible** (`/score`)

    - Raw scoring between query and documents
    - Returns similarity scores for manual sorting
    - Best for custom ranking logic

2. **Cohere API Compatible** (`/rerank`)
    - Direct reranking of documents against a query
    - Returns pre-sorted results
    - Best for quick integration

### OpenAI-Compatible Endpoint

First, we'll create a function to call the `/score` endpoint. We'll need to enter the `IP_ADDRESS` and `PORT` from above:

```python
import requests

IP_ADDRESS = ""
PORT = ""

def openai_score(query,documents):
    base_url = f"http://{IP_ADDRESS}:{PORT}"

    # Format request for score endpoint
    test_request = {
        "model": "BAAI/bge-reranker-base",
        "text_1": query,  # Query is text_1
        "text_2": documents  # Documents are text_2
    }

    # Make request and print raw response
    response = requests.post(f"{base_url}/score", json=test_request)
    print("Status code:", response.status_code)
    print("Raw response:", response.text)

    # If successful, print formatted results
    if response.status_code == 200:
        data = response.json()
        scores = [(doc, item["score"]) for doc, item in zip(documents, data["data"])]
        scores.sort(key=lambda x: x[1], reverse=True)

        print("\nRanked results:")
        for text, score in scores:
            print("\nScore:", score)
            print("Text:", text)
```

We'll start with a simple example to show what happens when we have completely irrelevant documents:

```python
query = "What is Deep Learning?"
documents = [
    "Deep learning is a subset of machine learning that uses neural networks with many layers",
    "The weather is nice today",
    "Deep learning enables computers to learn from large amounts of data",
    "I like pizza"
]
openai_score(query,documents)
```

In this example, we can see how effectively the reranker distinguishes between relevant and irrelevant content:

-   The most relevant document (score ~1.0) provides a direct definition of deep learning
-   The second document (score ~0.18) mentions deep learning but provides less specific information
-   The irrelevant documents about weather and pizza receive nearly zero scores (~0.00004)

```
Ranked results:

Score: 0.99951171875
Text: Deep learning is a subset of machine learning that uses neural networks with many layers

Score: 0.17626953125
Text: Deep learning enables computers to learn from large amounts of data

Score: 3.737211227416992e-05
Text: The weather is nice today

Score: 3.737211227416992e-05
Text: I like pizza
```

Next, we'll run it with a slightly more realistic set of documents to see how the reranker handles more nuanced differences in relevance:

```python
query = "What is Deep Learning?"

documents = [
    "Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input",
    "Machine learning algorithms enable computers to learn from data without being explicitly programmed",
    "The latest smartphone features advanced AI capabilities for photo enhancement",
    "Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection",
    "A neural network is inspired by the biological neural networks that constitute animal brains",
    "Cloud computing provides scalable infrastructure for training deep learning models",
    "The history of artificial intelligence dates back to the 1950s",
    "Deep learning models require significant computational resources and large datasets for training"
]
openai_score(query,documents)
```

The reranker shows sophisticated understanding in this more nuanced example:

1. The comprehensive definition gets a perfect score (1.0)
2. Application examples (computer vision) score moderately well (~0.43)
3. Related concepts (machine learning, neural networks) get low but non-zero scores (~0.03)
4. Infrastructure and historical context receive nearly zero scores (`<0.001`)

This demonstrates how the reranker can capture subtle differences in relevance, not just obvious distinctions between relevant and irrelevant content.

```
Ranked results:

Score: 1.0
Text: Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input

Score: 0.425537109375
Text: Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection

Score: 0.034942626953125
Text: Machine learning algorithms enable computers to learn from data without being explicitly programmed

Score: 0.001674652099609375
Text: A neural network is inspired by the biological neural networks that constitute animal brains

Score: 0.0004711151123046875
Text: The history of artificial intelligence dates back to the 1950s

Score: 0.00026535987854003906
Text: Deep learning models require significant computational resources and large datasets for training

Score: 5.7816505432128906e-05
Text: Cloud computing provides scalable infrastructure for training deep learning models

Score: 3.737211227416992e-05
Text: The latest smartphone features advanced AI capabilities for photo enhancement
```

### Cohere-Compatible Endpoint

The `/rerank` endpoint provides a higher-level interface that:

-   Directly reranks documents against a query
-   Returns pre-sorted results
-   Handles all scoring logic internally
-   Simplifies integration into existing pipelines

First, we'll install Cohere.

```bash
pip install --upgrade cohere
```

We'll then create a function to call the cohere endpoint.

```python
import cohere

IP_ADDRESS = ""
PORT = ""

def cohere_reranker(query,documents):
    base_url = f"http://{IP_ADDRESS}:{PORT}"

    # Initialize v2 client with our endpoint
    co = cohere.ClientV2("sk-fake-key", base_url=base_url)

    result = co.rerank(
        model="BAAI/bge-reranker-base",
        query=query,
        documents=documents
    )

    print("\nRanked results:")
    for doc in result.results:
        print(f"\nScore: {doc.relevance_score}")
        print(f"Text: {doc.document.text}")  # Access text through document.text
```

First, we'll test with our simple example:

```python
query = "What is Deep Learning?"
documents = [
    "Deep learning is a subset of machine learning that uses neural networks with many layers",
    "The weather is nice today",
    "Deep learning enables computers to learn from large amounts of data",
    "I like pizza"
]
cohere_reranker(query,documents)
```

Output:

```
Ranked results:

Score: 0.99951171875
Text: Deep learning is a subset of machine learning that uses neural networks with many layers

Score: 0.17626953125
Text: Deep learning enables computers to learn from large amounts of data

Score: 3.737211227416992e-05
Text: The weather is nice today

Score: 3.737211227416992e-05
Text: I like pizza
```

Notice that the Cohere endpoint produces identical scores to the OpenAI endpoint - this is because both are using the same underlying model, just with different APIs. The key difference is that the Cohere endpoint automatically handles the sorting and formatting of results.

Next, we'll try our more complex example:

```python
query = "What is Deep Learning?"

documents = [
    "Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input",
    "Machine learning algorithms enable computers to learn from data without being explicitly programmed",
    "The latest smartphone features advanced AI capabilities for photo enhancement",
    "Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection",
    "A neural network is inspired by the biological neural networks that constitute animal brains",
    "Cloud computing provides scalable infrastructure for training deep learning models",
    "The history of artificial intelligence dates back to the 1950s",
    "Deep learning models require significant computational resources and large datasets for training"
]
cohere_reranker(query,documents)
```

Output:

```
Ranked results:

Score: 1.0
Text: Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input

Score: 0.425537109375
Text: Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection

Score: 0.034942626953125
Text: Machine learning algorithms enable computers to learn from data without being explicitly programmed

Score: 0.001674652099609375
Text: A neural network is inspired by the biological neural networks that constitute animal brains

Score: 0.0004711151123046875
Text: The history of artificial intelligence dates back to the 1950s

Score: 0.00026535987854003906
Text: Deep learning models require significant computational resources and large datasets for training

Score: 5.7816505432128906e-05
Text: Cloud computing provides scalable infrastructure for training deep learning models

Score: 3.737211227416992e-05
Text: The latest smartphone features advanced AI capabilities for photo enhancement
```

Again we see identical scores to the OpenAI endpoint, demonstrating that:

1. Both APIs provide consistent access to the same model
2. The Cohere endpoint offers a more streamlined interface
3. You can choose whichever API better fits your application's needs

The main advantages of the Cohere endpoint are:

-   Pre-sorted results (no need to sort scores manually)
-   Simpler response format
-   Built-in batching support
-   Familiar interface for existing Cohere users

## Key Features

-   **Optimized Inference**: vLLM's server provides efficient batch processing and automatic memory management, making it easy to serve rerankers without dealing with low-level optimizations.
-   **Dual API Support**: Access the model through both OpenAI and Cohere-compatible APIs, allowing flexible integration with existing applications.
-   **Cost-Effective Deployment**: Vast.ai's marketplace lets you access powerful GPUs at a fraction of traditional cloud costs, with easy scaling as your needs grow.

## What's Now Possible

With this setup, you can build:

-   More accurate semantic search systems
-   Better RAG applications with filtered context
-   Content recommendation systems
-   Semantic duplicate detection
-   Document clustering and organization

The combination of vLLM's efficient serving and Vast.ai's affordable GPUs makes it practical to deploy rerankers in production. You can start small and scale up as needed, while maintaining high performance and cost efficiency. Happy ranking!


Deploy rerankers on Vast.ai using vLLM for cost-effective, high-performance semantic search. Supports OpenAI & Cohere APIs for flexible integration.

Serving Rerankers on Vast.ai using vLLM


The growth of AI is outpacing compute. Today, there's an unprecedented demand for computing power as interest in AI surges worldwide – and servers just can't keep up.

More advanced AI models require increasingly powerful hardware. **Without enough GPUs, these large-scale AI systems struggle to run, and also cannot improve.**

Even OpenAI's CEO, Sam Altman, has [admitted](https://techcrunch.com/2024/10/31/openai-ceo-sam-altman-says-lack-of-compute-is-delaying-the-companys-products/) that computing limitations are slowing down the company's ability to release new products. Microsoft's CFO, Amy Woods, [noted](https://www.microsoft.com/en-us/Investor/events/FY-2025/earnings-fy-2025-q1) last year that demand for AI "continues to be higher" than their "available capacity."

If industry giants like OpenAI and Microsoft are feeling the strain, smaller organizations face even steeper challenges. Without deep pockets or dedicated data centers, they often have no choice but to explore other options.

With GPUs in short supply and demand skyrocketing, many companies are turning to a more flexible, scalable solution: **GPU-as-a-service, or GPUaaS**.

## What Is GPU-as-a-Service?

Simply put, **GPUaaS is a cloud-based solution that provides on-demand access to high-performance GPUs.** It fills a gap in the AI industry by enabling customers to rent the compute power they need, scaling up or down as their workloads demand.

Big cloud providers like AWS, Google Cloud, and Azure offer GPU instances, but GPUaaS goes further. Platforms like our own here at Vast.ai tap into a decentralized network of idle GPUs from data centers, enterprises, and even individuals, making high-performance compute more affordable and widely available.

As a result, GPUaaS addresses a critical need: namely, that **AI infrastructure must be scalable, cost-efficient, and accessible to everyone.**

## Why GPUaaS Is the Key to Scalable AI Infrastructure

AI adoption shows no sign of slowing down – and with it comes fluctuating and unpredictable demand for computing power at any given time. Traditional centralized infrastructure isn't well suited to this type of volatility.

Relying on centralized GPU resources often leads to wasted potential and frustrating roadblocks, such as:

-   **High upfront costs** for purchasing and maintaining GPUs.
-   **Underutilized hardware** during slow periods, resulting in inefficiency and wasted costs.
-   **Bottlenecks** when demand spikes, slowing down innovation and delaying results.
-   **Limited access** to GPUs due to supply chain slowdowns and shortages, impacting AI scalability.

This is where GPU-as-a-Service offers a smarter solution. With GPUaaS, you can:

-   **Quickly scale AI workloads** to match demand without over-provisioning.
-   **Reduce costs** with a flexible pay-as-you-go model while eliminating the risks of heavy upfront investment.
-   **Access worldwide GPU resources**, minimizing latency and enhancing performance.
-   **Decrease energy consumption** by leveraging existing, idle GPUs instead of powering additional servers.

## GPUaaS: The Future of AI Compute is Flexible

As AI continues to advance, organizations that can efficiently scale their compute power will have a clear advantage. GPU-as-a-Service reduces the barriers of cost, accessibility, and scalability, offering an alternative to traditional infrastructure. With GPUaaS, you can tap into a global network of GPUs and ensure you always have the resources you need, when you need them.

So why wait? In a world where AI innovation moves fast, compute should never be the bottleneck. **Vast.ai makes enterprise-grade GPUs accessible and affordable through our decentralized marketplace, offering savings of 5-6X on GPU compute.** Whether you're training cutting-edge models or running large-scale inference, we provide the flexibility and efficiency to keep your AI projects moving forward.

Get started and harness the power of GPUaaS with Vast.ai today!

GPU shortages are slowing AI growth. GPU-as-a-Service (GPUaaS) offers a scalable, cost-effective solution for on-demand AI compute power.

Blog

Apple GPT: What We Know So Far

Harness the Power of Cloud GPUs with Vast.AI: Running the 70B LLama2 GPTQ

Fine-tuning the LLaMA 2 model on RTX 4090

Stretching Your Cloud Computing Budget: Introducing Reserved Instance Discounts

The Versatile Power of Large Language Models: Exploring Major Use Cases and Their Trade-offs

Nvidia RTX 4090 as a Deep Learning GPU