
dstack is an open-source GPU orchestration platform that automates instance provisioning and lifecycle management across cloud providers. This guide walks through using dstack with Vast.ai as the backend, combining declarative infrastructure with competitive GPU marketplace pricing.

## What Makes dstack Special?

Key features include:
- **Infrastructure as Code**: Define GPU requirements, pricing limits, and workloads in YAML
- **Automatic Provisioning**: dstack finds and provisions the best available instances
- **Cost Controls**: Set `max_price` to cap hourly costs automatically
- **Built-in Proxy**: Access services through dstack's authenticated endpoint

## What's in the New Guide

Our latest documentation walks you through deploying language models with dstack and vLLM on Vast.ai.

**Complete Setup Instructions**
- Installing and configuring dstack with your Vast.ai API key
- Starting the dstack server and CLI
- Creating service configurations for vLLM deployments

**Working Service Configurations**
- Ready-to-use YAML for deploying Qwen3-30B-A3B
- GPU memory and pricing parameters
- Real deployment outputs showing what to expect

**API Integration Examples**
- Python code using the OpenAI SDK
- cURL examples for testing
- Streaming response implementations

## Why dstack + Vast.ai?

The combination gives you orchestration on top of a GPU marketplace:

- **Simplified Workflow**: No manual instance hunting or environment setup
- **Cost Optimization**: dstack finds the cheapest instance meeting your requirements
- **Flexible Pricing**: Access Vast.ai's competitive rates with automatic cost caps
- **Production-Ready API**: vLLM provides OpenAI-compatible endpoints

The guide demonstrates deploying Qwen3-30B-A3B on an H100 80GB—provisioned with a single command.

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Teams** wanting reproducible, version-controlled GPU deployments
- **Developers** building LLM applications who want simple infrastructure management
- **Anyone** tired of manually provisioning and configuring GPU instances

## Get Started

The complete guide is now available in our documentation:

**[Read: Deploy LLMs with dstack and vLLM on Vast.ai →](https://docs.vast.ai/examples/serving-infrastructure/dstack-vllm)**

Whether you're new to GPU orchestration or looking for a better way to manage LLM deployments, this guide provides everything you need to get started with dstack on Vast.ai.

Ready to try it? [Sign up for Vast.ai](https://vast.ai) and follow the guide to deploy your first model.


Learn how to deploy language models with dstack and vLLM on Vast.ai, combining declarative infrastructure with competitive GPU marketplace pricing.

Deploy LLMs with dstack on Vast.ai


DR-Tulu is AI2's open-source research agent—an 8B model that autonomously plans research strategies, searches the web, reads pages, and synthesizes comprehensive answers with citations. Unlike typical LLMs with tools bolted on, DR-Tulu was trained end-to-end with its MCP tools, making it a compelling open alternative to proprietary research APIs. This guide walks through deploying DR-Tulu on Vast.ai, where you can run the full agent stack on affordable GPU instances and integrate it into your own agentic applications.

## What Makes DR-Tulu Special?

Key features include:
- **Native MCP Integration**: Trained with web search and page reading tools from the start
- **Autonomous Research**: Plans strategies, gathers sources, and synthesizes cited answers
- **Efficient Architecture**: 8B parameters with strong research capabilities

## What's in the New Guide

Our latest documentation walks you through deploying DR-Tulu on Vast.ai with a split architecture: vLLM for GPU inference on Vast.ai, MCP backend running locally for tool execution.

**Complete Deployment Instructions**
- Instance selection with the Vast.ai CLI
- vLLM configuration with the correct version (critical for compatibility)
- MCP backend setup with required API keys
- Workflow configuration to connect the components

**Three Ways to Use DR-Tulu**
- Interactive chat for exploratory research
- Batch evaluation against built-in datasets
- Python API for programmatic integration

## Why This Architecture?

The split deployment keeps GPU-intensive inference on Vast.ai while running tool orchestration locally. This means:

- Your API keys (Serper, Jina) stay on your machine (or wherever you deploy the MCP backend)
- Model inference doesn't require sending data to third-party APIs
- You can modify the MCP backend without redeploying the model
- Lower Vast.ai costs since you only need GPU for inference

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Researchers** who need cited, source-backed answers at scale
- **Developers** building applications that require web research capabilities
- **Teams** evaluating open alternatives to proprietary research APIs

## Get Started

The complete guide is now available in our documentation:

**[Read: Running DR-Tulu on Vast.ai →](https://docs.vast.ai/examples/mcp/dr-tulu)**

Whether you're exploring agentic AI architectures or ready to deploy your own research assistant, this guide provides everything you need to get DR-Tulu running on Vast.ai infrastructure.

Ready to try it? [Sign up for Vast.ai](https://vast.ai) and follow the guide to deploy your first instance.


Learn how to deploy DR-Tulu, AI2's open-source research agent, on Vast.ai for autonomous web research with citations.

Deploy Deep Search with DR-Tulu on Vast.ai


## Introduction

OpenAI has just released the GPT-OSS family of models, marking their return to open-weight model releases. The family includes GPT-OSS-120B (120 billion parameters) and GPT-OSS-20B (20 billion parameters), offering developers access to state-of-the-art language capabilities previously locked behind proprietary APIs.

GPT-OSS offers configurable reasoning capabilities with reasoning levels - from "low" for quick responses to "high" for complex problem-solving. The model uses OpenAI's harmony encoding system, which structures conversations and reasoning in a sophisticated format that enables fine-grained control over the model's thinking process.

This breakthrough means you can now:

-   Deploy OpenAI-quality models on your own infrastructure
-   Scale inference according to your exact needs

Vast.ai provides the perfect platform for running GPT-OSS models. With its marketplace of GPUs, you can choose the right hardware for your needs - whether running the efficient 20B model or the powerful 120B variant.

In this guide, we'll show you how to deploy both GPT-OSS models on Vast.ai using vLLM for optimized inference, with a focus on the 120B model. You'll learn how to interact with these models using the harmony encoding system for different reasoning levels.

## Setting Up the Environment

Before we can deploy our model, we need to set up our Vast.ai environment. First, install the Vast SDK:

```bash
# Install required packages
pip install --upgrade vastai
pip install --upgrade openai
pip install --upgrade openai-harmony
```

Set up your Vast API key (available from your [Account Page](https://cloud.vast.ai/account/)):

```bash
# Set your Vast.ai API key
export VAST_API_KEY="" # Your key here
vastai set api-key $VAST_API_KEY
```

## Choosing the Right Hardware

The GPT-OSS models have different hardware requirements:

-   **GPT-OSS-120B**: Requires an H100 GPU with 80GB VRAM
-   **GPT-OSS-20B**: Requires 16GB+ VRAM

For this guide, we'll demonstrate with the 120B model on H100. Let's search for suitable instances:

```bash
# Search for suitable GPU instances
vastai search offers " \
gpu_name = H100_SXM \
geolocation=US \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 120 \
rentable = true"
```

## Deploying GPT-OSS with vLLM

Now let's deploy our GPT-OSS instance using vLLM's optimized inference server. We'll use the latest vLLM image with special GPT-OSS support. The pip install commands are from the model's Hugging Face pages ([120B](https://huggingface.co/openai/gpt-oss-120b), [20B](https://huggingface.co/openai/gpt-oss-20b)).

### Deploying GPT-OSS-120B (H100 recommended)

```bash
# Deploy vLLM instance for 120B model
export INSTANCE_ID= # Insert instance ID from search results

vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 160  --onstart-cmd 'uv pip install --system --upgrade transformers kernels torch openai; uv pip install --system --pre vllm==0.10.1+gptoss --extra-index-url https://wheels.vllm.ai/gpt-oss/ --extra-index-url https://download.pytorch.org/whl/nightly/cu128; vllm serve openai/gpt-oss-120b --max-model-len 80000'
```

### Alternative: Deploying GPT-OSS-20B on A100

Here's how to deploy the 20B model on an A100 GPU:

```bash
# Deploy vLLM instance for 20B model
export INSTANCE_ID= # Insert instance ID from search results

vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 80  --onstart-cmd 'uv pip install --system --upgrade transformers kernels torch openai; uv pip install --system --pre vllm==0.10.1+gptoss --extra-index-url https://wheels.vllm.ai/gpt-oss/ --extra-index-url https://download.pytorch.org/whl/nightly/cu128; vllm serve openai/gpt-oss-20b'
```

After deployment, wait for the model to download and start serving. You can monitor the logs to see when it's ready. Once running, find your instance's IP address and port from the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console.

## Interacting with GPT-OSS Using Harmony Encoding

GPT-OSS uses OpenAI's harmony encoding system, which provides structured conversation formatting and enables different reasoning levels. Let's set up our client:

```python
from openai import OpenAI
from openai_harmony import (
    load_harmony_encoding,
    HarmonyEncodingName,
    Role,
    Message,
    Conversation,
    SystemContent,
    DeveloperContent,
)

# Your server details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize client
client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

# Load harmony encoding
enc = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)

## Simple Chat Function
def chat_gpt_oss(prompt, reasoning="medium"):
    """
    Simple chat with GPT-OSS
    reasoning: "low", "medium", or "high"
    """
    # Create conversation with harmony format
    conversation = Conversation.from_messages([
        Message.from_role_and_content(Role.SYSTEM, SystemContent.new()),
        Message.from_role_and_content(
            Role.DEVELOPER,
            DeveloperContent.new().with_instructions(f"Reasoning: {reasoning}")
        ),
        Message.from_role_and_content(Role.USER, prompt),
    ])

    # Get formatted prompt
    formatted_prompt = enc.render_conversation_for_completion(conversation, Role.ASSISTANT)

    # Make API call
    response = client.completions.create(
        model="openai/gpt-oss-120b",
        prompt=formatted_prompt,
        max_tokens=2000,
        temperature=0.7,
    )

    return response.choices[0].text


prompt = "Explain quantum computing to a 4th grader. Keep in concise."
```

## Comparing Reasoning Levels

One of GPT-OSS's most interesting features is its configurable reasoning levels. Let's see how the model responds differently to the same prompt with low and high reasoning:

### Low Reasoning Response

```python
result = chat_gpt_oss(prompt, "low")
print("----LOW REASONING----")
print(result)
```

Output:

```
----LOW REASONING----
analysisWe need to explain quantum computing to a 4th grader, concise. Use simple analogies, like bits as light switches, qubits as coins that can be heads, tails, or both. Keep short.assistantfinal**Quantum Computing in a Nutshell (for a 4th‑grader)**

1. **Regular computers = light switches**
   - A normal computer works with bits that are either **0** (off) or **1** (on), just like a light switch that's either down or up.

2. **Quantum computers = magic coins**
   - A quantum computer uses **qubits**.
   - Think of a qubit like a coin that can be **heads (0)**, **tails (1)**, **or both at the same time** while it's spinning in the air. That "both at once" is called **superposition**.

3. **Working together**
   - When you have many regular bits, you have to try every possible combination one after another.
   - With many qubits, they can be in many combinations **all at once** because they can be linked together in a special way called **entanglement**. It's like a group of magic coins that always know what the others are doing, even if they're far apart.

4. **Why it matters**
   - Because of superposition and entanglement, a quantum computer can solve some puzzles (like cracking codes or simulating molecules) **much faster** than a regular computer.

**Bottom line:** A regular computer is like a row of on/off switches; a quantum computer is like a bunch of spinning magic coins that can be many things at once, letting it try many solutions at the same time.
```

### High Reasoning Response

```python
result = chat_gpt_oss(prompt, "high")
print("----HIGH REASONING----")
print(result)
```

Output:

```
----HIGH REASONING----
analysisWe need to explain quantum computing to a 4th grader, concise. Use simple analogies, avoid jargon, but keep accurate. Should be short. Provide analogies like coins, dice, superposition, entanglement. Keep concise.assistantfinal**What is a quantum computer?**

Imagine you have a super‑smart robot that can solve puzzles much faster than a regular computer. A quantum computer is that robot, but instead of using normal "on/off" switches (bits) like a regular computer, it uses tiny, magical switches called **qubits**.

### Bits vs. Qubits
- **Bit (regular computer):** Like a tiny light that can be **off (0)** or **on (1)**.
- **Qubit (quantum computer):** Like a tiny spinning top that can be **0**, **1**, **or both at the same time** (this is called *superposition*).

### Why is that cool?
If you have 10 bits, they can show only one of 2¹⁰ = 1,024 possible patterns at a time.
If you have 10 qubits, they can be in all 2¹⁰ = 1,024 patterns **at once**! That lets a quantum computer try many solutions together.

### A Simple Analogy
Think of a maze puzzle:

- **Regular computer:** Sends a single mouse down the maze, step by step, until it finds the exit.
- **Quantum computer:** Sends a whole swarm of mice that can explore every possible path at the same time and tell you the exit almost instantly.

### Entanglement (Teamwork)
Sometimes two qubits become "best friends" and link together, no matter how far apart they are. Changing one instantly changes the other. This is called **entanglement**, and it lets quantum computers coordinate their work in a super‑efficient way.

### Bottom Line
A quantum computer uses qubits that can be 0, 1, or both at once, and they can be entangled with each other. This lets the machine look at many possibilities at the same time, solving certain problems way faster than ordinary computers.

So, it's like a super‑fast, super‑clever puzzle‑solver that can try lots of answers all at once!
```

## Comparing Low vs High Reasoning Outputs

The quantum computing explanations demonstrate the distinct capabilities of GPT-OSS-120B's reasoning levels. With low reasoning, the model produced a concise, structured explanation using simple analogies like "magic coins" and "light switches" - delivering clear concepts in a compact format suitable for quick understanding.

The high reasoning response revealed significantly more sophisticated thinking. The model provided detailed analogies like the maze puzzle with mice, explained mathematical concepts (2¹⁰ patterns), and structured the explanation with clear sections and formatting. Most notably, it offered multiple perspectives on the same concepts - from the spinning top analogy to the teamwork explanation of entanglement.

This difference in reasoning depth represents a significant advancement in model control. Rather than simply adjusting temperature or other parameters, GPT-OSS allows direct specification of how thoroughly the model should think through problems. The low reasoning approach prioritized efficiency while maintaining accuracy, while high reasoning prioritized comprehensive understanding with rich examples and detailed explanations.

## Conclusion

OpenAI's first open-source model thus introduces not just accessibility to advanced language capabilities, but a new paradigm for controlling model reasoning depth - enabling applications to dynamically adjust between quick responses and thorough analysis based on context and requirements.


Learn how to deploy and run OpenAI's GPT-OSS models on Vast.ai using vLLM, with configurable reasoning levels and harmony encoding system.

Running OpenAI's GPT-OSS on Vast.ai


At **Vast.ai**, we believe privacy and security are inextricably woven into our mission to equip customers worldwide with accessible, scalable GPUs. In honor of this commitment, we are pleased to announce that we have achieved **SOC 2 Type II certification** – an independent assurance that with us, your data is secure, reliably available, and continuously protected.

Vast.ai’s Enduring Security Commitment

Earlier this year, our [SOC 2 Type I audit](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) confirmed we have the right internal controls in place to meet trust services criteria for data security, integrity, and privacy. The Type II audit put those controls to the test, evaluating their operational effectiveness over an extended period.

Achieving **SOC 2 Type II certification** is independent validation that the security infrastructure we've built over 6+ years of trusted service continues to meet rigorous [compliance standards](https://vast.ai/article/security-and-compliance-at-vast-ai) in day-to-day execution. This achievement strengthens our ability to support the evolving security needs of enterprises, research institutions, and organizations building on our infrastructure.

Here's a quick look at SOC 2 Type II and what it entails.

### What Is SOC 2 Type II?

System and Organization Controls 2 (SOC 2\) audits – developed by the American Institute of Certified Public Accountants (AICPA) – protect customer data by rigorously evaluating an organization’s internal controls across security, confidentiality, privacy, and processing integrity benchmarks.

- **Type I** reports provide a control design snapshot at a single point in time.

- **Type II** reports go further, assessing how those controls perform over months of continuous operation – confirming they're both well-designed and consistently implemented.

Achieving both Type I and Type II certifications means your sensitive data is secure, available, and privately stored across all our systems. This establishes a foundation of enduring excellence aligned with rigorous industry standards for all customers, which we can tailor to your unique data security and regulatory compliance needs to scale infrastructure and further reduce risks.

After completing our initial 3-month Type II audit, we’ve launched a standard 12-month cycle to ensure we have no gaps in coverage. This two-phase strategy lets us move fast on improvements, then settle into an annual rhythm of verification. We are now in the 12-month cycle with continuous coverage year round.

### Vast.ai's Secure Cloud Offering

For customers with stricter security and compliance requirements, our [**Secure Cloud**](https://cloud.vast.ai/?secure_cloud=true) offering provides high-security GPU access through our certified datacenter partners. This environment, already backed by stringent operational and security controls, now benefits from the added assurance of a SOC 2 Type II audit across our platform infrastructure.

Our **[Trust Center](https://trust.vast.ai/?utm_source=blog)** offers a deeper look into our controls, compliance stance, FAQs and Subprocessors, and our other compliance certifications provide additional safeguards.

Why This Matters

Earning our **SOC 2 Type II certification** is another step forward in strengthening our foundation of trust and security that our customers rely on every day. It reflects both our adherence to proven industry practices and our ongoing commitment to transparency.

This achievement isn't the end of our compliance journey – it's part of an ongoing initiative. Vast.ai will undergo SOC 2 Type II audits every 12 months to ensure continual coverage.

Additionally, we've requested an [**SOC 3 report**](https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-3), which will provide a publicly available summary of our security posture for those who want to review our controls without accessing the full SOC 2 report.

### Our Ongoing Commitment

Achieving the **SOC 2 Type II certification** is a crucial step in our compliance roadmap as we continue to build trustworthy and dependable GPU infrastructure for our users. Alongside other security and [compliance measures](https://vast.ai/compliance) across cloud and AI services, such as our [SOC 2 Type I certification](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) and [Vulnerability Bounty Program](https://vast.ai/article/announcing-the-vast-ai-vulnerability-bounty-program), we are weaving security and trust throughout our platform for every member of our community.

Through this integrated approach, we’re dedicated to protecting the privacy and integrity of the data entrusted to us by developers, researchers, and enterprise teams around the world. We take this responsibility seriously as a distributed peer-to-peer platform powering global compute, and we look forward to sharing future updates as we continue to advance our security and compliance roadmap.

For more information on Vast.ai's comprehensive security and compliance stance, or to request a copy of our SOC 2 reports, please contact us at [compliance@vast.ai](mailto:compliance@vast.ai).

Thank you to our team at Vast.ai and our audit partners for helping us achieve this milestone.

We’re committed to protecting your data and upholding the trust you place in us, and we’ll keep you updated on our ongoing progress in our mission to provide powerful, dependable GPUs.


Vast.ai has achieved SOC 2 Type II certification, demonstrating our commitment to enterprise-grade security, data protection, and compliance. Learn about our comprehensive security controls, Secure Cloud offering, and ongoing compliance roadmap that ensures your data remains secure and protected.

Deploy LLMs with dstack on Vast.ai

What Makes dstack Special?

What's in the New Guide

Why dstack + Vast.ai?

Who Should Use This Guide?

Get Started

Deploy Deep Search with DR-Tulu on Vast.ai

Running OpenAI's GPT-OSS on Vast.ai

Vast.ai Achieves SOC 2 Type II Certification

Subscribe for our product updates.