
# Running Nemotron-Cascade-2 on Vast.ai

NVIDIA released [Nemotron-Cascade-2-30B-A3B](https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B), a reasoning model that achieves gold-medal-level scores on international math and programming competitions. It does this with 30B total parameters and only 3B active per token - 20x fewer than competing models - outperforming both Qwen3.5-35B-A3B and larger open-weight alternatives.

The model fits on a single 80 GB GPU at about 60 GB in BF16 with no quantization or multi-GPU setup required. It includes built-in chain-of-thought reasoning, tool-integrated reasoning for math and code, and an OpenAI-compatible API.

This guide walks through deploying the model on [Vast.ai](https://vast.ai) with vLLM. Vast.ai offers A100 and H100 GPUs at 3-5x less than traditional cloud providers, with no contracts or minimums.

## Model Overview

| Property | Value |
| --- | --- |
| Developer | [NVIDIA](https://research.nvidia.com/labs/nemotron/nemotron-cascade-2/) |
| Model | Nemotron-Cascade-2-30B-A3B |
| Architecture | Hybrid Mamba-Attention MoE |
| Total Parameters | 30B |
| Active Parameters | 3B per token |
| Training | Cascade RL post-training from Nemotron-3-Nano base |
| License | [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| HuggingFace | [nvidia/Nemotron-Cascade-2-30B-A3B](https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B) |

## Deploy Nemotron-Cascade-2 on Vast.ai with vLLM

The model requires a single 80 GB GPU (A100 or H100) with about 150 GB disk for weights and cache.

### Prerequisites

- A [Vast.ai](https://cloud.vast.ai) account with credits
- The Vast CLI installed and configured:

```bash
pip install --upgrade vastai
vastai set api-key <YOUR_API_KEY>
```

Generate an API key to secure your model endpoint:

```bash
export NEMOTRON_API_KEY=$(openssl rand -hex 24)
echo $NEMOTRON_API_KEY  # save this somewhere
```

### Find a GPU

Search for the cheapest single 80 GB GPU with direct port access:

```bash
# -o dph sorts results by price (dollars per hour, cheapest first)
vastai search offers "gpu_ram>=80 num_gpus=1 direct_port_count>=1 cuda_vers>=12.4 cuda_vers<13" -o dph
```

### Launch the Instance

Pick an instance ID from the search results and deploy with vLLM:

```bash
vastai create instance <INSTANCE_ID> \
  --image vllm/vllm-openai:latest \
  --env '-p 8000:8000' \
  --disk 150 \
  --onstart-cmd "vllm serve nvidia/Nemotron-Cascade-2-30B-A3B \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 32768 \
    --api-key $NEMOTRON_API_KEY \
    --trust-remote-code"
```

This pulls the model from HuggingFace and starts an OpenAI-compatible API server on port 8000. Initial startup takes several minutes while the model downloads.

Once the instance is running, get your connection details:

```bash
vastai show instances --raw | jq '.[] | select(.id == <INSTANCE_ID>) | {id, actual_status, public_ipaddr, ports}'
```

You can also find the IP and port mapping in the [Vast console](https://cloud.vast.ai/instances/).

## Call the API

Send a request to test the model:

```bash
curl http://<VAST_IP>:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $NEMOTRON_API_KEY" \
  -d '{
    "model": "nvidia/Nemotron-Cascade-2-30B-A3B",
    "messages": [{"role": "user", "content": "Find all integers n such that n^2 + 2n + 4 is divisible by 7."}],
    "max_tokens": 2048
  }'
```

The model thinks before it answers - you will see its step-by-step reasoning in the response, followed by the final answer. This chain-of-thought reasoning is what drives its competition-level performance on math and code.

## Cleanup

When you're done, destroy the instance to stop charges:

```bash
vastai destroy instance <INSTANCE_ID>
```

## Conclusion

Nemotron-Cascade-2 delivers reasoning performance that previously required models 20x its size, and it runs on a single GPU. That combination of capability and efficiency makes it a practical option for math, code, and general reasoning workloads. With Vast.ai, you can have it running in minutes at a fraction of what traditional cloud providers charge.

## Resources

- [Nemotron-Cascade-2-30B-A3B on HuggingFace](https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B)
- [Nemotron-Cascade-2 Research Page](https://research.nvidia.com/labs/nemotron/nemotron-cascade-2/)
- [Nemotron-Cascade-2 Paper (arXiv:2603.19220)](https://arxiv.org/abs/2603.19220)
- [Vast.ai Documentation](https://docs.vast.ai/)
- [vLLM Documentation](https://docs.vllm.ai/)


Deploy NVIDIA's Nemotron-Cascade-2-30B-A3B on Vast.ai with vLLM and run a powerful reasoning model on a single 80GB GPU.

Running Nemotron-Cascade-2 on Vast.ai


We're excited to announce a new deployment guide for MiniMax-M2, the breakthrough 230 billion parameter language model that's making waves in the open-source AI community. If you've been looking for a cost-effective way to run state-of-the-art LLMs, this guide is for you.

## What Makes MiniMax-M2 Special?

MiniMax-M2 achieves the #1 composite score among open-source models while maintaining incredible efficiency. Despite its 230B parameter size, it activates only 10B parameters per inference, delivering fast responses without the typical computational overhead of large models.

Key features include:
- **Interleaved Thinking**: Transparent reasoning via `<think>...</think>` tags
- **131K Token Context**: Handle large documents and complex conversations
- **OpenAI Compatible**: Drop-in replacement for existing applications
- **MIT Licensed**: No restrictions on commercial use

## What's in the New Guide

Our latest documentation example walks you through deploying MiniMax-M2 on Vast.ai from start to finish. The guide includes:

**Complete Deployment Instructions**
- Hardware requirements and instance selection
- Step-by-step provisioning with the Vast.ai CLI
- API key authentication for secure deployments
- Configuration parameters validated through real deployments
- Expected timelines and what to watch for

**API Integration Examples**
- Python code using the OpenAI SDK with authentication
- cURL examples for quick testing
- Streaming response implementations
- Best practices for multi-turn conversations

**Troubleshooting Section**
Based on real deployment testing, we've documented the four critical issues you might encounter and their solutions:
- Docker image version requirements
- Disk space allocation
- GPU memory optimization
- CUDA driver compatibility

**Performance Data**
All metrics in the guide come from actual deployments on Vast.ai infrastructure:
- 30-minute initial deployment time
- ~7 seconds for 100-token responses
- 131K token context window
- Production-ready performance on 4x H100 GPUs

## Why Deploy on Vast.ai?

Vast.ai's GPU marketplace gives you access to enterprise hardware at competitive rates, making it economical to run powerful models like MiniMax-M2 for production workloads. Our guide uses a 4x H100 (80GB) configuration with the vLLM nightly build. You get:

- Enterprise GPU access (H100 recommended for best compatibility)
- Predictable costs with no usage-based pricing
- Full control over your infrastructure
- OpenAI-compatible API for easy integration

The guide shows you how to leverage these advantages for cost-effective LLM inference that rivals cloud API quality.

## Scaling for Production

While the 4x H100 configuration demonstrated in our guide provides an excellent starting point for deploying and testing MiniMax-M2 on Vast.ai, production deployments typically require larger GPU configurations to support longer context lengths and higher concurrent request volumes. For production use cases, consider configurations such as 8x H100, 4x H200, or 8x H200, which provide substantially more GPU memory for handling concurrent requests and extended context windows.

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Developers** building AI-powered applications who want cost control and flexibility
- **Research teams** running experiments that need high-volume inference
- **Startups** prototyping AI features without committing to expensive cloud APIs
- **Anyone** curious about deploying open-source LLMs at scale

## Get Started

The complete guide is now available in our documentation:

**[Read: Running MiniMax-M2 on Vast.ai →](https://docs.vast.ai/examples/text-generation/minimax-m2#minimax-m2)**

Whether you're exploring open-source AI options or ready to deploy your first large language model, this guide provides everything you need to get MiniMax-M2 running on Vast.ai infrastructure.

Ready to try it? [Sign up for Vast.ai](https://cloud.vast.ai) and follow the guide to deploy your first instance.

---

*All performance metrics and cost estimates in this article are based on actual deployment testing on Vast.ai infrastructure. Results may vary based on instance availability and configuration.*


Learn how to deploy MiniMax-M2, the 230B parameter language model with state-of-the-art performance, on Vast.ai infrastructure.

New Guide: Deploy MiniMax-M2 on Vast.ai




Are you looking to run powerful open-source LLMs like **Llama 4**, **Kimi K2**, or **Qwen3** without the hassle of managing complex infrastructure? Vast.ai makes it easy to train and deploy these models with a curated set of templates built for speed, flexibility, and scale. 

Our [templates](https://docs.vast.ai/templates) cover everything from browser-based interfaces like Oobabooga and Open WebUI to optimized backends like HuggingFace TGI and vLLM. 

Below, we break down the top recommended templates on **Vast.ai** and how you can use them to get your LLM project off the ground quickly and easily. 

### 1\. [Oobabooga (LLM WebUI)](https://docs.vast.ai/oobabooga-llm-webui)

[Oobabooga](https://github.com/oobabooga/text-generation-webui) provides a user-friendly web interface for interacting with open-source LLMs. Its UI  resembles the original ChatGPT style – ideal for those who prefer a graphical interface over command-line operations. 

With support for **multiple text generation backends in one UI/API**, you can switch between different models easily without restarting and maintain fine control over settings. Models like **Falcon**, **Llama**, and **Vicuna** can be loaded and explored with just a few clicks.

Our [Oobabooga template](https://docs.vast.ai/oobabooga-llm-webui) facilitates quick deployment, making it suitable for both beginners and experienced users.

### 2\. [HuggingFace TGI with Llama 3](https://docs.vast.ai/huggingface-tgi-with-llama3)

Ideal for serving **Llama 3**, this template is optimized for **high-performance text generation tasks** using HuggingFace's [Text Generation Inference (TGI) server](https://github.com/huggingface/text-generation-inference). It supports other popular open-source LLMs from HuggingFace, including **Falcon**, **StarCoder**, **BLOOM**, and **GPT-NeoX**, and it's particularly beneficial for applications requiring low-latency responses and scalability, such as chatbots or content generation tools.

For this [template](https://docs.vast.ai/huggingface-tgi-with-llama3), you'll need your own HuggingFace access token – and you'll also need to apply for permission to use Llama 3 on HuggingFace, since the model is hosted in a gated repository.

### 3\. [Open WebUI (Ollama)](https://docs.vast.ai/ollama-webui)

[Open WebUI](https://github.com/open-webui/open-webui) is an extensible, user-friendly platform for self-hosted AI deployments. It operates entirely offline and supports a range of open-source LLM backends like **Llama 4**, **Kimi K2**, **DeepSeek R1**, and **OpenAI-compatible APIs**. 

With **built-in support for retrieval-augmented generation** (RAG), Open WebUI seamlessly integrates document interactions and web search into the chat experience. You can even incorporate image generation capabilities as well as engage with multiple LLMs simultaneously in parallel.

Our [Ollama \+ WebUI template](https://docs.vast.ai/ollama-webui) will automatically set up Open WebUI as a web-based interface and expose a port for the Ollama API, making it easy to run and interact with LLMs directly from your instance.

### 4\. [vLLM](https://docs.vast.ai/vllm-llm-inference-and-serving)

Optimized for [serving open-source LLMs](https://vast.ai/article/serving-online-inference-with-vllm-api-on-vast) with high-throughput inference, [vLLM](https://github.com/vllm-project/vllm) uses a novel architecture that drastically reduces memory overhead, making it easier to serve large models like **Llama**, **Mistral**, and **Falcon** efficiently. Notably, it supports continuous batching for faster and more efficient multi-user inference at scale.

vLLM provides an OpenAI-compatible API server for easy integration into existing workflows, and it's particularly well suited for developers building **commercial or research applications that require fast and stable model responses.** The vLLM framework seamlessly supports most open-source models on HuggingFace, including the ones named above as well as **Mixtral**, **DeepSeek**, **LLaVA**, **BLOOM**, **GPT-NeoX**, **Qwen**, and more.

Our [vLLM template](https://docs.vast.ai/vllm-llm-inference-and-serving) contains everything needed for you to get started – all you have to do is specify the model you want to serve and the corresponding vLLM configuration.

### From Setup to Serving in Minutes

With Vast.ai's market-based cloud GPU rental platform, you can avoid the usual infrastructure and budget roadblocks as you spin up high-performance instances tailored to your workloads. 

Our [library](https://cloud.vast.ai/templates) of **ready-to-use templates** – covering everything from web-based interfaces to optimized inference engines – make it easy to train and deploy LLMs at scale. We take care of the heavy lifting, so you can spend less time configuring and more time building.

Why wait? **Get started with open-source LLMs on Vast.ai today\!**

Run open-source LLMs like Llama 4, Qwen3, and Kimi K2 with ease. Vast.ai templates let you deploy fast using Oobabooga, vLLM, HuggingFace TGI, or Open WebUI.

Posts about: Llm Deployment

Running Nemotron-Cascade-2 on Vast.ai

New Guide: Deploy MiniMax-M2 on Vast.ai

Open-Source LLMs You Can Train and Deploy on Vast.ai Right Now