
We're back with a new update from the team at **Vast.ai**\! This month, we've rolled out fresh hardware options, new video-generation tools, and continued improvements behind the scenes.

The big news is that our GPU lineup now includes the **NVIDIA DGX B200** as well as the **RTX PRO 6000 WS**\! These powerful machines open the door to even more demanding workloads, bringing you the compute power required for everything from multi-modal inference to precision-driven simulations.

We also recently reached an exciting milestone: we now have **over one thousand RTX 5090 GPUs available on Vast** – 1,180 and counting.

Here are the details about what's new this month\!

### NVIDIA Cloud GPU Updates

Whether you're training foundation models, rendering high-resolution video, or running inference, our latest hardware additions give you extra flexibility:

-   **NVIDIA DGX B200** – A universal AI supercomputer built on the Blackwell architecture, equipped with **eight GPUs** and **1.4 TB** of GPU memory. With massive bandwidth and compute capacity, it's well suited to handle virtually any enterprise AI workload.

-   **RTX PRO 6000 WS** – A high-end Blackwell-based workstation GPU featuring **96 GB** of GDDR7 memory and up to **4000 TOPS** of AI compute. It can easily handle massive datasets and multi-billion-parameter models while supporting seamless multi-app workflows.

### New Guide and Templates

We've focused on video generation this month, adding a couple of new templates and a guide to help you go from idea to clip as seamlessly as possible:

-   [**Guide to Video Generation**](https://docs.vast.ai/video-generation) – A walkthrough for setting up and using ComfyUI on Vast.ai, so you can create advanced stable diffusion pipelines (ideal for video generation workflows) with this powerful node-based interface.

-   **LTX Video (ComfyUI)** – A ready-to-use template that pairs the LTX Video model with ComfyUI, optimized for consumer-grade GPUs.

-   **Open-Sora** – A template for this open-source video generation model with a streamlined, user-friendly platform that lowers the barrier to efficient, high-quality video content creation.

And that's a wrap for June 2025\!

### Our Commitment

Whether you're looking to launch a B200 instance, experiment with video generation using our latest templates, or scale up with our expanding lineup of RTX 5090 GPUs, we're here to support your work at every stage. Our goal at **Vast.ai** is to make powerful compute accessible, flexible, and easy to use – so you can focus on building, training, and creating.

Need help? Reach out to our customer support team anytime at [support@vast.ai](mailto:support@vast.ai), or connect with the community on our Discord server for tips, troubleshooting, and more.

### Change Log

#### New Templates & Guides:

-   [Guide to Video Generation](https://docs.vast.ai/video-generation)
-   LTX Video (Comfy UI) – Video-Gen model optimized for consumer-grade GPUs.
-   Open-Sora – Democratizing efficient video production for everyone.

#### New GPUs:

-   DGX B200
-   RTX PRO 6000 WS

Plus we now have over one thousand 5090 GPUs available.


New DGX B200 and RTX PRO 6000 WS GPUs, over 1000 RTX 5090s available, and new video generation templates.

June 2025 Product Update


Ten years ago today, our CEO and founder, Jacob Cannell, published ["The Brain as a Universal Learning Machine"](https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine) on LessWrong.com.

Drawing from neuroscience, machine learning, and systems theory, he proposed that **the human brain is not a patchwork of specialized, genetically programmed modules, but a general-purpose learning system** – or, as he put it, "a biological implementation of a _Universal Learning Machine_."

In hindsight, the timing was notable. Cannell published the piece just as deep learning was moving from niche research to broader relevance in artificial intelligence (AI). In the decade since then, rapid progress in scalable, trainable, data-driven models has validated the idea that learning, not pre-engineered structure, is the key to building intelligent systems.

So on the tenth anniversary of its publication, we'd like to revisit the key ideas of the article, reflect on what's aged well, and look ahead at what's still unfolding.

We’d encourage you to read the full article on [Less Wrong](https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine).


A look at the brain as a universal learning machine, and how it can be used to learn about the world.

A Decade Later: Revisiting the Brain as a Universal Learning Machine


Serving multiple machine learning models simultaneously is a key challenge in production AI systems. The increasing size of foundational models and constraints on GPU memory often mean that deploying separate models for each task results in high RAM usage, underutilized hardware, and latency problems caused by repeatedly swapping entire models in and out of GPU memory. This not only increases cost but also complicates scaling.

However, by leveraging Lorax, a framework for dynamically loading LoRA adapters—and Vast.ai's flexible GPU marketplace, developers can efficiently serve several specialized models on a single base model deployment. This results in significantly better hardware utilization, reduced infrastructure costs, and low-latency inference for diverse AI workloads, making this a game-changer for enterprises deploying their own AI infra.

In this blog post, we'll explore how to deploy and run multiple LoRA adapters on a shared base model using Lorax on Vast.ai's cloud platform. This setup enables hosting thousands of fine-tuned models simultaneously, loading task-specific adapters on demand, and seamlessly switching between tasks such as math problem solving and customer support classification with minimal overhead.

---

## What Are LoRA Adapters and Lorax?

- **LoRAs (Low-Rank Adaptation):** Lightweight, task-specific parameter adjustments that adapt a large base model to new tasks with far fewer parameters than full fine-tuning. LoRAs can be swapped in and out without needing to reload the entire large model, saving significant RAM and time.

- **Lorax (LoRA eXchange):** An efficient serving framework that enables running thousands of LoRA adapters on a single GPU by dynamically loading these adapters into a base model at inference time. Lorax maintains high throughput and low latency while drastically reducing the deployment cost of hosting multiple fine-tuned variants.

---

## Deploying Multiple Models Using Lorax and Vast.ai

The following example uses the `mistralai/Mistral-7B-v0.1` base model hosted on Vast.ai, along with two LoRA adapters:

- `predibase/gsm8k`: Designed for solving math problems like the GSM8K benchmark.
- `predibase/customer_support`: Specializes in customer service query classification.

We deploy the base model once and dynamically switch between these specialized LoRA adapters, enabling efficient multi-model serving on the same underlying GPU instance.

---

### Step 1: Set Up Vast.ai Environment

1. **Install Vast CLI and configure API key:**

```bash
pip install vastai==0.2.6

export VAST_API_KEY="your_vast_api_key"
vastai set api-key $VAST_API_KEY
```

2. **Search for a suitable GPU instance:**

The `mistralai/Mistral-7B-v0.1` model requires at least 16GB VRAM, but it's safer to select a larger instance (e.g., 32GB VRAM) for LoRA adapter loading.

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 32 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

3. **Deploy the instance:**

Make sure you have accepted the usage terms for `Mistral-7B-v0.1` on Huggingface before proceeding.

```bash
export INSTANCE_ID=your_instance_id

vastai create instance $INSTANCE_ID --image ghcr.io/predibase/lorax:main \
--env '-p 8080:80 --shm-size 1g -e HUGGING_FACE_HUB_TOKEN="your_hf_token"' \
--disk 80 --args --model-id mistralai/Mistral-7B-v0.1
```

This command runs the Lorax server that hosts the base model and dynamically loads adapters on demand.

4. **Get Instance IP Address and Port**

Now, we need to get our IP address and port to call our model. We first need to wait for our machine to download the image and the model and start serving. This will take a few minutes. The logs will show you when it's done.

Next, go to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console and find the instance you just created.

At the top of the instance, there is a button with an IP address in it. Click this and a panel will show up of the ip address and the forwarded ports.
You should see something like:

```
Open Ports
XX.XX.XXX.XX:YYYY -> 8000/tcp
```

You will need the IP address (XX.XX.XXX.XX) and the port (YYYY) for the next step.

---

### Step 2: Query LoRA Adapters via HTTP Requests

Lorax supports on-demand loading of LoRAs from the Huggingface Hub, minimizing memory overhead versus deploying multiple large models simultaneously.

First, configure environment variables to connect to your Vast instance and set your Huggingface token:

```python
HF_TOKEN = "your_hf_token"
VAST_IP_ADDRESS = "your_instance_ip"
VAST_PORT = "your_vast_port"
```

#### Example: Math Problem Solving with `predibase/gsm8k`

Prepare a prompt:

```python
question = "James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week?"
math_prompt = f"Please answer the following question: {question}\nAnswer"
```

Make a POST request to the `/generate` endpoint specifying the LoRA adapter ID:

```python
import requests

headers = {
    "Content-Type": "application/json",
    "Authorization": f"Bearer {HF_TOKEN}"
}
data = {
    "inputs": math_prompt,
    "parameters": {
        "max_new_tokens": 100,
        "adapter_id": "predibase/gsm8k",
        "adapter_source": "hub"
    }
}

url = f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/generate"
response = requests.post(url, headers=headers, json=data)
print("Response:", response.json()["generated_text"])
```

**Response:**

```
He runs 3*3=<<3*3=9>>9 sprints a week
So he runs 9*60=<<9*60=540>>540 meters a week
#### 540
```

#### Example: Customer Support Classification with `predibase/customer_support`

Prepare a customer support transcript prompt:

```python
transcript = "Hi I am having trouble with my account. It says I need to reset my password to log in but I already reset my password."

customer_support_prompt = f"""Consider the case of a customer contacting the support center.
The term task type refers to the reason for why the customer contacted support.

### The possible task types are: account issue, billing issue, product issue, none of the above

Summarize the issue/question/reason that drove the customer to contact support:

Transcript: {transcript}

Task Type:
"""
```

Make the API call:

```python
data = {
    "inputs": customer_support_prompt,
    "parameters": {
        "max_new_tokens": 100,
        "adapter_id": "predibase/customer_support",
        "adapter_source": "hub"
    }
}

response = requests.post(url, headers=headers, json=data)
print("Response:", response.json()["generated_text"])
```

**Response:**

```
account issue
```

---

### Step 3: Access Models Using the OpenAI SDK

Lorax provides compatibility with the OpenAI API, allowing easy integration into existing apps with minimal code changes.

1. **Install OpenAI SDK**

```bash
pip install --upgrade openai
```

2. **Call the math adapter:**

```python
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1",
    default_headers={"Authorization": f"Bearer {HF_TOKEN}"}
)

resp = client.completions.create(
    model="predibase/gsm8k",
    prompt=math_prompt,
    max_tokens=100,
    temperature=0.7
)

print("Response:", resp.choices[0].text)
```

Output:

```
James runs 3 x 3 = <<3*3=9>>9 sprints in a week.
He runs 9 x 60 = <<9*60=540>>540 meters in a week.
#### 540
```

3. **Call the customer support adapter:**

```python
resp = client.completions.create(
    model="predibase/customer_support",
    prompt=customer_support_prompt,
    max_tokens=100,
    temperature=0.7
)
print("Response:", resp.choices[0].text)
```

Output:

```
account issue
```

---

## Why This Approach Matters

Serving multiple specialized AI models traditionally involves deploying each fine-tuned model independently, consuming lots of GPU memory and resulting in higher cost and slower inference when switching tasks.

By contrast, Lorax’s approach uses a single shared base model and dynamically loads lightweight LoRA adapters tailored for different tasks. This yields several benefits:

- **Reduced RAM and GPU usage:** Only the small adapter layers are loaded on demand, not the entire large model.
- **Lower infrastructure cost:** A single GPU instance can serve thousands of specialized models.
- **Faster context switching:** Switching between tasks is near-instant as you don’t reload the full model.
- **Flexibility:** New adapters can be added or updated independently.
- **Ease of integration:** OpenAI-compatible APIs simplify adoption.

Combined with Vast.ai’s cost-effective, flexible GPU rental options, this method provides a scalable, production-ready solution for deploying multi-model AI services with a minimal footprint.

---

## Conclusion

Efficiently serving multiple machine learning models is vital for scaling AI-powered products. The combination of Lorax’s dynamic LoRA adapter serving and Vast.ai’s flexible GPU infrastructure offers a powerful, cost-effective, and scalable way to support many specialized models on a single deployment.

Whether solving math problems, classifying customer inquiries, or other tasks, this approach reduces overhead and unlocks practical multi-model serving for modern AI applications.

Try setting up your own multi-LoRA serving environment today to experience the benefits firsthand!

---

## Useful Resources

- [Lorax GitHub Repository](https://github.com/predibase/lorax)
- [Vast.ai GPU Marketplace](https://vast.ai)
- [Mistral-7B-v0.1 on Huggingface](https://huggingface.co/mistralai/Mistral-7B-v0.1)
- [OpenAI Python SDK](https://github.com/openai/openai-python)

---

_Happy modeling! 🚀_


Serve multiple ML models efficiently using Lorax and LoRA adapters on Vast.ai—cut costs, reduce latency, and maximize GPU use with dynamic adapter loading.

Efficiently Serving Multiple Machine Learning Models with Lorax and vLLM on Vast.ai


Nearly every startup faces the same two pressures: to move fast and make every dollar count.

Time and budget may be limited while demands remain high – especially for teams working with AI, machine learning, or other compute-heavy workloads. For startups, the pressure to deliver quickly and prove technical viability makes access to reliable infrastructure even more critical.

Traditional cloud providers often aren't built with startups in mind. Pricing may be steep or overly complex, resources can be constrained, and it's easy to end up locked into systems that don't scale with your needs. On the other hand, buying your own hardware adds upfront costs and delays, which can be difficult to justify for early-stage teams.

Vast.ai offers another path. **Our market-based cloud GPU rental platform gives startups the flexibility to run powerful workloads without the usual friction.**

Here's a closer look at why startups choose to build with Vast.ai.

### Why Vast.ai Makes Sense for Startups

Startups need infrastructure that keeps up with their pace, budget, and technical demands. The following are some of the reasons Vast.ai hits the mark.

#### 1\. Instant Access to GPUs

With Vast.ai, there are no waitlists or delays. Startups can instantly access powerful GPUs like **H100s**, **A100s**, and **RTX 5090s**, backed by a global network of providers that ensures better availability and built-in redundancy.

Plus, with machines available around the world, startups can deploy GPUs in regions closer to their major user concentrations, reducing network latency for performance-sensitive applications.

#### 2\. Cost-Effective Compute Power

Vast.ai offers significantly cheaper GPU rental compared to traditional cloud providers – **on average 5-6X lower in cost** – making high-performance compute far more accessible.

For bootstrapped startups and solo founders, Vast.ai levels the playing field, enabling smaller teams on a budget to compete in compute-intensive spaces, as well as handle temporary spikes in workload without having to invest in expensive infrastructure.

#### 3\. Flexible Startup-Friendly Billing

Vast.ai's **pay-as-you-go pricing** gives startups the freedom to scale compute resources up or down as needed, with no lock-in contracts. This way, **you only pay for what you use**, and you can quickly adapt to new demands as they arise.

On-demand and interruptible (bid auction) [rental options](https://docs.vast.ai/instances/rental-types) offer even more control – ideal for variable usage patterns and budget-conscious teams. On top of that, real-time cost tracking and detailed resource utilization dashboards help you monitor and optimize performance.

#### 4\. Enterprise-Grade Security

With Vast.ai, **startups can choose the level of trust and security that best fits their needs.** Providers [range](https://vast.ai/compliance) from verified hosts, suitable for general computing needs, to our vetted datacenter partners who maintain third-party compliance certifications (at minimum ISO 27001 certified) and sign comprehensive Data Processing Agreements (DPAs) with Vast.ai.

For sensitive or production workloads, our Secure Cloud tier ensures you’re connecting with partners in certified locations that are current with ISO 27001 and/or Tier 3/4 standards. (You can even filter for it. Look for the blue datacenter label\!)

We've also [completed](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) our SOC 2 Type I certification – part of our ongoing commitment to the highest standards of regulatory compliance.

#### 5\. Developer Control

Vast.ai provides **transparent instance specs** (GPU type, RAM, storage, and more), so startups can choose exactly what fits their workload and budget. Developers can bring their own Docker images, run their own stack, and configure environments to match their specific requirements.

With a low barrier to entry, even early-stage teams without dedicated DevOps support can get up and running quickly. Our selection of pre-configured [templates](https://docs.vast.ai/templates) are like recipes that help users easily set up instances; templates can define the Docker image, environment variables, and launch mode for each new instance.

#### 6\. Ideal for AI Workloads

Vast.ai is well suited for AI startups working on large-scale training, fine-tuning, and inference. Our customizable, on-demand [CPU/GPU clusters](https://vast.ai/clusters) ensure reliable throughput and flexible resource allocations for continuous or large-scale workloads.

With support for large language models (LLMs) and diffusion models, our platform is compatible with all major machine learning (ML) frameworks and offers the flexibility to run custom containers and tooling.

For teams that want to get started quickly, Vast.ai provides **ready-to-use GPU instances pre-configured for frameworks like TensorFlow and PyTorch**, reducing setup time and simplifying deployment.

#### 7\. Built to Experiment

With Vast.ai, **startups can prototype, test, and explore without any long-term commitments** or expensive upfront investment.

For instance, teams can run quick tests and train models on demand, access and evaluate cutting-edge GPUs like the H200 that may otherwise be out of reach, and even spin up high-performance environments on short notice for hackathons and demos.

### The Bottom Line

**Great ideas can come from anywhere, and they deserve the resources to grow.** By making GPU compute power more open and accessible, Vast.ai gives startups and early-stage teams a real shot at achieving their goals.

Built with transparency, accessibility, and security in mind – and aligned with the open-source ethos that many founders value – our platform supports the kind of flexibility and control that your team needs from day one.

Ready to build, test, and scale on your terms? Discover what's possible with Vast.ai\!


Vast.ai offers startups instant, low-cost access to powerful GPUs—perfect for AI, ML, and compute-heavy work—without lock-ins or upfront infrastructure costs.

7 Reasons Why Startups Use Vast.ai


The GeForce RTX 4090 and the A100 both sit at the high end of NVIDIA's GPU lineup, but they're built for very different worlds. 

The **RTX 4090** is a powerful consumer-grade GPU, designed for ultra high-performance gaming, professional creative work, and even some entry-level AI workloads. It's surprisingly capable across a variety of tasks, especially for its price point.

The **A100**, on the other hand, isn't aimed at consumers at all. It's an enterprise-grade GPU built for data centers, research labs, and AI teams running large-scale model training, inference, and simulation workloads – designed to move serious data at serious speed.

In this post, we'll look at how these two machines compare with each other, and why you might choose one over the other based on your specific needs.

### NVIDIA GeForce RTX 4090: A Workhorse with Range

When it launched in September 2022 as the top-tier consumer GPU of its generation, the RTX 4090 quickly became the go-to choice for 4K gaming, 3D rendering, and AI-enhanced creative workflows. 

Built on NVIDIA's **Ada Lovelace** architecture and powered by the **AD102 chip**, the RTX 4090 introduced significant upgrades in ray tracing and Tensor Core acceleration, and delivered notable performance gains with **DLSS 3** in a variety of games.

While it's no longer the flagship GPU of NVIDIA's GeForce line (that title now goes to the RTX 5090, which you can read about [here](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know)), the RTX 4090 definitely still holds its own. It remains one of the most capable and well-rounded GPUs available at its price point for consumers and technical teams alike.

#### Some of the RTX 4090's key specs and features include:

* **24 GB of GDDR6X memory** with **1,008 GB/s bandwidth** on a **384-bit** interface.  
    
* **16,384 CUDA cores** for high-parallel throughput.  
    
* **128 third-gen RT cores** and **512 fourth-gen Tensor cores** for advanced ray tracing and AI workloads.  
    
* **DLSS 3** support for real-time upscaling and performance boosts in many games.  
    
* **450 watts** of thermal design power (TDP), requiring an **850-watt power supply** due to its high processing capabilities.

If you're looking for serious performance in a desktop form factor, the RTX 4090 continues to deliver. It may not be the newest card on the market, but it remains a workhorse that gets the job done across a wide range of applications.

As capable as the RTX 4090 is, it's not built for everything. If your workloads go beyond what a high-end consumer GPU can handle, the A100 just might bring the compute power you need.

### NVIDIA A100: A Powerhouse with Purpose

The A100 is built for scale. Based on NVIDIA's Ampere architecture and powered by the **GA100 chip**, it's designed to tackle massive workloads – like foundational AI model training – thanks to its raw throughput, high-bandwidth memory, and ability to scale across multi-GPU environments. 

When the A100 launched in June 2021, it introduced **third-gen Tensor Cores**, support for **TF32** precision, and **Multi-Instance GPU** (MIG) capabilities – enabling flexible resource partitioning and efficient parallel processing across shared environments.

While some of its core specs might look modest compared to the RTX 4090, the A100 delivers superior performance in AI and data-intensive workloads because it's optimized where it counts: memory bandwidth, interconnects, and specialized compute.

For instance, the A100's memory clock is much lower than the RTX 4090's on paper (roughly 3 Gbps vs. 21 Gbps). However, the A100 uses **HBM2e memory** with a much wider **5,120-bit interface**. This design allows it to deliver around **2 TB/s** of bandwidth – double the RTX 4090 – despite the lower frequency. It's an approach that prioritizes efficiency and scale.

#### To recap, here are a few notable features of the A100:

* **80 GB of HBM2e memory** with about **2 TB/s of bandwidth** (twice that of the RTX 4090).  
    
* **5,120-bit memory bus**, an exceptionally wide interface enabling fast access to memory, reducing bottlenecks in AI and high-performance computing (HPC) workloads.  
    
* **432 third-gen Tensor Cores**, with high-throughput **TF32** performance for AI and robust **FP64** support for HPC workloads.  
    
* **Multi-Instance GPU (MIG)** support, so it can be partitioned into up to 7 isolated GPU instances – ideal for multi-tenant workloads and fine-grained resource allocation.  
    
* **NVLink and NVSwitch** allow up to 16 A100 GPUs to be interconnected at up to **600 GB/s** of bi-directional bandwidth per GPU.


  
In short, the A100 delivers serious performance for large-scale AI and HPC workloads. But if you're comparing it head-to-head with the RTX 4090 on specs alone, here's how the two stack up.

### RTX 4090 vs. A100: Key Specs Compared

For reference, this table highlights some of the features and specs of the RTX 4090 and A100:

| Specification | RTX 4090 | A100 |
| :---- | :---: | ----- |
| **Architecture** | Ada Lovelace | Ampere |
| **VRAM** | 24 GB GDDR6X | 80 GB HBM2e |
| **Memory Clock** | 1313 MHz 21 Gbps | 1512 MHz 3 Gbps |
| **Memory Bus Width** | 384 | 5120 |
| **Bandwidth** | 1.01 TB/s | 1.94 TB/s |
| **Streaming Multiprocessors (SMs)** | 128 | 108 |
| **Tensor Cores** | 512 | 432 |
| **CUDA Cores** | 16,384 | 6,912 |
| **Ray Tracing Cores** | 128 | N/A |
| **Base Clock** | 2235 MHz | 1065 MHz |
| **Boost Clock** | 2520 MHz | 1410 MHz |
| **Multi-Instance GPUs** | No MIG Support | Up to 7 MIGs @ 10GB each |
| **Thermal Design Power (TDP)** | 450W | 300W |
| **Recommended Power Supply** | 850W | 700W |
| **Launch Date** | Sept. 20, 2022 | June 28, 2021 |

As mentioned, specs are just one part of the equation. Ultimately, choosing the right GPU depends on your workload, environment, and priorities. 

### Use Cases: Which GPU Should You Choose?

Here’s how the strengths of each GPU align with different types of use cases.

#### You should consider the RTX 4090 if you prefer:

* **A powerful, desktop-friendly GPU** for high-end gaming, creative workflows, and light AI experimentation.  
* **Excellent performance-per-dollar in single-GPU setups**, particularly for developers and small teams.  
* **A widely available, less costly option** for rendering, video editing, or running small to medium AI models locally.

#### Whereas the A100 may be a better fit if you need:

* **Scalable performance** for training large models, running high-throughput inference, or powering complex simulations.  
* **A data center-optimized GPU** with advanced memory architecture and precision modes tailored for AI and HPC workloads.  
* **Flexible deployment options**, including support for multi-instance workloads and high-speed interconnects in clustered environments.

\------------------------------

At **Vast.ai**, we understand there are other considerations, as well. Access, flexibility, and affordability are just as important, especially when you're experimenting, scaling, or moving fast. 

That's why our market-based cloud GPU rental platform gives you affordable, on-demand access to RTX 4090s, A100s, and much more. With Vast.ai, you can instantly match the right hardware to your workload – without the upfront investment and overhead of managing your own infrastructure.

Try Vast today and discover what you can achieve with compute power that fits your needs and your budget\!

Compare NVIDIA’s RTX 4090 vs A100 GPUs to find out which is better for gaming, AI, HPC, or creative work—and how to access both on Vast.ai.

NVIDIA RTX 4090 vs. A100: Two Powerhouses, Two Purposes


Welcome back to another update from the team at **Vast.ai**\! Over the past two months, we've been hard at work refining our platform and rolling out features that make your experience on Vast faster, smoother, and more powerful. These updates are part of our ongoing effort to support your projects with scalable, flexible infrastructure you can rely on.

We're also pleased to have achieved [**SOC 2 Type I certification**](https://vast.ai/article/vast-ai-achieves-soc-2-type-1-certication) recently, a clear validation of the controls we've put in place to protect customer data and ensure the integrity of our infrastructure. The independent SOC 2 audit confirms that our security practices meet the highest standards of regulatory compliance – giving you added confidence as you build on our platform.

Another exciting update: [**H200 GPUs**](https://vast.ai/article/nvidia-h100-vs-h200-two-hopper-based-heavyweights) are now available on Vast\! If you're pushing the limits of high-performance compute, these machines are ready to meet the challenge.

Let's take a look at what else is new\!

### NVIDIA Cloud GPU Updates

Whether it's expanded hardware options, new templates, or updated guides, our latest improvements give you a more powerful, customizable environment for every stage of your workflow.

#### Highlights of the latest enhancements:

* **New Templates Added**

  * [***Dia 1.6b TTS***](https://cloud.vast.ai/?ref_id=62897&template_id=6ba071acb5f590913c5eaf6c2dc3d0db) – Convert text to lifelike speech using the open-source text-to-speech model from Nari Labs (full guide [here](https://docs.vast.ai/tts-with-nari-labs-dia))..  
  * [***Tari Universe***](https://cloud.vast.ai/?ref_id=62897&template_id=338f9628e43e586a87d37d18c719291d) – Run crypto mining workloads on Vast via desktop application (note: cryptocurrency funding is required).  
  * [***Pinokio Desktop***](https://cloud.vast.ai/?ref_id=62897&template_id=06c4818322f002904b324165b9dba52f) – Deploy AI applications with verified scripts from trusted publishers, without manual setup.

* **Introducing Local Volumes (CLI-only)**

  * Local Volumes are **persistent Docker volumes** that keep your data intact across restarts, new instances, and failures – so your storage survives instance deletion.  
  * **Benefits:** Share one volume across multiple containers on the same machine, and spin up fresh instances without data-transfer overhead. 

* **Upgraded Instance Portal**

  * For Vast.ai instances using [recommended templates](https://cloud.vast.ai/templates/), instances now launch via **secure Cloudflare tunnels** by default.  
  * Create a user-friendly **custom tunnel URL** that points to a port on your instance.  
  * **Live logs** and **live stats** (CPU, Memory, and Disk) for your instance are now viewable.

* **Vast Teams Improvements**

  * **Smoother invites** (fast and easy team creation), **standalone teams** (no need to convert your individual account), and **context switching** (seamlessly switch between Individual and Team views).  
  * **Access Keys:** Share API Keys and view Team-wide Session Keys as owner.  
  * Various stability and usability improvements.

For more information and helpful links, check out the full change log for our April–May 2025 updates at the bottom of this page.

### Our Commitment

Whether you're trying out the H200, deploying AI applications with just a few clicks, launching a new template, or exploring our latest platform features, we're here to make your experience on **Vast.ai** as smooth and flexible as possible. However you use Vast, our goal remains the same: to deliver reliable, cost-effective compute whenever you need it.

Got questions? Help is always within reach\! Contact our customer support team at [support@vast.ai](mailto:support@vast.ai) anytime, or join our Discord server for additional resources and community support.

### Change Log

#### New Templates

* **Dia 1.6b TTS** – Transform text into ultra-realistic speech using Dia, the open-source text-to-speech model from Nari Labs.  
  * [Full Guide](https://docs.vast.ai/tts-with-nari-labs-dia)  
  * [Launch an Instance with the Dia Template](https://cloud.vast.ai/?ref_id=62897&template_id=6ba071acb5f590913c5eaf6c2dc3d0db)  
* **Tari Universe** – Desktop Crypto mining application you can run on Vast. IMPORTANT: You must fund your Vast account with cryptocurrency if you want to use the service for mining.  
  * [Launch an Instance with the Tari Universe Template](https://cloud.vast.ai/?ref_id=62897&template_id=338f9628e43e586a87d37d18c719291d)  
* **Pinokio Desktop** – Access an extensive library of verified scripts from trusted publishers and deploy AI applications effortlessly without manual setup.  
  * [Launch an Instance on Vast with the Pinokio Desktop Template](https://cloud.vast.ai/?ref_id=62897&template_id=06c4818322f002904b324165b9dba52f)

####  Introducing Local Volumes (CLI only – UI coming soon\!)

**What are Local Volumes?**  
Persistent Docker volumes to keep your data intact across restarts, new instances, and failures.

**Key client benefits:**  
• Persistent storage survives instance deletion.  
• Share one volume across multiple containers on the same machine.  
• Spin up fresh instances on the same machine without data-transfer overhead.

**Get started:**  
• Search for available Volumes via CLI:  vastai search volumes  
• General docs: [https://docs.vast.ai/volumes](https://docs.vast.ai/volumes) 

*Note: Currently, clients can only rent Local Volumes through CLI. UI support and Network Volumes are coming soon.*

#### Teams Improvements

We’ve improved the Vast Teams experience for user access management and collaboration:

* **Smoother Invites:** Get team members connected on Vast quickly.  
* **Standalone Teams:** Create Teams without converting your individual account.  
* **Context Switching:** Seamlessly switch between individual and Team views.  
* **Access Keys:** Share API Keys and view Team-wide Session Keys as owner.  
* **Team Basics:** Transfer credits to/from your team, transfer ownership, and other basics.  
* **Bug Fixes:** Various stability and usability improvements.

[Create a team](https://cloud.vast.ai/manage-members/) and let us know your feedback\!

#### Instance Portal upgrade

We've also rolled out a major upgrade to the **Instance Portal** that appears on Vast.ai instances using [recommended templates](https://cloud.vast.ai/templates/).

**What’s New:**

* **Secure Tunnel URLs:** Instances now launch via secure Cloudflare tunnels by default, eliminating the “not secure” screen.  
* **Custom Tunnel URLs:** Create a user-friendly URL that points to a port on your Instance. Optionally use a named tunnel to use a custom domain.  
* **Live Logs:** See a live stream of log updates on your Instance.  
* **Live Stats:** See live CPU, Memory, and Disk stats for your Instance.

Deploy a recommended template instance or check out the [updated Instance Portal guide](https://docs.vast.ai/instance-portal) to learn more\!

#### Opt in to the new Instances UI

We've rolled out an (optional) UI update as part of our ongoing UI refresh:

* Cleaner instance card design  
* Instance data fetching improvements  
* Various instance UI bug fixes

Opt-in: Go to [Settings \> Early Access](https://cloud.vast.ai/account/) and toggle it on. You can switch back anytime.

#### SOC2 Type I Certification complete.

#### H200s available now\!

Vast.ai adds H200 GPUs, SOC 2 Type I cert, new templates, persistent storage, and better teams & instance tools for faster, secure cloud compute.

April-May 2025 Product Updates


## Introduction

The recent advancements in AI for video generation have been very fast paced. Models now create much more realistic videos than before, with multiple different use cases. One such model is [HunyuanVideo](https://huggingface.co/tencent/HunyuanVideo), a model known for its impressive outputs—and tackle the practical challenges of running it efficiently on high-memory GPUs like the A100 or H100.

HunyuanVideo is Tencent's state-of-the-art text-to-video generation model that rivals or surpasses leading closed-source alternatives. As the largest open-source video generation model with over 13 billion parameters, HunyuanVideo represents a significant breakthrough in AI-powered video creation.

We also have a [notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/LznBGR/aec37016e507d00895435abda7f0a3bc6e29625a/files/hunyuan_video_vast.ipynb) to follow along once you deploy the Vast Instance.

### Key Innovations

- **Unified Architecture**: Employs a "Dual-stream to Single-stream" transformer design that effectively handles both image and video generation.
- **MLLM Text Encoder**: Utilizes a multimodal large language model for superior text-to-visual alignment compared to previous text-encoder architectures.
- **Efficient Compression**: Implements causal 3D VAE for efficient spatial-temporal compression to enable training at higher resolution.
- **Automatic Prompt Rewriting**: Features intelligent prompt enhancement to optimize user-given prompts tailored for the model's consumption.

### What You’ll Learn

In this guide, we will:

1. Set up a custom Docker template for HunyuanVideo on Vast.ai.
2. Launch the Docker template on a suitable GPU instance.
3. Download pretrained models and generate high-quality videos from text prompts.

Let’s get started with HunyuanVideo!

---

## Renting an Instance on Vast

### Create a Custom Template

Tencent maintains a custom Docker image for HunyuanVideo: `hunyuanvideo/hunyuanvideo:cuda_12`. To run the model on Vast, we'll create a custom template using this image.

Follow these steps:

1. Ensure you have a [Vast.ai](https://vast.ai/) account.
2. Navigate to the **Templates** page on the [Vast Console](https://cloud.vast.ai/templates/).
3. Find the existing template named **PyTorch (CuDNN Runtime)** and click **Edit**.
4. Rename the template to **HunyuanVideo**.
5. Replace the Docker image in the **Image Path:Tag** field with `hunyuanvideo/hunyuanvideo:cuda_12`.
6. Set the **On-start Script** to:
   ```
   git clone https://github.com/tencent/HunyuanVideo
   ```
   This ensures the HunyuanVideo repository is downloaded on instance startup.
7. Allocate sufficient disk space, for example, **80 GB**.
8. Click **Save & Use** to save your template and prepare to deploy it.

---

### Selecting an Instance

HunyuanVideo requires a GPU with **at least 80GB VRAM** to run smoothly. Available GPUs meeting this requirement include Nvidia’s A100 and H100 cards.

To select an instance:

1. Filter Vast instances based on:
   - **Instance Type**: A100 or H100
   - **Number of GPUs**: 1
   - **VRAM**: ≥ 80GB
2. Choose and rent a suitable instance.
3. Install the Vast TLS certificate in your browser, enabling secure access to your Jupyter notebook server: [Installing TLS Certificate Guide](https://docs.vast.ai/instances/jupyter#1SmCz).
4. Open the Jupyter server via https://cloud.vast.ai/instances/, clicking **Open** for your rented instance.
5. Upload this notebook into the directory `/workspace/HunyuanVideo/` on the server.

---

## Downloading the Model Weights

Before generating videos, we need to download pretrained model weights from Hugging Face. These include the video model weights and the text encoders.

Run the following commands inside your instance:

```bash
huggingface-cli download tencent/HunyuanVideo --local-dir ./ckpts
huggingface-cli download xtuner/llava-llama-3-8b-v1_1-transformers --local-dir ./ckpts/llava-llama-3-8b-v1_1-transformers
python hyvideo/utils/preprocess_text_encoder_tokenizer_utils.py --input_dir ckpts/llava-llama-3-8b-v1_1-transformers --output_dir ckpts/text_encoder
huggingface-cli download openai/clip-vit-large-patch14 --local-dir ./ckpts/text_encoder_2
```

For a more in-depth discussion of the checkpoints, refer to [Tencent’s checkpoint README](https://github.com/Tencent/HunyuanVideo/blob/main/ckpts/README.md).

---

## Generating Videos using HunyuanVideo

### Displaying Videos in Jupyter

To conveniently view generated videos within the Jupyter notebook, let’s define a helper function using `IPython.display.Video`:

```python
from IPython.display import Video

def show_video(video_path, width=640, height=360, embed=True):
    """
    Display a video in a Jupyter notebook.

    Parameters:
    -----------
    video_path : str
        Path to the video file (local file or URL)
    width : int, optional
        Width of the video player in pixels
    height : int, optional
        Height of the video player in pixels
    embed : bool, optional
        Whether to embed the video in the notebook (True)
        or just link to it (False)

    Returns:
    --------
    IPython.display.Video
        Video display object
    """
    return Video(video_path, width=width, height=height, embed=embed)
```

---

### Generating Your First Video: Cat on the Grass

Let’s generate a realistic video of a cat walking on the grass using the example prompt provided by Tencent.

Run the following command in your terminal:

```bash
python sample_video.py \
    --video-size 720 1280 \
    --video-length 129 \
    --infer-steps 50 \
    --prompt "A cat walks on the grass, realistic style." \
    --flow-reverse \
    --use-cpu-offload \
    --save-path ./results
```

This command will create a video approximately 129 frames long (roughly 4-5 seconds at standard frame rates) at 720x1280 resolution.

Once the generation finishes, locate the video file inside the `./results` directory.

To display it in your notebook, run:

```python
show_video("./results/[your_video_filename].mp4")
```

Replace `[your_video_filename].mp4` with the actual file name.

You should see a high-quality, realistically animated cat walking on grass, showcasing HunyuanVideo’s remarkable detail in animal motion and textures.

---

### Creating a Video of an Astronaut Walking on the Moon

Next, try generating a completely different scene:

```bash
python sample_video.py \
    --video-size 720 1280 \
    --video-length 129 \
    --infer-steps 50 \
    --prompt "An astronaut walks across the moon, realistic style." \
    --flow-reverse \
    --use-cpu-offload \
    --save-path ./results
```

After the video is generated, display it in your notebook the same way:

```python
show_video("./results/[your_astronaut_video].mp4")
```

The resulting video will demonstrate the model’s versatility at rendering vastly different environments and characters, from furry animals to astronauts in space suits.

---

## Conclusion and Next Steps

You've now successfully generated your first videos using HunyuanVideo on a Vast-powered cloud instance. This powerful model unlocks creative possibilities in AI-driven video generation with:

- **Flexible Resolution Options**: Try different `--video-size` values (e.g., 1280×720, 960×960) to optimize for your needs.
- **Adjustable Quality Settings**: Modify `--infer-steps` (default 50) to trade off video generation quality and speed.
- **Creative Control**: Experiment with `--embedded-cfg-scale` (default 6.0) to balance prompt fidelity and creative variance.
- **Deterministic Outputs**: Set `--seed` to reproduce favorite generated videos reliably.

Deploying large models like HunyuanVideo can be resource-intensive but is now accessible with cloud platforms like Vast, giving you access to top-tier GPUs without upfront hardware costs.

With these foundations, you’re ready to explore more sophisticated prompt engineering, fine-tune generation parameters, or even integrate HunyuanVideo into multimedia projects!

---

**Resources:**

- [HunyuanVideo GitHub Repository](https://github.com/tencent/HunyuanVideo)
- [Vast.ai Documentation](https://docs.vast.ai/)
- [HunyuanVideo Checkpoints README](https://github.com/Tencent/HunyuanVideo/blob/main/ckpts/README.md)

Learn how to run Tencent’s HunyuanVideo text-to-video model on Vast.ai using A100/H100 GPUs to generate realistic AI-powered videos from prompts.

Blog

June 2025 Product Update

A Decade Later: Revisiting the Brain as a Universal Learning Machine

Efficiently Serving Multiple Machine Learning Models with Lorax and vLLM on Vast.ai

7 Reasons Why Startups Use Vast.ai

NVIDIA RTX 4090 vs. A100: Two Powerhouses, Two Purposes

April-May 2025 Product Updates

Text to Video Generation Using HunyuanVideo on Vast

Blog

June 2025 Product Update

A Decade Later: Revisiting the Brain as a Universal Learning Machine

Efficiently Serving Multiple Machine Learning Models with Lorax and vLLM on Vast.ai

7 Reasons Why Startups Use Vast.ai

NVIDIA RTX 4090 vs. A100: Two Powerhouses, Two Purposes

April-May 2025 Product Updates

Text to Video Generation Using HunyuanVideo on Vast

Subscribe for our product updates.