
# Announcing Virtual Machine Rental on Vast.ai

Have you ever thought about what you could accomplish with more flexibility in your GPU rentals? At Vast.ai, we're always looking for ways to innovate and improve our platform, and today we're excited to announce our latest feature: [virtual machine (VM) rental](https://vast.ai/docs/instance-setup/virtual-machines).

We now offer full VMs alongside Docker-based instance rentals. This new offering is designed to support larger customers and more diverse workflows via GPU remote desktop.

Here's everything you need to know to get started!

## Virtual Machine Rental: Expanding the Possibilities

The new VM rental option will help meet the demand for use cases that Docker can't easily support. **VMs also offer features not available in Docker, such as support for init managers and process tracing.**

### Key Benefits of Virtual Machine Support Include:

- **Enhanced Flexibility:** Users can run low level debugging and performance profiling utilities and are not constrained by container environments.
- **Kernel-space Customizations:** Developers gain granular control over their systems, enabling custom driver installations, kernel tweaks, and integration of specialized frameworks.
- **Faster Iteration:** Rather than pre-building and pushing custom Docker images, users can run Docker inside the instance.
- **Improved Security:** Virtual Machines offers an improved  security posture for sensitive workloads.

Host machines can now tap into these advantages, allowing them to attract more renters and capitalize on the increased need for GPU-powered tasks beyond Docker’s capabilities.

### Important VM Differences to Know:

VMs do come with some trade-offs compared to Docker-based instances. For example, they have slower creation and boot times, higher disk overhead, and users will have to deal with a smaller selection of machines and lesser availability of preconfigured templates. In addition, only SSH launch mode is currently supported.

Notably, the on-start script field allows users to specify a script to run on instance start. Unlike in Docker-based instances, the interpreter must be specified by a shebang.

Another difference is that **the cloud copy utility for VMs only allows migrating the entire VM instead of allowing individual folders to be copied.**

Despite these drawbacks, VMs open up new opportunities for both renters and hosts, offering greater flexibility, broader use cases, and the potential for higher earnings.

### Empowering Renters, Rewarding Hosts

One thing that sets us apart from other providers is our focus on cost-efficiency. We strive to keep rental costs low for users across the board – and in fact, Vast.ai GPU rentals are ~3-5X cheaper than current alternatives.

At the same time, we want to provide hosts with the opportunity to earn more.

As our COO, Travis Cannell, explains, "We've always believed that democratizing GPU access means offering maximum flexibility. Virtual Machine support was the natural next step, providing more control, ease of use, and enabling researchers to build and iterate."

With the opportunity for both renters and hosts to benefit, it's equally important to ensure that setup is seamless and operations remain secure. Here's how we make that happen.

## VMs on Vast: Simple Setup and Strong Security

Getting started with VM rental on Vast.ai is simple. **Our hosting software automatically tests host machines for VM compatibility and enables the feature on any machine where it's supported.** (There is no requirement to be VM compatible, however.)

For new host machines, these tests will be run upon installation. For machines configured before the VM feature release, testing for VM compatibility will happen at a time when the machine is unoccupied.

And that's it! Host machines that don't have VM support enabled will simply be hidden from search results when VM-based templates are selected.

Privacy and security are built into the system, as well. Hosts and users cannot view someone else's VM, and no one has access to see what any other VM is doing. Users can rent VMs with peace of mind knowing that their virtual environment is private and secure.

## Getting Started: Creating a VM

Ready to spin up your first VM? It's as easy as selecting the right template!

1.  Make sure you have an SSH key in your account (guide [here](https://vast.ai/docs/instance-setup/ssh)).
2.  Select the **Ubuntu 22.04 VM template** in the interface or [click here](https://cloud.vast.ai/?ref_id=108031&template_id=e2f271ce754aeef772181fbf9c82a354) to load it.
3.  Adjust the disk storage needed and then rent a GPU!

Note: SSH keys cannot be edited on running VMs, and SSH access is currently the only method preconfigured on VM instances. Also, when you select the correct template, it will automatically filter for machines that support VMs and will launch a VM when you rent with that template.

For more detailed information about VM template customization, see our Docs page [here](https://vast.ai/docs/instance-setup/virtual-machines).

## Power Up with VM Rental on Vast.ai

With our new virtual machine rental feature, we're opening up a world of possibilities for both hosts and renters. Whether you need the flexibility of GPU remote desktops for complex workflows or you're looking to maximize your earning potential as a host, Vast.ai has you covered. We're committed to giving you the tools to succeed – easily, affordably, and securely.

Ready to see what you can accomplish? Experience the [power of VM rental](https://vast.ai/docs/instance-setup/virtual-machines) with Vast.ai today!


At Vast.ai, we're always looking for ways to innovate and improve our platform, and today we're excited to announce our latest feature: virtual machine (VM) rental. 

Announcing Virtual Machine Rental on Vast.ai


# Google Colab vs. Vast: Which Platform Is Best for Your Projects?

If you’ve ever dabbled in machine learning, data science, or coding in general, chances are you’ve heard of Google Colaboratory – better known as **Google Colab**. But what exactly is it, what are its limitations, and how can you make the best use of it?

Let's take a closer look at these questions and more…

## What Is Google Colab?

In a nutshell, **Google Colab** is a free, browser-based platform that lets you run and write code in a Jupyter notebook environment without having to set up anything locally on your machine. It's widely used for Python-based projects, especially in machine learning and data analysis.

Colab offers an easy starting point for experiments, since you can use the Jupyter notebook interface to write and execute code, visualize results, and keep everything neatly organized in one place. This type of interactive programming allows you to experiment with code in real time, making it ideal for data science and research projects that require immediate feedback and iteration.

### Some other benefits include:

- **Accessibility** – The browser-based platform allows you to access your projects from anywhere.
- **Integration with Google Drive** – Save your notebooks and datasets directly to your Google Drive.
- **Collaboration** – Easily share your notebooks with others for real-time collaboration across teams.
- **Wealth of Resources** – Plenty of tutorials, sample notebooks, and community support are available in the Colab ecosystem, making it easier for you to learn and troubleshoot along the way.

These features make Google Colab a convenient tool for a wide range of users. However, it does have some significant limitations that may prompt you to look for other solutions – such as running a Jupyter instance right here on **Vast** and uploading your notebook directly.

## Why Choose Vast Over Google Colab?

While Google Colab is helpful for smaller-scale projects or initial experimentation, **Vast's** cloud GPU platform offers a much more flexible, powerful, and reliable experience through our Jupyter integration. Here’s why you should consider using Vast instead.

### No Resource Limits

With Google Colab, free-tier users face dynamic usage limits with idle timeout periods and resource caps that sometimes fluctuate. The paid plans have their own set of restrictions as well. For example, notebooks can run for up to 12 hours on a free plan, and the highest-tier paid plan supports continuous code execution for at most 24 hours.

But with **Vast**, there are no imposed time limits, so your projects can run uninterrupted for as long as you need. Whether your project takes hours or days, you won't be cut off or forced to restart.

> **Note**: Google Colab does include the option to pay for a dedicated virtual machine (VM) to get around usage limits, but Vast's pricing is extremely competitive as we are the market leader in low-cost cloud GPU rental.

### Customizable GPU Power for Every Project

Google Colab offers different VMs based on the plan you choose. The free tier only allows access to VMs with a standard system memory profile, restricting the compute power you can use. Paid versions of Colab provide access to machines with a higher system memory profile, but you still don't get to select specific GPUs unless you opt to manually pay for a dedicated machine.

**Vast** gives you access right from the start to powerful and customizable hardware options that are also highly affordable. You can select the exact GPU setup you need, whether you're running small tasks or large-scale machine learning models. For high-end performance, Vast supports multi-GPU configurations, making it easy to scale up your projects without hitting resource bottlenecks.

### Simplicity and Reliability

**Vast** simplifies the entire process by running your Jupyter notebook directly in the cloud with no need for SSH or complex setups. Jupyter by itself is much more reliable than Google Colab and doesn't require setting up SSH keys – you can always open a terminal right inside the browser.

And while Google Colab does facilitate sharing and collaboration, **Vast** offers the same, with the added benefit of running on a more stable platform without timeouts or interruptions. Team members can collaborate in real time, sharing notebooks and results without worrying about session limits.

## Making the Switch to Vast

If you've been using Google Colab for simple notebooks and want to step up your workflow, switching to Vast is straightforward. Simply download your notebook from Colab as a `.ipynb` file, run a Vast Jupyter instance with the recommended [PyTorch template](https://cloud.vast.ai/?ref_id=43484&template_id=f5540ef1a1398b8499546edb53dae704), and upload your notebook into Jupyter directly.

For a step-by-step guide on running Jupyter notebooks on Vast, check out our detailed doc [here](https://vast.ai/docs/use-cases/google-colab).

While Google Colab is a useful starting point for basic tasks, it often falls short when you're dealing with more complex workflows. **Vast** provides a more robust and customizable solution through its Jupyter notebook integration, with greater control, higher performance, and fewer interruptions.

With **Vast**, you'll enjoy the freedom to push your projects even further. **Explore the possibilities today!**


Google Colab is a free, browser-based platform that lets you run and write code in a Jupyter notebook environment without having to set up anything locally on your machine.

Google Colab Explained: Simplifying Your Workflow with Cloud Tools


# Serving vLLM Embeddings on Vast.ai

## Background:

[vLLM](https://github.com/vllm-project/vllm) is a developer-friendly framework for serving large language models. It is very fast and focused on high throughput serving multiple simultaneous calls for production use cases. `vLLM` is now more flexible than ever as it also supports embedding models. This brings vLLM's dynamic batching and Paged Attention to embedding models for much faster throughput, all from the docker image that developers are used to.

This guide will show you how to setup vLLM to serve embedding models on Vast.

## Setup and Querying

First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

To serve an embedding model, we'll want a machine with a static IP address + ports to forward. We only will need a single GPU, which can have a small amount of RAM since embedding models are small. We'll prefer a modern GPU so we will add a filter for a compute cap of more than 800.

```bash
vastai search offers 'compute_cap >= 800 gpu_ram >= 24 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.4'
```

## Deploying the Image:

The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below.

### Hosting a Single Embedding Model:

For now, we'll host just one embedding model.
The easiest way to deploy a single model on this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below.

We particularly need `v2` so that we use the correct version of the api, `--port 8000` so it serves on the correct model, and `--model-id intfloat/e5-mistral-7b-instruct` to serve the correct model.

```bash
vastai create instance <instance-id> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 40 --args --model intfloat/e5-mistral-7b-instruct
```

## Connecting and Testing:

To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/serve_vllm_embeddings_instance_view.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/serve_vllm_embeddings_ip_address_view.png)

## Connecting with the python requests library

```
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'intfloat/e5-mistral-7b-instruct',
    "input": ["San francisco is a"],
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/embeddings', headers=headers, json=json_data)
print(response.content)
```

## Connecting with the OpenAI SDK

The OpenAI SDK also allows for generating embeddings. We can use it as well with the following code.

```bash
pip install openai
```

We will copy over the IP address and the port into the cell below.

```python

from openai import OpenAI

# Modify OpenAI's API key and API base to use Infinity's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
model = "intfloat/e5-mistral-7b-instruct"
embeddings = client.embeddings.create(model=model, input="San francisco is a").data[0].embedding
print("Embeddings:")
print(embeddings)
```

Now we can integrate our embedding server into apps that use the OpenAI SDK.

## Next Steps:

Embeddings are an important part of GenAI applications along with running inference of Generative Models. Now with vLLM, you can run them with the same docker image on Vast for ultimate flexibility.


Serving vLLM Embeddings on Vast.ai


When it comes to high-performance computing (HPC), the NVIDIA H100 is among the best of the best GPUs on the market. Its predecessor, the NVIDIA A100, is also a very impressive GPU in its own right. From enterprise to exascale, these GPUs are enabling groundbreaking advancements in everything from AI research to complex scientific simulations.

Today we're taking a look at how the H100 and A100 compare with each other. Depending on your use case, one or the other might be better suited for your needs and preferences. So let's dive in!

## Introduction to the NVIDIA H100 and A100

As mentioned, the H100 GPU is the successor to the A100. It was specifically optimized for machine learning and designed to [deliver](https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/) an "order-of-magnitude performance leap for large-scale AI and HPC" over the A100, with "substantial improvements in architectural efficiency."

Based on the innovative Hopper architecture, the H100 can speed up large language models (LLMs) by 30X over the A100, according to NVIDIA. It features fourth-gen Tensor Cores and – unlike the A100 – a dedicated Transformer Engine to solve trillion-parameter language models.

Whether it's used in smaller enterprise systems or massive, unified GPU clusters, the H100 delivers efficient scalability thanks to its combination of fourth-gen NVLink (with 900 GB/s of GPU-to-GPU interconnect), NDR Quantum-2 InfiniBand networking (speeding up communication by every GPU across nodes), and PCIe Gen5. With superior FP64 and FP8 performance, the H100 is ideal for next-gen AI workloads.

The A100 GPU is known for its versatility, however. Built on the Ampere architecture, it was groundbreaking when released in 2020, providing up to 20X higher performance over the prior NVIDIA Volta generation. It can efficiently scale up or be partitioned using Multi-Instance GPU (MIG) into seven isolated GPU instances, just like the H100, which is ideal for shifting workload demands.

In short, the A100 excels in both training and inference tasks, and while it may not offer the sheer power of the H100, its robust FP64 performance makes it a great fit for scientific simulations and data analytics.

## Key Differences: H100 vs. A100

The H100 has an updated chip design and various other factors that distinguish it from the A100. The following are a few that are important to consider:

### Memory Bandwidth

The A100 has a memory bandwidth of about 2 TB/s, which is plenty for most use cases; however, the H100 offers an impressive 3.35 TB/s, making it ideal for workloads where data transfer bottlenecks may occur.

### CUDA Cores and Tensor Cores

A higher number of CUDA and Tensor Cores means a GPU can perform more parallel computations simultaneously. With 14,592 CUDA Cores and 456 Tensor Cores, the H100 outperforms the A100’s 6,912 CUDA Cores and 432 Tensor Cores, delivering superior performance in applications that can leverage increased parallelism.

### Confidential Computing

The H100's Hopper architecture introduced NVIDIA's new confidential computing environment. It protects data-in-use by performing computation in a trusted execution environment (TEE) that is both hardware-based and attested. This means you essentially have absolute security for use cases where security, privacy, and regulatory compliance are paramount.

### Tensor Memory Accelerator (TMA)

Another feature of the H100's Hopper architecture is the TMA unit, which can transfer large blocks of data efficiently between global memory and shared memory. It supports asynchronous copies between thread blocks in a cluster as well, optimizing data movement and memory management for tensor operations. Ultimately, this enhances the efficiency and speed of large-scale AI and deep learning tasks.

### MIG Capability

The Multi-Instance GPU (MIG) feature is present in both the H100 and A100, providing flexible workload management so you can run multiple LLMs in parallel, for instance. The H100's MIG capability is more robust, however, with better resource allocation and more memory and compute power to split.

## Performance Benchmarks

According to NVIDIA's own benchmarks, the H100 offers up to 30X better inference performance than the A100. In another test where the A100 and H100 were used in clusters to train two large language models (LLMs), the H100 demonstrated remarkable speed improvements. The H100's new NVLink Switch System provided an even bigger boost, enabling the H100 cluster to train the models up to nine times faster than the A100 cluster:

![H100 vs A100 Speed Comparison](/uploads/Speed-up-over-A100.png)

## Choosing the Right GPU

So what does all this mean for you? The best GPU for your needs will depend on numerous factors. Here's a quick outline of the various features and specs of the NVIDIA A100 and H100:

<table class="w-full border-collapse border border-gray-300">
  <thead>
    <tr class="bg-gray-100">
      <th class="border border-gray-300 px-4 py-2 text-left font-semibold">Feature</th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">A100</th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">H100</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">GPU Architecture</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Ampere</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Hopper</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">GPU Memory</td>
      <td class="border border-gray-300 px-4 py-2 text-center">40 or 80 GB HBM2e</td>
      <td class="border border-gray-300 px-4 py-2 text-center">80 MB HBM3</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">GPU Memory Bandwidth</td>
      <td class="border border-gray-300 px-4 py-2 text-center">1.6 to 2 TB/s</td>
      <td class="border border-gray-300 px-4 py-2 text-center">3.35 TB/s</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">CUDA Cores</td>
      <td class="border border-gray-300 px-4 py-2 text-center">6912</td>
      <td class="border border-gray-300 px-4 py-2 text-center">14,592</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">FP64 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">9.7</td>
      <td class="border border-gray-300 px-4 py-2 text-center">33.5</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">FP32 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">19.6</td>
      <td class="border border-gray-300 px-4 py-2 text-center">67</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">TF32 Tensor Core Flops*</td>
      <td class="border border-gray-300 px-4 py-2 text-center">156 | 312</td>
      <td class="border border-gray-300 px-4 py-2 text-center">378 | 756</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">FP16 Tensor Core Flops*</td>
      <td class="border border-gray-300 px-4 py-2 text-center">312 | 624</td>
      <td class="border border-gray-300 px-4 py-2 text-center">756 | 1513</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">FP8 Tensor Core TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">N/A</td>
      <td class="border border-gray-300 px-4 py-2 text-center">3958</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">Peak INT8 TOPS*</td>
      <td class="border border-gray-300 px-4 py-2 text-center">624 | 1248</td>
      <td class="border border-gray-300 px-4 py-2 text-center">3958</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">Media Engine</td>
      <td class="border border-gray-300 px-4 py-2 text-center">0 NVENC<br>5 NVDEC<br>5 NVJPEG</td>
      <td class="border border-gray-300 px-4 py-2 text-center">0 NVENC<br>7 NVDEC<br>7 NVJPEG</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">L2 Cache</td>
      <td class="border border-gray-300 px-4 py-2 text-center">40 MB</td>
      <td class="border border-gray-300 px-4 py-2 text-center">50 MB</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">Power</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Up to 400 W</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Up to 700 W</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">Form Factor</td>
      <td class="border border-gray-300 px-4 py-2 text-center">SXM4 - 8-way HGX</td>
      <td class="border border-gray-300 px-4 py-2 text-center">SXM5 - 8-way HGX</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">Interconnect</td>
      <td class="border border-gray-300 px-4 py-2 text-center">PCIe 4.0 x16</td>
      <td class="border border-gray-300 px-4 py-2 text-center">PCIe 5.0 x16</td>
    </tr>
  </tbody>
</table>

\*Without and with structured sparsity.

---

## Use Cases

When it comes to use cases, you may want to consider some general guidelines.

### The A100 is well suited for:

-   Versatile AI workloads including training and inference.
-   Applications needing a balance of performance and cost-efficiency.
-   Scenarios where ultra-high precision isn't the priority.

### The H100 is ideal for:

-   Cutting-edge AI training, large-scale simulations, and other HPC workloads.
-   Environments where computational speed and memory bandwidth are critical.
-   Future-proofing investments in AI infrastructure.

---

When deciding between the A100 and H100, consider your specific workload requirements. If you need top-tier double-precision performance and superior memory bandwidth, or you're dealing with next-gen HPC at datacenter scale and trillion-parameter AI, the H100 is the clear winner. For a more versatile and cost-effective solution that still delivers powerful AI performance, the A100 is a solid choice.

At Vast.ai, we understand the importance of finding the right GPU for your needs as well as your budget. Our cloud GPU rental marketplace offers a variety of machines at the lowest prices possible, saving you 5-6X on GPU compute. Whether you prefer on-demand pricing or interruptible instances through spot auction-based pricing, we've got you covered.

We invite you to discover what Vast can do for you and accelerate your projects with the right GPU today!


Explore the differences between NVIDIA's H100 and A100 GPUs, from architectural advancements to performance benchmarks.

H100 vs A100: Comparing Two Powerhouse GPUs


# Serving Online Inference with LMDeploy on Vast.ai

## Background
LMDeploy is an open-source framework for Large Language Model inference. It is particularly good at high-throughput serving for multi-user or high-load use cases and is one of the most popular serving frameworks today.

LMDeploy provides an OpenAI-compatible server, which means that it can be integrated into applications that use the OpenAI API. This makes it easy to switch from using OpenAI's services to running your own models on more affordable compute.

This guide will show you how to set up LMDeploy to serve an LLM on Vast. We reference a notebook that you can use [here](https://bitbucket.org/jsbcannell/workspace/snippets/EdjpXL#file-serving_lmdeploy_on_vast.json).

### Performance Benefits:
LMDeploy is specifically focused on maintaining throughput while under higher loads. They include [benchmarks](https://github.com/InternLM/lmdeploy?tab=readme-ov-file#performance) where they compare LMDeploy to vLLM on throughput.
![Benchmark](/uploads/lmdeploy_benchmarks.png)
Here we can see LMDeploy's superior performance in the amount of requests per second that it can handle. This is important for reducing the costs of your deployment, and handling traffic spikes more smoothly.

Combining the performance benefits of LMDeploy with accessing affordable compute on Vast enables developers to ship AI applications quickly, affordably, and with low latency.

## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving an LLM, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU with decent RAM since we're going to serve a single small model. We'll also search for cuda drivers 12.0 or higher for stability. We will query the vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.0'
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 
```bash
vastai create instance <instance-id> \
  --image openmmlab/lmdeploy:latest \
  --env '-p 8000:8000' \
  --disk 40 \
  --entrypoint "lmdeploy serve api_server internlm/internlm2_5-7b-chat --model-name internlm/internlm2_5-7b-chat --server-port 8000" \
```

This command sets up LMDeploy to serve the `internlm/internlm2_5-7b-chat` model on internal port 8000.

## Connecting and Testing
To connect to your instance, you'll need to get the IP address and port number. Once your instance is done setting up, you should see something like this in the Vast.ai interface:
![Instance_view](/uploads/instance_view_lmdeploy.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_lmdeploy.png)

We will copy over the IP address and the port into the cell below.

``` bash
curl -X POST http://<Instance-IP-Address>:<Port>/v1/completions \
     -H "Content-Type: application/json" \
     -d '{"model": "internlm/internlm2_5-7b-chat", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

In the [notebook](https://bitbucket.org/jsbcannell/workspace/snippets/EdjpXL#file-serving_lmdeploy_on_vast.json) we include ways to call this model with requests or OpenAI

### Running with OpenAI
We can interact with this model via the OpenAI SDK as well:
``` bash
pip install openai
```

``` python
from openai import OpenAI

# Modify OpenAI's API key and API base to use lmDeploy 's's API server.
openai_api_key = "PASTE_YOUR_OPEN_API_KEY"

openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
completion = client.completions.create(model="internlm/internlm2_5-7b-chat",
                                      prompt="Hello, how are you?",
                                      max_tokens=50)
print("Completion result:", completion)
```

Now you're all set up with super fast inference!

LMDeploy is an open-source framework for Large Language Model inference.

Serving Online Inference with LMDeploy on Vast.ai


For both technology enthusiasts and industry professionals, keeping a close eye on the latest GPU advancements isn't just a hobby – it's crucial for staying competitive and maximizing performance and compute power. Here at Vast.ai, we're just as excited to track the biggest leaps forward in graphics technology.

Today, we're looking at two GPUs in particular: the **NVIDIA RTX 4090** and its formidable successor, the **RTX 5090**. The RTX 4090 has already earned its reputation for performance, while the RTX 5090 has recently made its debut and is setting new standards in the industry.

Let's explore how the two compare with each other\!

### Spotlight on NVIDIA GeForce RTX 4090

The **RTX 4090** made waves as the top-tier consumer-grade GPU of its generation. Built on the AD102 chip and Ada Lovelace architecture, it quickly became the go-to choice for high-end gaming and demanding professional workloads.

With a triple-slot design, the RTX 4090 boasts **128 third-gen RT cores** and **512 fourth-gen Tensor cores**, elevating ray-tracing performance and speeding up AI-driven tasks like **DLSS 3** in games.

Plus, it's packed with **16,384 CUDA cores** and has **24GB of GDDR6X memory** with a maximum bandwidth of **1,008 GB/s** and a **384-bit memory bus** interface. Error-correcting code (ECC) can be enabled, but this reduces performance while making no difference in GPU rendering – so keeping it turned off is appropriate for most uses when you're prioritizing speed.

Due to its high processing capabilities, the RTX 4090 draws plenty of power: **450 watts**. It requires an 850-watt power supply and robust cooling solutions, making it one of the more demanding GPUs to run, to this day.

When it launched in October 2022, the RTX 4090 raised the bar for consumer GPUs. It may no longer sit at the top, but its capabilities continue to make it a powerhouse for serious workloads and high-end gaming.

### The New NVIDIA GeForce RTX 5090

The RTX 5090, released earlier this year, is a game-changer, redefining high-end performance in the consumer space with its groundbreaking features and next-gen architecture. With the RTX 50 series, we've now moved beyond NVIDIA's Ada Lovelace and into the era of the Blackwell architecture.

Joining the new Blackwell GPU lineup, the flagship GeForce RTX 5090 utilizes the "physically monolithic" **GB202 die**. Among its highlights are a **512-bit memory bus** and **32GB of next-gen GDDR7 memory** (the successor to the GDDR6/X memory used by the RTX 40-series GPUs). The RTX 5090 also boasts memory bandwidth of 1,792 GB/s, a substantial jump from the RTX 4090's 1,008 GB/s​.

Here's a quick side-by-side comparison of the RTX 5090 and RTX 4090's specs:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Spec
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 4090
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                RTX 5090
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">AD102</td>
            <td class="border border-gray-300 px-4 py-2 text-center">GB202</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">VRAM</td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                24GB GDDR6X
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                32GB GDDR7
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                VRAM Speed
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">21Gbps</td>
            <td class="border border-gray-300 px-4 py-2 text-center">28Gbps</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                VRAM Bus Width
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">384</td>
            <td class="border border-gray-300 px-4 py-2 text-center">512</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Bandwidth
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1.01TB/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1792GB/s
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Streaming Multiprocessors (SMs)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">128</td>
            <td class="border border-gray-300 px-4 py-2 text-center">170</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">512</td>
            <td class="border border-gray-300 px-4 py-2 text-center">680</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Ray Tracing Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">128</td>
            <td class="border border-gray-300 px-4 py-2 text-center">170</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Boost Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2520 MHz
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2407 MHz
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Total Graphics Power (TGP)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">450W</td>
            <td class="border border-gray-300 px-4 py-2 text-center">575W</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Recommended Power Supply
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">850W</td>
            <td class="border border-gray-300 px-4 py-2 text-center">1000W</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Launch Date
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Sept. 20, 2022
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Jan. 30, 2025
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Launch Price
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">$1,599</td>
            <td class="border border-gray-300 px-4 py-2 text-center">$1,999</td>
        </tr>
    </tbody>
</table>

Additionally, the RTX 5090 comes equipped with **170 streaming multiprocessors** (SMs) and **21,760 CUDA cores**, as well as **170 fourth-gen RT cores** and **680 fifth-gen Tensor cores** – up from 128 and 512 respectively in the RTX 4090\.

In terms of power consumption, the RTX 5090 has a total graphics power (TGP) of **575 watts**, considerably higher than the RTX 4090's 450 watts.

When it comes to demanding AI and video workloads, the RTX 5090 shows clear gains over its predecessor. In Procyon's AI XL (FP16) benchmark, it outperformed the RTX 4090 by 40%, and in PugetBench's DaVinci Resolve test, it delivered a 12% performance boost.

Next-generation gaming features are a major draw for the RTX 5090\. One of the most significant leaps forward comes from **DLSS 4**, NVIDIA's AI-powered upscaling technology that further enhances performance and image quality. With advancements like **Multi Frame Generation**, **Neural Rendering**, and **Mega Geometry** – and with 4K 240FPS gaming becoming more attainable – the RTX 5090 is perfectly positioned to tackle the most demanding gaming scenarios and deliver the best possible gameplay experience.

(For a detailed examination of the RTX 5090, delving into its gaming features, real-world performance, and more, see our in-depth post [here](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know).)

### Concluding Thoughts

Even after the release of the RTX 5090, it's important to recognize that the RTX 4090 still delivers exceptional performance, handling everything available at 4K with ease. The RTX 5090, while impressive, may be considered more of a luxury upgrade than a necessity for gamers and professionals who can (or already do) benefit from the power of the RTX 4090\. It really depends on whether the extra power will make enough of a difference for your needs.

Cost is another critical factor. The RTX 5090 is a top-of-the-line machine with a matching price tag. And the RTX 4090 is no slouch, either, still commanding a steep price to this day, over two years after its release.

Here at Vast.ai, we take pride in providing access to powerful machines for rent on the cloud at unbeatable prices. Vast offers unmatched flexibility, enabling you to utilize state-of-the-art technology without the expense and upkeep of purchasing your own hardware.

Whether you're looking to explore the capabilities of high-end consumer GPUs like the RTX 4090 or 5090, or check out any other machine on Vast, we'd be happy to help you discover the possibilities\!


A detailed comparison between NVIDIA's RTX 4090 and RTX 5090 GPUs, analyzing their performance, features, and value for both gaming and professional workloads.

Posts about: Industry

Announcing Virtual Machine Rental on Vast.ai

Google Colab Explained: Simplifying Your Workflow with Cloud Tools

Serving vLLM Embeddings on Vast.ai

H100 vs A100: Comparing Two Powerhouse GPUs

Serving Online Inference with LMDeploy on Vast.ai

NVIDIA RTX 4090 vs. RTX 5090: How They Stack Up