
# Serving Online Inference with LMDeploy on Vast.ai

## Background
LMDeploy is an open-source framework for Large Language Model inference. It is particularly good at high-throughput serving for multi-user or high-load use cases and is one of the most popular serving frameworks today.

LMDeploy provides an OpenAI-compatible server, which means that it can be integrated into applications that use the OpenAI API. This makes it easy to switch from using OpenAI's services to running your own models on more affordable compute.

This guide will show you how to set up LMDeploy to serve an LLM on Vast. We reference a notebook that you can use [here](https://bitbucket.org/jsbcannell/workspace/snippets/EdjpXL#file-serving_lmdeploy_on_vast.json).

### Performance Benefits:
LMDeploy is specifically focused on maintaining throughput while under higher loads. They include [benchmarks](https://github.com/InternLM/lmdeploy?tab=readme-ov-file#performance) where they compare LMDeploy to vLLM on throughput.
![Benchmark](/uploads/lmdeploy_benchmarks.png)
Here we can see LMDeploy's superior performance in the amount of requests per second that it can handle. This is important for reducing the costs of your deployment, and handling traffic spikes more smoothly.

Combining the performance benefits of LMDeploy with accessing affordable compute on Vast enables developers to ship AI applications quickly, affordably, and with low latency.

## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving an LLM, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU with decent RAM since we're going to serve a single small model. We'll also search for cuda drivers 12.0 or higher for stability. We will query the vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.0'
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 
```bash
vastai create instance <instance-id> \
  --image openmmlab/lmdeploy:latest \
  --env '-p 8000:8000' \
  --disk 40 \
  --entrypoint "lmdeploy serve api_server internlm/internlm2_5-7b-chat --model-name internlm/internlm2_5-7b-chat --server-port 8000" \
```

This command sets up LMDeploy to serve the `internlm/internlm2_5-7b-chat` model on internal port 8000.

## Connecting and Testing
To connect to your instance, you'll need to get the IP address and port number. Once your instance is done setting up, you should see something like this in the Vast.ai interface:
![Instance_view](/uploads/instance_view_lmdeploy.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_lmdeploy.png)

We will copy over the IP address and the port into the cell below.

``` bash
curl -X POST http://<Instance-IP-Address>:<Port>/v1/completions \
     -H "Content-Type: application/json" \
     -d '{"model": "internlm/internlm2_5-7b-chat", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

In the [notebook](https://bitbucket.org/jsbcannell/workspace/snippets/EdjpXL#file-serving_lmdeploy_on_vast.json) we include ways to call this model with requests or OpenAI

### Running with OpenAI
We can interact with this model via the OpenAI SDK as well:
``` bash
pip install openai
```

``` python
from openai import OpenAI

# Modify OpenAI's API key and API base to use lmDeploy 's's API server.
openai_api_key = "PASTE_YOUR_OPEN_API_KEY"

openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
completion = client.completions.create(model="internlm/internlm2_5-7b-chat",
                                      prompt="Hello, how are you?",
                                      max_tokens=50)
print("Completion result:", completion)
```

Now you're all set up with super fast inference!

LMDeploy is an open-source framework for Large Language Model inference.

Serving Online Inference with LMDeploy on Vast.ai


For both technology enthusiasts and industry professionals, keeping a close eye on the latest GPU advancements isn't just a hobby – it's crucial for staying competitive and maximizing your performance and compute power. Here at Vast.ai, we're just as excited to witness the next big leaps forward in graphics technology!

Today, we're looking at two GPUs in particular: the formidable NVIDIA RTX 4090 versus its highly anticipated successor, the not-yet-released RTX 5090. While the RTX 4090 has solidified its reputation for performance, the RTX 5090 is still a subject of much speculation.

As we explore what we know and what we anticipate, keep in mind our knowledge of the RTX 5090 is still evolving. But nonetheless, let's take a look at the facts, the early insights, and emerging rumors…

### Spotlight on NVIDIA GeForce RTX 4090

Known for being the fastest consumer-grade GPU on the market, the RTX 4090 is truly a beast of a machine. Based on the AD102 chip using the Ada Lovelace architecture, it epitomizes super-high-end gaming and professional performance.

The RTX 4090 has a triple-slot design and boasts an incredible 128 dedicated third-gen RT cores and 512 fourth-gen Tensor cores – more than its closest competitors – elevating ray-tracing performance and speeding up AI-driven tasks like DLSS 3 in games.

Plus, it's packed with 16,384 CUDA cores and has 24GB of GDDR6X graphics memory with a maximum bandwidth of 1,008 GB/s and a 384-bit memory bus interface. Error-correcting code (ECC) can be enabled, but this reduces performance while making no difference in GPU rendering – so keeping it turned off is appropriate for most uses when you're prioritizing speed.

Due to its immense processing capability, the RTX 4090 draws a lot of power to deliver maximum performance: 450 watts. It requires an 850-watt power supply and robust cooling solutions, which does increase usage costs, of course.

Launched in September 2022, the RTX 4090 still currently stands as the pinnacle of NVIDIA’s offerings for enthusiasts seeking unmatched graphics performance and computational power.

But that will change in due time.

### Anticipating NVIDIA GeForce RTX 5090

The RTX 5090 is expected to be released later this year – and it's a game-changer, poised to redefine high-end performance with its groundbreaking features and next-gen architecture. That's right, we're now moving beyond NVIDIA's Ada Lovelace and into the era of the Blackwell architecture.

Joining the new Blackwell GPU lineup, the flagship GeForce RTX 5090 will reportedly utilize the "physically monolithic" GB202 die. Among its highlights are a potential 512-bit memory bus and up to 32GB of next-gen GDDDR7 memory (the successor to the GDDR6/X memory used by the RTX 40-series GPUs). Early reports suggest the RTX 5090 could achieve memory bandwidths of up to 1,532 GB/s, a substantial boost from the RTX 4090's 1,008 GB/s​.

Some are expecting to see up to 192 streaming multiprocessors (SMs) in the RTX 5090, with an unprecedented 24,576 CUDA cores – assuming the GB202 chip keeps the 128-cores-per-SM design as seen in the RTX 4090. Plus, rumors say the RTX 5090 will feature 192 third-gen RT cores and 768 fourth-gen Tensor cores, up from 128 and 512 respectively in the RTX 4090.

The RTX 5090 is anticipated to deliver 50-70% performance improvements across the board compared to the RTX 4090 – particularly excelling in 4K resolution. And with 4K 240FPS gaming on the horizon, the RTX 5090 will be perfectly positioned to tackle the most demanding gaming scenarios.

Some are also looking forward to seeing the RTX 5090 support next-gen DLSS 4, NVIDIA's AI-powered upscaling that will further enhance performance and image quality​.

In terms of power consumption, the RTX 5090 will likely follow the trend of its predecessors, with total graphics power (TGP) ranging from 450W to 600W, about on par with the RTX 4090.

For a quick look at some of the anticipated specs of the RTX 5090 compared to the RTX 4090 – based on the latest speculation and leaks – here's a table highlighting the upgrades:

| Spec                   | RTX 4090   | RTX 5090   |
|------------------------|------------|------------|
| Streaming Multiprocessors | 128        | 192        |
| CUDA Cores             | 16,384     | 24,576     |
| Ray-Tracing Cores      | 128        | 192        |
| Tensor Cores           | 512        | 768        |
| Boost Clock            | 2.52 GHz   | 2.9 GHz    |
| L2 Cache               | 72MB       | 128MB      |
| Memory Bandwidth       | 1,008 GB/s | 1,532 GB/s |

### Concluding Thoughts

As we anticipate the release of the RTX 5090, it's important to recognize that the RTX 4090 already delivers exceptional performance, handling everything available at 4K with ease. The RTX 5090, while sure to be extremely impressive, may be seen more as a luxury upgrade rather than a necessity for gamers and professionals who can (or already do) benefit from the power of the RTX 4090.

Cost is another critical factor. The RTX 5090 will undoubtedly be a top-of-the-line machine with a matching price tag. And the RTX 4090 is no slouch, either, still commanding a steep price to this day, nearly two years after its release.

Here at Vast.ai, we take pride in providing access to powerful machines for rent on the cloud at unbeatable prices. GPU rental offers unmatched flexibility, enabling you to utilize state-of-the-art technology without the expense and upkeep of purchasing your own hardware. Whether you're looking to explore the capabilities of the RTX 4090, prepare for the arrival of the RTX 5090, or check out any other machine on Vast, we'd be happy to help you discover the possibilities!


The RTX 4099 is a very powerful card that is good for both gaming and professional use. The RTX 5090 is expected to be even more powerful and will likely be more expensive.

NVIDIA RTX 4090 vs. the Upcoming RTX 5090


# Monthly Updates

We're pleased to share this month's updates, as we continuously refine our GPU rental platform to enhance the customer experience. At Vast.ai, our focus is on improving every day, ensuring that we meet your needs with the utmost efficiency and reliability at all times.

Though occasional bugs are inevitable, addressing them quickly is one of our top priorities. Here are some of the latest bug fixes we’ve rolled out since our last update:

- Fixed issue with template editor crashing
- Double reserve deposit warning
- Confirmation of host account conversion
- Fixed earnings page inconsistencies and display errors
- Resending verification email works again
- SSH regex accepts more valid key formats
- CLI geo location filter fixed

## Cloud GPU Rental Platform Updates

At Vast.ai, we're not just fixing bugs – we're also making strides in upgrading our platform to serve you better. This month, we've made some enhancements aimed at simplifying your experience and boosting performance. Here's a quick look at some key changes:

- Save template button added
- New GPU filter for the search page with UX improvements
- GPU select now saves local storage
- Added "today" button for selecting start date

As always, you can check out the full change log for our May 2024 product updates at the bottom of this page to discover all the enhancements we've made this month.

We’re committed to providing a top-tier, cost-effective GPU rental marketplace for both renters and hosts here at Vast.ai. Whether you’re just starting out or looking for ways to optimize your existing projects, we’re here to help every step of the way. Connect with our knowledgeable team via the chatbox at the bottom right, send us an email at [support@vast.ai](mailto:support@vast.ai), or join our engaging Discord community for additional support and resources. We'd be happy to assist you in making the most of what Vast.ai has to offer!

## Change Log

### Features:

- Save template button
- New GPU filter for the search page
- GPU select now saves local storage

### Bug Fixes:

- PDF invoice date range fix
- Fixed issue with template editor crashing
- Exported invoice dates, timezones, SSH key validation & cpu_ghz filter
- Cloud copy errors properly cleared
- Double reserve deposit warning
- Price breakdown disk value display corrected
- Confirm host account conversion
- Fixed earnings page inconsistencies and display errors
- Do not show reserved discounts on unverified search offers
- Resending verification email works again
- SSH regex accepts more valid key formats
- Instance IP modal scroll issues fixed
- Instance count now accurate
- Occupancy filter
- Price breakdown disk value
- Negative credit properly shows in header
- CLI geo location filter fixed
- CLI search offers "gpu_frac" unrecognized
- Removed newline from CLI command results show instances

### Miscellaneous Tasks:

- Today button for start date
- Typos
- Account key warning
- Instance label sort
- Clearer cloud copy errors
- UX improvements for GPU filter
- Formatting and warnings


May 2024 Product Update


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support! 

With this latest update, we're welcoming both the Radeon and Instinct product lines from AMD to the existing array of NVIDIA machines available for rent on our marketplace. As we widen the range of supported hardware, we can help connect even more customers with the specific GPU compute requirements they're looking for. 

Here are the [AMD GPUs we have available now](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76).

## Now Supporting AMD Instinct and Radeon GPUs

The Instinct line is AMD's brand of data center GPUs, intended to accelerate deep learning, artificial neural network, and high-performance computing tasks, while the Radeon line is aimed at gaming and consumer-grade applications. With both now supported on Vast, this expansion means more versatility and options for all of our users.

Travis Cannell, COO at Vast.ai, highlights the strategic advantage of the update: "We are interested in partnering with managed service providers who have the Instinct line and who want to increase the utilization of their clusters, as well as the Radeon line for as our users have needs for remote gaming, rendering and more." 

Ultimately, our platform will now be better equipped to meet the diverse needs and preferences of our user base – from data centers to gamers and everyone in between. Whether you're scaling up operations or just starting out, we're here to support your computing journey.

## How to Get Started

To browse the active AMD listings on cloud.vast.ai, you will need to select an AMD template which will then filter results for AMD GPUs. Renters looking to leverage these new GPU options on Vast should use the ROCm template. The two templates supported now include an Ubuntu 22.04 image and a PyTorch image.

To see the active results, select the desired template:

- [AMD Ubuntu 22.04 ROCm 6.02](https://cloud.vast.ai/?ref_id=43484&template_id=9e190d88664e20818ee75eb33dce943c)
- [AMD ROCm PyTorch](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76)

You can also find the latest AMD templates by looking for AMD on the [template page](https://cloud.vast.ai/templates/).

![Vast.ai AMD Templates](/uploads/amd-templates.PNG)


## How to Host AMD Equipment

For those interested in hosting their GPUs, Vast.ai provides comprehensive instructions in our [Machine Hosting Setup guide](https://cloud.vast.ai/host/setup/). 

Whether you're setting up an AMD or NVIDIA machine, the step-by-step guide gives you all the information you need to get started. From system requirements and networking configuration to troubleshooting and how to list your machine, learn to optimize your hosting setup and get up and running as soon as possible.

## Leading the Way in GPU Rental Innovation

The addition of support for AMD GPUs comes at a time when Vast.ai has announced an incredible 265% year-over-year growth since 2019. This growth illustrates the heavy demand in the GPU rental market. With more than 350 independent hosts supplying 16,000+ GPUs – and growing – Vast.ai efficiently matches GPU supply and demand, ensuring that every user finds the best deal for their specific compute needs.

As Vast.ai leads the charge in GPU rental innovation, the addition of AMD support is more than just an upgrade – it's also a reflection of our commitment to the democratization of AI. Join us as we continue to make powerful computing more accessible to everyone.


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support!

Announcing AMD Support


Just this past year, NVIDIA introduced a groundbreaking security feature in the Hopper architecture to protect sensitive data, AI models, and applications in use. The new feature, called NVIDIA Confidential Computing, offers a hardware-based solution for securely processing data and code that is in use, preventing unauthorized access and modification. 

  

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible – while you also get to enjoy the cutting-edge acceleration of NVIDIA H200 and H100 Tensor Core GPUs.

  

### What Is NVIDIA Confidential Computing?

  

To put it simply, "confidential computing" (CC) protects data-in-use by performing computation in a trusted execution environment (TEE) that is both hardware-based and attested. 

  

The NVIDIA H200 and H100 GPUs have a CVM (confidential virtual machine) TEE on the CPU that is anchored in an on-die root of trust (RoT). When it boots in CC-On mode, hardware protections that provide confidentiality and integrity of code and data are enabled. (In this context, "confidentiality" means that an attacker cannot access any data or code, and "integrity" means that they cannot modify the execution.) 

  

Here's how it works:

  

1. A chain of trust is established via a GPU boot sequence with a secure and measured boot.
    

  

2. An SPDM (Security Protocol and Data Model) session facilitates a secure connection to the driver in a CPU TEE. 
    

  

3. An attestation report with a cryptographically signed set of measurements is generated, verifying system integrity.
    

  

In order to run a confidential workload, the GPU must be authenticated as a legitimate NVIDIA GPU that supports confidential computing (a process involving a device-unique private key fused onto each GPU corresponding to a certified public key); the GPU must not have been revoked for CC; and the GPU attestation report must be verified. 

  

Upon successful verification, the NVIDIA driver in the CVM establishes a secure channel, via a session key, with the GPU hardware TEE in order to transfer data, perform computation, and retrieve results. The CVM and GPU communicate through a shared memory region outside the CVM since the CPU prevents the GPU from directly accessing a CVM's memory. They use AES-GCM – an advanced encryption standard – to prevent the host system from reading this communication. 

  

Finally, the GPU copies and decrypts all inputs to its internal memory, where everything runs in plaintext and direct access is blocked. Performance counters are disabled as well, since these could be used for side-channel attacks. 

  

In this way, a confidential-computing environment is extended from a CVM (or a secure enclave) to a GPU, with attestation, encrypted communication, and memory isolation making it possible. 

  

### Benefits & Positive Impacts

  

The benefits of NVIDIA Confidential Computing are significant:

  

- Hardware-Based Security and Isolation – You can achieve full isolation of virtual machines (VMs) in any environment – on-premises, at the edge, or on the cloud. Your entire workload is secured with these built-in hardware firewalls that provide unprecedented protection. 
    

  

- Verifiability with Device Attestation – You will always rest assured that only authorized end users can deploy data and code for execution in the H200 or H100's TEE. Plus, through device attestation, you'll know that you're dealing with an authentic NVIDIA GPU, the firmware hasn't been modified, and updates have been done as expected. 
    

  

- Protection from Unauthorized Access – Your sensitive data, AI workloads, and intellectual property are protected, with confidentiality and integrity preserved at all times. All unauthorized entities (including the hypervisor, cloud provider, host operating system, and even anyone with physical access to the infrastructure) are blocked from viewing or modifying the AI application and data during execution. 
    

  

- No Application Code Change – As NVIDIA puts it, the new confidential computing feature "just works," without any code changes required for your GPU-accelerated workloads. 
    

  

As you can imagine, confidential computing-enabled GPUs usher in a whole host of use cases where security, privacy, and regulatory compliance are paramount. For instance, intellectual property protection is now possible for proprietary AI models, even when these models are distributed and deployed at scale on shared or remote infrastructure. 

  

Privacy and confidentiality can also now be preserved in AI training and inference – most notably in industries like healthcare, finance, and the public sector, where data may be sensitive and/or regulated. On top of that, collaboration between multiple parties can be done securely when building and improving AI models across participating sites, for use cases like medical imaging, drug development, and fraud detection. 

  

Another bonus: organizations, governments, and individuals can outsource AI workloads to a cloud provider while still ensuring protection from the underlying infrastructure. 

  

### Final Thoughts

  

It's exciting that confidential computing is now possible for even the most demanding workloads! Granted, for the vast majority of use cases, CC-enabled GPUs may arguably be overkill. But for those who need or prefer absolute confidentiality and security, NVIDIA Confidential Computing in the Hopper architecture can now provide that for you. 

  

At Vast.ai, we understand how important privacy and security can be for our customers. It is our hope that you'll find exactly what you need on our cost-effective, on-demand cloud GPU rental platform. 

  

In our global network of compute providers on Vast, some hosts are experimenting with NVIDIA Confidential Computing. Complete support for this feature is not yet available. Join our Discord for the latest product updates and up-to-date support.

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible 

Industry

Serving Online Inference with LMDeploy on Vast.ai

NVIDIA RTX 4090 vs. the Upcoming RTX 5090

May 2024 Product Update

Announcing AMD Support

Absolute Security with NVIDIA Confidential Computing