
# Monthly Updates

We're pleased to share this month's updates, as we continuously refine our GPU rental platform to enhance the customer experience. At Vast.ai, our focus is on improving every day, ensuring that we meet your needs with the utmost efficiency and reliability at all times.

Though occasional bugs are inevitable, addressing them quickly is one of our top priorities. Here are some of the latest bug fixes we’ve rolled out since our last update:

- Fixed issue with template editor crashing
- Double reserve deposit warning
- Confirmation of host account conversion
- Fixed earnings page inconsistencies and display errors
- Resending verification email works again
- SSH regex accepts more valid key formats
- CLI geo location filter fixed

## Cloud GPU Rental Platform Updates

At Vast.ai, we're not just fixing bugs – we're also making strides in upgrading our platform to serve you better. This month, we've made some enhancements aimed at simplifying your experience and boosting performance. Here's a quick look at some key changes:

- Save template button added
- New GPU filter for the search page with UX improvements
- GPU select now saves local storage
- Added "today" button for selecting start date

As always, you can check out the full change log for our May 2024 product updates at the bottom of this page to discover all the enhancements we've made this month.

We’re committed to providing a top-tier, cost-effective GPU rental marketplace for both renters and hosts here at Vast.ai. Whether you’re just starting out or looking for ways to optimize your existing projects, we’re here to help every step of the way. Connect with our knowledgeable team via the chatbox at the bottom right, send us an email at [support@vast.ai](mailto:support@vast.ai), or join our engaging Discord community for additional support and resources. We'd be happy to assist you in making the most of what Vast.ai has to offer!

## Change Log

### Features:

- Save template button
- New GPU filter for the search page
- GPU select now saves local storage

### Bug Fixes:

- PDF invoice date range fix
- Fixed issue with template editor crashing
- Exported invoice dates, timezones, SSH key validation & cpu_ghz filter
- Cloud copy errors properly cleared
- Double reserve deposit warning
- Price breakdown disk value display corrected
- Confirm host account conversion
- Fixed earnings page inconsistencies and display errors
- Do not show reserved discounts on unverified search offers
- Resending verification email works again
- SSH regex accepts more valid key formats
- Instance IP modal scroll issues fixed
- Instance count now accurate
- Occupancy filter
- Price breakdown disk value
- Negative credit properly shows in header
- CLI geo location filter fixed
- CLI search offers "gpu_frac" unrecognized
- Removed newline from CLI command results show instances

### Miscellaneous Tasks:

- Today button for start date
- Typos
- Account key warning
- Instance label sort
- Clearer cloud copy errors
- UX improvements for GPU filter
- Formatting and warnings


May 2024 Product Update


When it comes to high-performance computing (HPC), the NVIDIA H100 is among the best of the best GPUs on the market. Its predecessor, the NVIDIA A100, is also a very impressive GPU in its own right. From enterprise to exascale, these GPUs are enabling groundbreaking advancements in everything from AI research to complex scientific simulations.

Today we're taking a look at how the H100 and A100 compare with each other. Depending on your use case, one or the other might be better suited for your needs and preferences. So let's dive in!

## Introduction to the NVIDIA H100 and A100

As mentioned, the H100 GPU is the successor to the A100. It was specifically optimized for machine learning and designed to [deliver](https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/) an "order-of-magnitude performance leap for large-scale AI and HPC" over the A100, with "substantial improvements in architectural efficiency."

Based on the innovative Hopper architecture, the H100 can speed up large language models (LLMs) by 30X over the A100, according to NVIDIA. It features fourth-gen Tensor Cores and – unlike the A100 – a dedicated Transformer Engine to solve trillion-parameter language models.

Whether it's used in smaller enterprise systems or massive, unified GPU clusters, the H100 delivers efficient scalability thanks to its combination of fourth-gen NVLink (with 900 GB/s of GPU-to-GPU interconnect), NDR Quantum-2 InfiniBand networking (speeding up communication by every GPU across nodes), and PCIe Gen5. With superior FP64 and FP8 performance, the H100 is ideal for next-gen AI workloads.

The A100 GPU is known for its versatility, however. Built on the Ampere architecture, it was groundbreaking when released in 2020, providing up to 20X higher performance over the prior NVIDIA Volta generation. It can efficiently scale up or be partitioned using Multi-Instance GPU (MIG) into seven isolated GPU instances, just like the H100, which is ideal for shifting workload demands.

In short, the A100 excels in both training and inference tasks, and while it may not offer the sheer power of the H100, its robust FP64 performance makes it a great fit for scientific simulations and data analytics.

## Key Differences: H100 vs. A100

The H100 has an updated chip design and various other factors that distinguish it from the A100. The following are a few that are important to consider:

### Memory Bandwidth

The A100 has a memory bandwidth of about 2 TB/s, which is plenty for most use cases; however, the H100 offers an impressive 3.35 TB/s, making it ideal for workloads where data transfer bottlenecks may occur.

### CUDA Cores and Tensor Cores

A higher number of CUDA and Tensor Cores means a GPU can perform more parallel computations simultaneously. With 14,592 CUDA Cores and 456 Tensor Cores, the H100 outperforms the A100’s 6,912 CUDA Cores and 432 Tensor Cores, delivering superior performance in applications that can leverage increased parallelism.

### Confidential Computing

The H100's Hopper architecture introduced NVIDIA's new confidential computing environment. It protects data-in-use by performing computation in a trusted execution environment (TEE) that is both hardware-based and attested. This means you essentially have absolute security for use cases where security, privacy, and regulatory compliance are paramount.

### Tensor Memory Accelerator (TMA)

Another feature of the H100's Hopper architecture is the TMA unit, which can transfer large blocks of data efficiently between global memory and shared memory. It supports asynchronous copies between thread blocks in a cluster as well, optimizing data movement and memory management for tensor operations. Ultimately, this enhances the efficiency and speed of large-scale AI and deep learning tasks.

### MIG Capability

The Multi-Instance GPU (MIG) feature is present in both the H100 and A100, providing flexible workload management so you can run multiple LLMs in parallel, for instance. The H100's MIG capability is more robust, however, with better resource allocation and more memory and compute power to split.

## Performance Benchmarks

According to NVIDIA's own benchmarks, the H100 offers up to 30X better inference performance than the A100. In another test where the A100 and H100 were used in clusters to train two large language models (LLMs), the H100 demonstrated remarkable speed improvements. The H100's new NVLink Switch System provided an even bigger boost, enabling the H100 cluster to train the models up to nine times faster than the A100 cluster:

![H100 vs A100 Speed Comparison](/uploads/Speed-up-over-A100.png)

## Choosing the Right GPU

So what does all this mean for you? The best GPU for your needs will depend on numerous factors. Here's a quick outline of the various features and specs of the NVIDIA A100 and H100:

| Feature                         | A100                   | H100                    |
|---------------------------------|------------------------|-------------------------|
| **GPU Architecture**            | Ampere                 | Hopper                  |
| **GPU Memory**                  | 40 or 80 GB HBM2e      | 80 MB HBM3              |
| **GPU Memory Bandwidth**        | 1.6 to 2 TB/s          | 3.35 TB/s               |
| **CUDA Cores**                  | 6912                   | 14,592                  |
| **FP64 TFLOPS**                 | 9.7                    | 33.5                    |
| **FP32 TFLOPS**                 | 19.6                   | 67                      |
| **TF32 Tensor Core Flops\***    | 156 \| 312             | 378 \| 756              |
| **FP16 Tensor Core Flops\***    | 312 \| 624             | 756 \| 1513             |
| **FP8 Tensor Core TFLOPS**      | N/A                    | 3958                    |
| **Peak INT8 TOPS\***            | 624 \| 1248            | 3958                    |
| **Media Engine**                | 0 NVENC<br>5 NVDEC<br>5 NVJPEG | 0 NVENC<br>7 NVDEC<br>7 NVJPEG |
| **L2 Cache**                    | 40 MB                  | 50 MB                   |
| **Power**                       | Up to 400 W            | Up to 700 W             |
| **Form Factor**                 | SXM4 - 8-way HGX       | SXM5 - 8-way HGX        |
| **Interconnect**                | PCIe 4.0 x16           | PCIe 5.0 x16            |

\*Without and with structured sparsity.

---

## Use Cases

When it comes to use cases, you may want to consider some general guidelines.

### The A100 is well suited for:
- Versatile AI workloads including training and inference.
- Applications needing a balance of performance and cost-efficiency.
- Scenarios where ultra-high precision isn't the priority.

### The H100 is ideal for:
- Cutting-edge AI training, large-scale simulations, and other HPC workloads.
- Environments where computational speed and memory bandwidth are critical.
- Future-proofing investments in AI infrastructure.

---

When deciding between the A100 and H100, consider your specific workload requirements. If you need top-tier double-precision performance and superior memory bandwidth, or you're dealing with next-gen HPC at datacenter scale and trillion-parameter AI, the H100 is the clear winner. For a more versatile and cost-effective solution that still delivers powerful AI performance, the A100 is a solid choice.

At Vast.ai, we understand the importance of finding the right GPU for your needs as well as your budget. Our cloud GPU rental marketplace offers a variety of machines at the lowest prices possible, saving you 5-6X on GPU compute. Whether you prefer on-demand pricing or interruptible instances through spot auction-based pricing, we've got you covered.

We invite you to discover what Vast can do for you and accelerate your projects with the right GPU today!


Explore the differences between NVIDIA's H100 and A100 GPUs, from architectural advancements to performance benchmarks.

H100 vs A100: Comparing Two Powerhouse GPUs


# Serving Online Inference with LMDeploy on Vast.ai

## Background
LMDeploy is an open-source framework for Large Language Model inference. It is particularly good at high-throughput serving for multi-user or high-load use cases and is one of the most popular serving frameworks today.

LMDeploy provides an OpenAI-compatible server, which means that it can be integrated into applications that use the OpenAI API. This makes it easy to switch from using OpenAI's services to running your own models on more affordable compute.

This guide will show you how to set up LMDeploy to serve an LLM on Vast. We reference a notebook that you can use [here](https://bitbucket.org/jsbcannell/workspace/snippets/EdjpXL#file-serving_lmdeploy_on_vast.json).

### Performance Benefits:
LMDeploy is specifically focused on maintaining throughput while under higher loads. They include [benchmarks](https://github.com/InternLM/lmdeploy?tab=readme-ov-file#performance) where they compare LMDeploy to vLLM on throughput.
![Benchmark](/uploads/lmdeploy_benchmarks.png)
Here we can see LMDeploy's superior performance in the amount of requests per second that it can handle. This is important for reducing the costs of your deployment, and handling traffic spikes more smoothly.

Combining the performance benefits of LMDeploy with accessing affordable compute on Vast enables developers to ship AI applications quickly, affordably, and with low latency.

## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving an LLM, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU with decent RAM since we're going to serve a single small model. We'll also search for cuda drivers 12.0 or higher for stability. We will query the vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.0'
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 
```bash
vastai create instance <instance-id> \
  --image openmmlab/lmdeploy:latest \
  --env '-p 8000:8000' \
  --disk 40 \
  --entrypoint "lmdeploy serve api_server internlm/internlm2_5-7b-chat --model-name internlm/internlm2_5-7b-chat --server-port 8000" \
```

This command sets up LMDeploy to serve the `internlm/internlm2_5-7b-chat` model on internal port 8000.

## Connecting and Testing
To connect to your instance, you'll need to get the IP address and port number. Once your instance is done setting up, you should see something like this in the Vast.ai interface:
![Instance_view](/uploads/instance_view_lmdeploy.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_lmdeploy.png)

We will copy over the IP address and the port into the cell below.

``` bash
curl -X POST http://<Instance-IP-Address>:<Port>/v1/completions \
     -H "Content-Type: application/json" \
     -d '{"model": "internlm/internlm2_5-7b-chat", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

In the [notebook](https://bitbucket.org/jsbcannell/workspace/snippets/EdjpXL#file-serving_lmdeploy_on_vast.json) we include ways to call this model with requests or OpenAI

### Running with OpenAI
We can interact with this model via the OpenAI SDK as well:
``` bash
pip install openai
```

``` python
from openai import OpenAI

# Modify OpenAI's API key and API base to use lmDeploy 's's API server.
openai_api_key = "PASTE_YOUR_OPEN_API_KEY"

openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
completion = client.completions.create(model="internlm/internlm2_5-7b-chat",
                                      prompt="Hello, how are you?",
                                      max_tokens=50)
print("Completion result:", completion)
```

Now you're all set up with super fast inference!

LMDeploy is an open-source framework for Large Language Model inference.

Serving Online Inference with LMDeploy on Vast.ai


For both technology enthusiasts and industry professionals, keeping a close eye on the latest GPU advancements isn't just a hobby – it's crucial for staying competitive and maximizing your performance and compute power. Here at Vast.ai, we're just as excited to witness the next big leaps forward in graphics technology!

Today, we're looking at two GPUs in particular: the formidable NVIDIA RTX 4090 versus its highly anticipated successor, the not-yet-released RTX 5090. While the RTX 4090 has solidified its reputation for performance, the RTX 5090 is still a subject of much speculation.

As we explore what we know and what we anticipate, keep in mind our knowledge of the RTX 5090 is still evolving. But nonetheless, let's take a look at the facts, the early insights, and emerging rumors…

### Spotlight on NVIDIA GeForce RTX 4090

Known for being the fastest consumer-grade GPU on the market, the RTX 4090 is truly a beast of a machine. Based on the AD102 chip using the Ada Lovelace architecture, it epitomizes super-high-end gaming and professional performance.

The RTX 4090 has a triple-slot design and boasts an incredible 128 dedicated third-gen RT cores and 512 fourth-gen Tensor cores – more than its closest competitors – elevating ray-tracing performance and speeding up AI-driven tasks like DLSS 3 in games.

Plus, it's packed with 16,384 CUDA cores and has 24GB of GDDR6X graphics memory with a maximum bandwidth of 1,008 GB/s and a 384-bit memory bus interface. Error-correcting code (ECC) can be enabled, but this reduces performance while making no difference in GPU rendering – so keeping it turned off is appropriate for most uses when you're prioritizing speed.

Due to its immense processing capability, the RTX 4090 draws a lot of power to deliver maximum performance: 450 watts. It requires an 850-watt power supply and robust cooling solutions, which does increase usage costs, of course.

Launched in September 2022, the RTX 4090 still currently stands as the pinnacle of NVIDIA’s offerings for enthusiasts seeking unmatched graphics performance and computational power.

But that will change in due time.

### Anticipating NVIDIA GeForce RTX 5090

The RTX 5090 is expected to be released later this year – and it's a game-changer, poised to redefine high-end performance with its groundbreaking features and next-gen architecture. That's right, we're now moving beyond NVIDIA's Ada Lovelace and into the era of the Blackwell architecture.

Joining the new Blackwell GPU lineup, the flagship GeForce RTX 5090 will reportedly utilize the "physically monolithic" GB202 die. Among its highlights are a potential 512-bit memory bus and up to 32GB of next-gen GDDDR7 memory (the successor to the GDDR6/X memory used by the RTX 40-series GPUs). Early reports suggest the RTX 5090 could achieve memory bandwidths of up to 1,532 GB/s, a substantial boost from the RTX 4090's 1,008 GB/s​.

Some are expecting to see up to 192 streaming multiprocessors (SMs) in the RTX 5090, with an unprecedented 24,576 CUDA cores – assuming the GB202 chip keeps the 128-cores-per-SM design as seen in the RTX 4090. Plus, rumors say the RTX 5090 will feature 192 third-gen RT cores and 768 fourth-gen Tensor cores, up from 128 and 512 respectively in the RTX 4090.

The RTX 5090 is anticipated to deliver 50-70% performance improvements across the board compared to the RTX 4090 – particularly excelling in 4K resolution. And with 4K 240FPS gaming on the horizon, the RTX 5090 will be perfectly positioned to tackle the most demanding gaming scenarios.

Some are also looking forward to seeing the RTX 5090 support next-gen DLSS 4, NVIDIA's AI-powered upscaling that will further enhance performance and image quality​.

In terms of power consumption, the RTX 5090 will likely follow the trend of its predecessors, with total graphics power (TGP) ranging from 450W to 600W, about on par with the RTX 4090.

For a quick look at some of the anticipated specs of the RTX 5090 compared to the RTX 4090 – based on the latest speculation and leaks – here's a table highlighting the upgrades:

| Spec                   | RTX 4090   | RTX 5090   |
|------------------------|------------|------------|
| Streaming Multiprocessors | 128        | 192        |
| CUDA Cores             | 16,384     | 24,576     |
| Ray-Tracing Cores      | 128        | 192        |
| Tensor Cores           | 512        | 768        |
| Boost Clock            | 2.52 GHz   | 2.9 GHz    |
| L2 Cache               | 72MB       | 128MB      |
| Memory Bandwidth       | 1,008 GB/s | 1,532 GB/s |

### Concluding Thoughts

As we anticipate the release of the RTX 5090, it's important to recognize that the RTX 4090 already delivers exceptional performance, handling everything available at 4K with ease. The RTX 5090, while sure to be extremely impressive, may be seen more as a luxury upgrade rather than a necessity for gamers and professionals who can (or already do) benefit from the power of the RTX 4090.

Cost is another critical factor. The RTX 5090 will undoubtedly be a top-of-the-line machine with a matching price tag. And the RTX 4090 is no slouch, either, still commanding a steep price to this day, nearly two years after its release.

Here at Vast.ai, we take pride in providing access to powerful machines for rent on the cloud at unbeatable prices. GPU rental offers unmatched flexibility, enabling you to utilize state-of-the-art technology without the expense and upkeep of purchasing your own hardware. Whether you're looking to explore the capabilities of the RTX 4090, prepare for the arrival of the RTX 5090, or check out any other machine on Vast, we'd be happy to help you discover the possibilities!


The RTX 4099 is a very powerful card that is good for both gaming and professional use. The RTX 5090 is expected to be even more powerful and will likely be more expensive.

Blog

May 2024 Product Update

Monthly Updates

Cloud GPU Rental Platform Updates

Change Log

Features:

Bug Fixes:

Miscellaneous Tasks:

Continue Reading: