
# Welcome back to our monthly check-in here at Vast.ai!

The big news this month is that hosting fees on Vast have been removed. Previously, a hosting fee was deducted from host earnings before payout, but it has now been replaced with a more flexible internal surcharge model. For renters, nothing will change.

For hosts, since there's no longer a hosting fee, you won't have to mentally deduct that amount from your earnings or adjust your prices to cover it. Moving forward, the prices hosts set will be the prices they earn. (And if you're a host using the CLI to adjust prices automatically, you'll want to modify your code to reflect the change in pricing semantics if you haven't done so already.) Your net earnings will not be affected by this update.

## Bug Fixes

Bug fixes are also a critical part of our ongoing improvements. Below are some of the latest adjustments we've made to keep Vast running smoothly for all our users:

- Fixed support for environment variables containing '='
- The CLI command from a template now does create the instance
- Reworked and fixed issues with PDF invoice exports
- Added additional error reporting to host listing function

## Cloud GPU Rental Platform Updates

At Vast.ai, updating our product means more than just troubleshooting – we're also committed to making other meaningful changes. The latest updates are as follows:

- Added copy and delete buttons to templates
- Improved rsync to try multiple IPs (public and private) with timeout
- Added a search engine for templates
- Reserved discounts now only available for verified machines

You can find the full change log for our June 2024 product updates at the bottom of this page.

If you have any questions about these updates or need assistance with our platform, don't hesitate to get in touch! You can chat with us directly through the chatbox at the bottom right, email us at support@vast.ai, or join the vibrant community on our Discord server for ongoing support and updates. We'd be happy to help with any of your needs!

## Change Log

### Bug Fixes:
- Corrected typos in triggered emails
- Fixed Docker drive mount point on Daemon in 22.04 and later
- The CLI command from a template now does create the instance
- Fixed support for environment variables containing '='
- Reworked and fixed issues with PDF invoice exports
- Added additional error reporting to host listing function

### Updated Features:
- Changed the hosting fees to 0%
- Reserved discounts now only available for verified machines
- Added copy and delete buttons to templates
- Improved rsync to try multiple IPs (public and private) with timeout
- Added a search engine for templates

there's no longer a hosting fee, you won't have to mentally deduct that amount from your earnings or adjust your prices to cover it...

June 2024 Product Update


# NVIDIA GeForce RTX 5090: Rumors and Speculations

As the year winds down, rumors are intensifying around NVIDIA's highly anticipated **GeForce RTX 5090** GPU. Industry insiders are divided on the release date, with some sources suggesting a launch just in time for Christmas, while other reports point to a formal announcement at **CES 2025** in the new year.

**NVIDIA's CEO Jensen Huang** will be hosting the keynote at CES on **January 6th, 2025** – which could be the perfect opportunity to launch the full trio of GPUs in the 50-series lineup: the **RTX 5090**, **5080**, and **5070**. But it's also possible that the **5090** itself will debut in time for last-minute holiday shopping.

The exact timing of the RTX 5090 release may be uncertain, but one thing is clear: NVIDIA's next flagship GPU is poised to redefine performance. Let's take a quick look at the speculation around this next-gen graphics powerhouse and its companions in the new lineup!

## NVIDIA RTX 5090: Power, Redefined

As we explored in a [previous blog post](https://vast.ai/article/nvidia-rtx-5090-leaks-rumors-gpu-performance), the flagship consumer-grade **RTX 5090** is set to dethrone its predecessor, the RTX 4090, with some truly remarkable upgrades. Here's what the various leaks and rumors suggest:

- Built on the cutting-edge **Blackwell architecture**, leaving Ada Lovelace behind
- **32GB** of next-gen **GDDR7 memory**, setting a new standard
- **512-bit** memory bus (compared to the 4090's 384-bit)
- **21,760 CUDA cores**, providing incredible horsepower for heavy workloads
- Base clock speed of ~**2900 MHz**, with potential boost clock speeds exceeding **3 GHz**
- Features the new **GB202 gaming chip**
- Powered by a single **12VHPWR connector**
- **600W** power consumption

The RTX 5090's little siblings in the 50 series are expected to pack a punch as well, balancing performance and cost for those who may want to upgrade without breaking the bank.

## NVIDIA RTX 5080 & 5070: Cost-effective Alternatives

The **RTX 5080** will likely be an excellent option for anyone needing a high-performance GPU without going all-in on the 5090. In fact, it reportedly could be even more powerful than initially expected. Rumors suggest it will feature a **32Gbps** memory speed – instead of the previously reported 28Gbps – and a maximum bandwidth of **1024GB/s** rather than **896GB/s**.

If these specs turn out to be true, then the RTX 5080 could outperform the RTX 4090 while also being more power-efficient, making it an exciting contender for high-performance users. To round out the rumored details, the **5080** will reportedly feature a **256-bit memory bus** with **16GB of GDDR7 VRAM**, around **10,000 CUDA cores**, and **400W** power consumption.

Finally, the **RTX 5070** will serve as a mid-range option without the extreme capabilities of the 5090 or 5080. It's rumored to offer a **192-bit memory bus** with **12GB of GDDR7 VRAM** (to the consternation of some who feel 12GB is surprisingly low), around **6,400 CUDA cores**, plus **250W** power consumption.

With all three cards – the **RTX 5090**, **5080**, and **5070** – likely being powered by a single 12VHPWR connector, built on the latest Blackwell architecture, and delivering everything from outstanding 4K resolution to potentially next-gen DLSS 4, these GPUs are expected to deliver the kind of leap in performance that will cater to a wide range of needs.

## Comparing the Rumored Specs: RTX 5090, 5080, and 5070

For easy reference, here's a side-by-side comparison of the rumored specifications for the entire 50-series lineup.

| Spec                  | RTX 5090 (Rumored)  | RTX 5080 (Rumored)  | RTX 5070 (Rumored) |
| --------------------- | ------------------- | ------------------- | ------------------ |
| **Architecture**      | Blackwell GB202-300 | Blackwell GB203-400 | Blackwell GB205    |
| **Memory**            | GDDR7               | GDDR7               | GDDR7              |
| **VRAM**              | 32GB                | 16GB                | 12GB               |
| **CUDA Cores**        | 21,760              | 10,752              | 6,400              |
| **Boost Clock**       | 3.0 GHz             | TBD                 | TBD                |
| **Memory Bus**        | 512-bit             | 256-bit             | 192-bit            |
| **Memory Speed**      | 28 Gbps             | 32 Gbps             | 28 Gbps            |
| **Bandwidth**         | 1792 GB/s           | 1024 GB/s           | 672 GB/s           |
| **Total Board Power** | 600W                | 400W                | 250W               |

> **Note:** All specifications mentioned are based on current rumors and reports. Final specifications may vary upon release.

## Can't Wait? We've Got You Covered

While we all wait for the official release (whether it's Christmas or CES), in the meantime you don't have to put your projects on hold. **Vast.ai** offers immediate access to high-performance GPUs through our [cloud GPU rental platform](https://vast.ai/).

Ready to supercharge your compute power today? Check out the flexible GPU rental options on Vast and get started in minutes – no waiting required!


GPU prices continue to increase, but there exists a happy and unexpected middle ground for AI applications.

NVIDIA RTX 5090: Out by Christmas? A Look at the Latest Rumors


# Serving Online Inference with SGLang on Vast.ai

## Background

SGLang is an open-source framework for serving language models, focusing on throughput for serving and batch workloads. This makes it ideal for building apps that serve multiple users and need to scale. SGLang is used by many high-tech companies for serving their Language Models to the public, most notably [X AI and their Grok Model.](https://venturebeat.com/ai/grok-2-gets-a-speed-bump-after-developers-rewrite-code-in-three-days/)

SGLang provides an OpenAI-compatible server, allowing you to easily integrate it into chatbots and other applications.

As companies develop their AI products, they often face challenges like rate limits and high costs when using these models. With SGLang on Vast, you can run your own models in the form factor you need, at a much more affordable price point. As inference demand grows with agents and complex workflows, SGLang on Vast excels in performance and affordability where it matters most.

This guide will show you how to set up SGLang to serve a language model on Vast. We reference a notebook that you can use [here](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/XEyMo8/72acf45925f8230b75195b7da9fff1884fa4052d/files/serving_vllm_on_vast.json)

## Setup and Querying

First, set up your environment and Vast API key:

```bash
pip install --upgrade vastai
```

Once you create your account, go [here](https://cloud.vast.ai/cli/) to find your API key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving a language model, we're looking for a machine with a static IP address, available ports to host on, and a single modern GPU with decent RAM, CUDA graphs take a lot of space and can affect deployment options for certain models. SGLang also requires CUDA version 12.2 or higher, so we'll filter for that as well. We will query the Vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 40 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.2'
```

## Deploying the Image

The easiest way to deploy this instance is to use the command line. Copy and paste a specific instance ID you choose from the list above into `instance-id` below. We'll also need to input the Huggingface API Token into `<Huggingface-Token>`.

```bash
vastai create instance <instance-id> --image lmsysorg/sglang:latest --env '-p 8000:8000 -e HF_TOKEN=<Huggingface-Token>' --disk 60 --args python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --port 8000
```

## Connecting and Testing

To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view_sglang.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_sglang.png)

We will copy over the IP address and the port into the cell below.

```bash
# This request assumes you haven't changed the model. If you did, fill it in the "model" value in the payload json below
curl -X POST http://<IP-Address>:<Port>/v1/completions -H "Content-Type: application/json"  -d '{"model" : "meta-llama/Meta-Llama-3.1-8B-Instruct", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

In the [notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/XEyMo8/72acf45925f8230b75195b7da9fff1884fa4052d/files/serving_vllm_on_vast.json), we include ways to call this model with requests or OpenAI

### Other things to look out for with other configurations

If you are downloading a model that needs authentication from the Hugging Face Hub, passing `-e HF_token=<Your-Read-Only-Token>` within Vast's `--env` variable string should help.

Sometimes the full context of a model can't be used given the space allocated for SGLang on the GPU + the model's size. In those cases, you might want to increase `--gpu-memory-utilization`, or decrease the `max-model-len`. Increasing `--gpu-memory-utilization` does come with CUDA OutOfMemory issues that can be hard to predict ahead of time.

We won't need either of these for this specific model and GPU configuration.

### Testing

Copy the IP address from your instance once it is ready, and then we can use the following code to call it. Note that while your server might have ports ready, the model might not have downloaded yet as it is much larger this time. You can check the status of this via the logs to see when it has started serving.

```bash
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'meta-llama/Meta-Llama-3.1-8B-Instruct',
    'prompt': 'Hello, how are you?',
    'max_tokens': 50,
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/completions', headers=headers, json=json_data)
print(response.content)
```

Or use OpenAI:

```bash
pip install openai
```

```python
from openai import OpenAI

# Modify OpenAI's API key and API base to use SGLang's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
completion = client.completions.create(model="meta-llama/Meta-Llama-3.1-8B-Instruct",
                                      prompt="Hello, how are you?",
                                      max_tokens=50)
print("Completion result:", completion)
```

## Conclusions

Model inference is expensive, and leveraging more affordable compute/models makes a huge difference for engineering teams in terms of margins and shipping velocity.

Using SGLang on Vast is perfect for this, pairing Vast's access to affordable compute with the simplicity and state-of-the-art throughput of the SGLang backend.

SGLang is a great beginning to building Generative AI Apps. We will continue to explore using this tool more with Vast in future posts.

Llama-3 is already ready to go on Vast to start experimenting and building!


SGLang is an open-source framework for serving language models, focusing on throughput for serving and batch workloads.

Serving sglang on Vast


With all of the GPU options, finding the one that's right for your needs is challenging. Whether you're into high-end cloud gaming, training AI models, or even outfitting a university lab or data center, understanding the nuances between different models is key.

Today, we’re focusing on two powerful GPUs from NVIDIA: the L40 and [L40S](https://cloud.vast.ai/?gpu_option=L40S). While they may not have received as much attention as some of the company's other offerings, they certainly deserve a closer look! These two GPUs, built on the [Ada Lovelace](https://vast.ai/article/rtx-4090-deep-learning) architecture, pack a punch in performance and capabilities.

But how do they stack up against each other? What about compared to other GPUs? And more importantly, which one should you choose for your specific project needs?

## The Basics: NVIDIA L40 vs. L40S

The L40S is essentially an upgrade to the L40 – as well as a close relative to the [RTX 4090 gaming graphics card](https://vast.ai/article/nvidia-rtx-4090-vs-5090).

Initially designed for data center graphics and simulation tasks, the L40 is the engine of NVIDIA Omniverse. It's a powerhouse for extended (XR) and virtual reality (VR) applications, design collaboration, and digital twins, thanks to its advanced RTX and AI capabilities. It enables accelerated ray-traced and path-traced rendering, delivers physically accurate simulations, and generates highly realistic 3D synthetic data for the most demanding Omniverse workloads.

Compared to the previous generation, the L40 GPU delivers 5X higher inference performance for compute-intensive AI workloads – so it can generate high-quality images and immersive visual content lightning fast. It's perfectly suited for image generative AI applications. If that's all you need, then the L40 might be an ideal option for you.

However, the L40S can do all of this and more. It was developed to meet the surging demand for GPUs that can handle the intense computational requirements of machine learning training and inference.

NVIDIA describes the L40S as "the most powerful universal GPU," offering "breakthrough multi-workload performance." Here's a quick snapshot of its performance in a couple of areas:

!["Graph comparing NVIDIA L40S GPU performance for Stable Diffusion image generation and LLM inference, showing significant differences in images per minute and token latency across various configurations."](/uploads/l4-vs-l40S-image-generative-ai.png)

After its release in late 2023, the L40S achieved popularity due to the lack of availability of the NVIDIA A100 and H100 Tensor Core GPUs. There was a massive chip shortage and backlogged shipments, forcing many buyers to turn to other options – like the L40S. Since the L40S can not only be used for the same workloads as the L40 but can also power AI training and inference at a high level like the A100 and H100, it was a logical alternative.

The L40S, A100, and H100 GPUs are therefore in a unique position to be compared with each other. Let's take a look!

## L40S vs. A100 vs. H100: Specs and Performance

Each of the three GPUs brings its own strengths to the table. The table below compares various features and specs between them.

| Feature                | A100          | L40S           | H100          |
|------------------------|---------------|----------------|---------------|
| **GPU Architecture**   | Ampere        | Ada Lovelace   | Hopper        |
| **GPU Memory**         | 40 or 80 GB HBM2e | 48 GB GDDR6    | 80 MB HBM3    |
| **GPU Memory Bandwidth** | 1.6 to 2 TB/s | 864 GB/s       | 3.35 TB/s     |
| **CUDA Cores**         | 6912          | 18,176         | 14,592        |
| **FP64 TFLOPS**        | 9.7           | N/A            | 33.5          |
| **FP32 TFLOPS**        | 19.6          | 91.6           | 67            |
| **TF32 Tensor Core Flops** | 156 | 312 | 183 | 366 | 378 | 756 |
| **FP16 Tensor Core Flops** | 312 | 624 | 362 | 733 | 756 | 1513 |
| **FP8 Tensor Core TFLOPS** | N/A         | 1466           | 3958          |
| **Peak INT8 TOPS**     | 624 | 1248   | 733 | 1466      | 3958          |
| **Media Engine**       | 0 NVENC 5 NVDEC 5 NVJPEG | 0 NVENC 5 NVDEC 5 NVJPEG | 0 NVENC 7 NVDEC 7 NVJPEG |
| **L2 Cache**           | 40 MB         | 96 MB          | 50 MB         |
| **Power**              | Up to 400 W   | Up to 350 W    | Up to 700 W   |
| **Form Factor**        | SXM4 - 8-way HGX | Dual Slot Width | SXM5 - 8-way HGX |
| **Interconnect**       | PCIe 4.0 x16  | PCIe 4.0 x16   | PCIe 5.0 x16  |

*Without and with structured sparsity.

As shown, there are some clear differences in performance when it comes to FP64 (double-precision), FP32 (single-precision), and FP16 (half-precision) computations.

For instance, the L40S does not natively support FP64. This means that it may not perform as well as the A100 and H100 in applications that require high precision. The H100 in particular, with its notably superior FP64 performance, is generally preferred for the most demanding tasks.

In terms of FP32 and FP16 Tensor Core performance, the L40S outperforms the A100 40GB and can hold its own against the H100. However, in memory-intensive machine learning scenarios, the L40S's lower memory bandwidth may counterbalance its performance compared to both the A100 80GB and H100.

Still, the L40S can really shine next to the A100 in certain situations. According to NVIDIA, when it comes to complex AI workloads involving billions of parameters and multiple data modalities, including text and video, the L40S can [achieve](https://nvidianews.nvidia.com/news/nvidia-global-data-center-system-manufacturers-to-supercharge-generative-ai-and-industrial-digitalization) up to 1.2 times the generative AI inference performance and up to 1.7 times the training performance of the A100.

Another point in the L40S's favor? While it may not be able to beat the H100 on certain metrics, it's definitely more flexible. The higher-performance H100 can't be used as an Omniverse server and doesn't support graphics, whereas the L40S can be and does.

## The Bottom Line


The NVIDIA L40 excels in powering image generative AI applications and Omniverse workloads. You may not need it for much else, and if that's the case, the L40 would probably serve you well.

On the other hand, the NVIDIA L40S nicely balances versatility as well as performance. The following are some reasons you might choose it for your next project:

- **You have multi-modal workloads** – You'll achieve remarkable efficiency with the L40S, enabling you to run HPC simulations, train AI models, and render images using the same computing infrastructure. Plus, video output is built in.
- **You need compute power fast in a familiar form factor** – The L40S is designed for rapid deployment within existing systems, offering powerful performance without requiring extensive reconfiguration. Its Dual Slot PCIe form factor ensures compatibility with a wide range of setups.
- **Your workloads don't require extreme precision or maximum speed** – Without the math performance (FLOPS), high-bandwidth memory, and NVLINK of the H100, the L40S's inference and training performance for smaller models is nonetheless perfectly suitable in most cases.
- **You prioritize cost-effectiveness** – The L40S provides an excellent price-to-performance ratio, particularly for enterprises and research institutions that need robust AI capabilities without breaking the budget.

For many of us, cost is a major factor in [GPU decision-making](https://vast.ai/pricing). At Vast.ai, our mission is to democratize AI and bring massive compute power, at an affordable price, to anyone who needs it. You don't have to make a huge hardware investment upfront.

Our cloud GPU rental marketplace offers the lowest prices possible, enabling you to access top-tier machines at any time – and save 5-6X on GPU compute. We offer low-cost on-demand pricing as well as interruptible instances through spot auction-based pricing to bring you even more savings.

Come explore what Vast can do for you and accelerate your projects today!

Blog

June 2024 Product Update

Welcome back to our monthly check-in here at Vast.ai!

Bug Fixes

Cloud GPU Rental Platform Updates

Change Log

Bug Fixes:

Updated Features:

Continue Reading: