
For both technology enthusiasts and industry professionals, keeping a close eye on the latest GPU advancements isn't just a hobby – it's crucial for staying competitive and maximizing your performance and compute power. Here at Vast.ai, we're just as excited to witness the next big leaps forward in graphics technology!

Today, we're looking at two GPUs in particular: the formidable NVIDIA RTX 4090 versus its highly anticipated successor, the not-yet-released RTX 5090. While the RTX 4090 has solidified its reputation for performance, the RTX 5090 is still a subject of much speculation.

As we explore what we know and what we anticipate, keep in mind our knowledge of the RTX 5090 is still evolving. But nonetheless, let's take a look at the facts, the early insights, and emerging rumors…

### Spotlight on NVIDIA GeForce RTX 4090

Known for being the fastest consumer-grade GPU on the market, the RTX 4090 is truly a beast of a machine. Based on the AD102 chip using the Ada Lovelace architecture, it epitomizes super-high-end gaming and professional performance.

The RTX 4090 has a triple-slot design and boasts an incredible 128 dedicated third-gen RT cores and 512 fourth-gen Tensor cores – more than its closest competitors – elevating ray-tracing performance and speeding up AI-driven tasks like DLSS 3 in games.

Plus, it's packed with 16,384 CUDA cores and has 24GB of GDDR6X graphics memory with a maximum bandwidth of 1,008 GB/s and a 384-bit memory bus interface. Error-correcting code (ECC) can be enabled, but this reduces performance while making no difference in GPU rendering – so keeping it turned off is appropriate for most uses when you're prioritizing speed.

Due to its immense processing capability, the RTX 4090 draws a lot of power to deliver maximum performance: 450 watts. It requires an 850-watt power supply and robust cooling solutions, which does increase usage costs, of course.

Launched in September 2022, the RTX 4090 still currently stands as the pinnacle of NVIDIA’s offerings for enthusiasts seeking unmatched graphics performance and computational power.

But that will change in due time.

### Anticipating NVIDIA GeForce RTX 5090

The RTX 5090 is expected to be released later this year – and it's a game-changer, poised to redefine high-end performance with its groundbreaking features and next-gen architecture. That's right, we're now moving beyond NVIDIA's Ada Lovelace and into the era of the Blackwell architecture.

Joining the new Blackwell GPU lineup, the flagship GeForce RTX 5090 will reportedly utilize the "physically monolithic" GB202 die. Among its highlights are a potential 512-bit memory bus and up to 32GB of next-gen GDDDR7 memory (the successor to the GDDR6/X memory used by the RTX 40-series GPUs). Early reports suggest the RTX 5090 could achieve memory bandwidths of up to 1,532 GB/s, a substantial boost from the RTX 4090's 1,008 GB/s​.

Some are expecting to see up to 192 streaming multiprocessors (SMs) in the RTX 5090, with an unprecedented 24,576 CUDA cores – assuming the GB202 chip keeps the 128-cores-per-SM design as seen in the RTX 4090. Plus, rumors say the RTX 5090 will feature 192 third-gen RT cores and 768 fourth-gen Tensor cores, up from 128 and 512 respectively in the RTX 4090.

The RTX 5090 is anticipated to deliver 50-70% performance improvements across the board compared to the RTX 4090 – particularly excelling in 4K resolution. And with 4K 240FPS gaming on the horizon, the RTX 5090 will be perfectly positioned to tackle the most demanding gaming scenarios.

Some are also looking forward to seeing the RTX 5090 support next-gen DLSS 4, NVIDIA's AI-powered upscaling that will further enhance performance and image quality​.

In terms of power consumption, the RTX 5090 will likely follow the trend of its predecessors, with total graphics power (TGP) ranging from 450W to 600W, about on par with the RTX 4090.

For a quick look at some of the anticipated specs of the RTX 5090 compared to the RTX 4090 – based on the latest speculation and leaks – here's a table highlighting the upgrades:

| Spec                   | RTX 4090   | RTX 5090   |
|------------------------|------------|------------|
| Streaming Multiprocessors | 128        | 192        |
| CUDA Cores             | 16,384     | 24,576     |
| Ray-Tracing Cores      | 128        | 192        |
| Tensor Cores           | 512        | 768        |
| Boost Clock            | 2.52 GHz   | 2.9 GHz    |
| L2 Cache               | 72MB       | 128MB      |
| Memory Bandwidth       | 1,008 GB/s | 1,532 GB/s |

### Concluding Thoughts

As we anticipate the release of the RTX 5090, it's important to recognize that the RTX 4090 already delivers exceptional performance, handling everything available at 4K with ease. The RTX 5090, while sure to be extremely impressive, may be seen more as a luxury upgrade rather than a necessity for gamers and professionals who can (or already do) benefit from the power of the RTX 4090.

Cost is another critical factor. The RTX 5090 will undoubtedly be a top-of-the-line machine with a matching price tag. And the RTX 4090 is no slouch, either, still commanding a steep price to this day, nearly two years after its release.

Here at Vast.ai, we take pride in providing access to powerful machines for rent on the cloud at unbeatable prices. GPU rental offers unmatched flexibility, enabling you to utilize state-of-the-art technology without the expense and upkeep of purchasing your own hardware. Whether you're looking to explore the capabilities of the RTX 4090, prepare for the arrival of the RTX 5090, or check out any other machine on Vast, we'd be happy to help you discover the possibilities!


The RTX 4099 is a very powerful card that is good for both gaming and professional use. The RTX 5090 is expected to be even more powerful and will likely be more expensive.

NVIDIA RTX 4090 vs. the Upcoming RTX 5090


# Monthly Updates

We're pleased to share this month's updates, as we continuously refine our GPU rental platform to enhance the customer experience. At Vast.ai, our focus is on improving every day, ensuring that we meet your needs with the utmost efficiency and reliability at all times.

Though occasional bugs are inevitable, addressing them quickly is one of our top priorities. Here are some of the latest bug fixes we’ve rolled out since our last update:

- Fixed issue with template editor crashing
- Double reserve deposit warning
- Confirmation of host account conversion
- Fixed earnings page inconsistencies and display errors
- Resending verification email works again
- SSH regex accepts more valid key formats
- CLI geo location filter fixed

## Cloud GPU Rental Platform Updates

At Vast.ai, we're not just fixing bugs – we're also making strides in upgrading our platform to serve you better. This month, we've made some enhancements aimed at simplifying your experience and boosting performance. Here's a quick look at some key changes:

- Save template button added
- New GPU filter for the search page with UX improvements
- GPU select now saves local storage
- Added "today" button for selecting start date

As always, you can check out the full change log for our May 2024 product updates at the bottom of this page to discover all the enhancements we've made this month.

We’re committed to providing a top-tier, cost-effective GPU rental marketplace for both renters and hosts here at Vast.ai. Whether you’re just starting out or looking for ways to optimize your existing projects, we’re here to help every step of the way. Connect with our knowledgeable team via the chatbox at the bottom right, send us an email at [support@vast.ai](mailto:support@vast.ai), or join our engaging Discord community for additional support and resources. We'd be happy to assist you in making the most of what Vast.ai has to offer!

## Change Log

### Features:

- Save template button
- New GPU filter for the search page
- GPU select now saves local storage

### Bug Fixes:

- PDF invoice date range fix
- Fixed issue with template editor crashing
- Exported invoice dates, timezones, SSH key validation & cpu_ghz filter
- Cloud copy errors properly cleared
- Double reserve deposit warning
- Price breakdown disk value display corrected
- Confirm host account conversion
- Fixed earnings page inconsistencies and display errors
- Do not show reserved discounts on unverified search offers
- Resending verification email works again
- SSH regex accepts more valid key formats
- Instance IP modal scroll issues fixed
- Instance count now accurate
- Occupancy filter
- Price breakdown disk value
- Negative credit properly shows in header
- CLI geo location filter fixed
- CLI search offers "gpu_frac" unrecognized
- Removed newline from CLI command results show instances

### Miscellaneous Tasks:

- Today button for start date
- Typos
- Account key warning
- Instance label sort
- Clearer cloud copy errors
- UX improvements for GPU filter
- Formatting and warnings


May 2024 Product Update


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support! 

With this latest update, we're welcoming both the Radeon and Instinct product lines from AMD to the existing array of NVIDIA machines available for rent on our marketplace. As we widen the range of supported hardware, we can help connect even more customers with the specific GPU compute requirements they're looking for. 

Here are the [AMD GPUs we have available now](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76).

## Now Supporting AMD Instinct and Radeon GPUs

The Instinct line is AMD's brand of data center GPUs, intended to accelerate deep learning, artificial neural network, and high-performance computing tasks, while the Radeon line is aimed at gaming and consumer-grade applications. With both now supported on Vast, this expansion means more versatility and options for all of our users.

Travis Cannell, COO at Vast.ai, highlights the strategic advantage of the update: "We are interested in partnering with managed service providers who have the Instinct line and who want to increase the utilization of their clusters, as well as the Radeon line for as our users have needs for remote gaming, rendering and more." 

Ultimately, our platform will now be better equipped to meet the diverse needs and preferences of our user base – from data centers to gamers and everyone in between. Whether you're scaling up operations or just starting out, we're here to support your computing journey.

## How to Get Started

To browse the active AMD listings on cloud.vast.ai, you will need to select an AMD template which will then filter results for AMD GPUs. Renters looking to leverage these new GPU options on Vast should use the ROCm template. The two templates supported now include an Ubuntu 22.04 image and a PyTorch image.

To see the active results, select the desired template:

- [AMD Ubuntu 22.04 ROCm 6.02](https://cloud.vast.ai/?ref_id=43484&template_id=9e190d88664e20818ee75eb33dce943c)
- [AMD ROCm PyTorch](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76)

You can also find the latest AMD templates by looking for AMD on the [template page](https://cloud.vast.ai/templates/).

![Vast.ai AMD Templates](/uploads/amd-templates.PNG)


## How to Host AMD Equipment

For those interested in hosting their GPUs, Vast.ai provides comprehensive instructions in our [Machine Hosting Setup guide](https://cloud.vast.ai/host/setup/). 

Whether you're setting up an AMD or NVIDIA machine, the step-by-step guide gives you all the information you need to get started. From system requirements and networking configuration to troubleshooting and how to list your machine, learn to optimize your hosting setup and get up and running as soon as possible.

## Leading the Way in GPU Rental Innovation

The addition of support for AMD GPUs comes at a time when Vast.ai has announced an incredible 265% year-over-year growth since 2019. This growth illustrates the heavy demand in the GPU rental market. With more than 350 independent hosts supplying 16,000+ GPUs – and growing – Vast.ai efficiently matches GPU supply and demand, ensuring that every user finds the best deal for their specific compute needs.

As Vast.ai leads the charge in GPU rental innovation, the addition of AMD support is more than just an upgrade – it's also a reflection of our commitment to the democratization of AI. Join us as we continue to make powerful computing more accessible to everyone.


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support!

Announcing AMD Support


Just this past year, NVIDIA introduced a groundbreaking security feature in the Hopper architecture to protect sensitive data, AI models, and applications in use. The new feature, called NVIDIA Confidential Computing, offers a hardware-based solution for securely processing data and code that is in use, preventing unauthorized access and modification. 

  

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible – while you also get to enjoy the cutting-edge acceleration of NVIDIA H200 and H100 Tensor Core GPUs.

  

### What Is NVIDIA Confidential Computing?

  

To put it simply, "confidential computing" (CC) protects data-in-use by performing computation in a trusted execution environment (TEE) that is both hardware-based and attested. 

  

The NVIDIA H200 and H100 GPUs have a CVM (confidential virtual machine) TEE on the CPU that is anchored in an on-die root of trust (RoT). When it boots in CC-On mode, hardware protections that provide confidentiality and integrity of code and data are enabled. (In this context, "confidentiality" means that an attacker cannot access any data or code, and "integrity" means that they cannot modify the execution.) 

  

Here's how it works:

  

1. A chain of trust is established via a GPU boot sequence with a secure and measured boot.
    

  

2. An SPDM (Security Protocol and Data Model) session facilitates a secure connection to the driver in a CPU TEE. 
    

  

3. An attestation report with a cryptographically signed set of measurements is generated, verifying system integrity.
    

  

In order to run a confidential workload, the GPU must be authenticated as a legitimate NVIDIA GPU that supports confidential computing (a process involving a device-unique private key fused onto each GPU corresponding to a certified public key); the GPU must not have been revoked for CC; and the GPU attestation report must be verified. 

  

Upon successful verification, the NVIDIA driver in the CVM establishes a secure channel, via a session key, with the GPU hardware TEE in order to transfer data, perform computation, and retrieve results. The CVM and GPU communicate through a shared memory region outside the CVM since the CPU prevents the GPU from directly accessing a CVM's memory. They use AES-GCM – an advanced encryption standard – to prevent the host system from reading this communication. 

  

Finally, the GPU copies and decrypts all inputs to its internal memory, where everything runs in plaintext and direct access is blocked. Performance counters are disabled as well, since these could be used for side-channel attacks. 

  

In this way, a confidential-computing environment is extended from a CVM (or a secure enclave) to a GPU, with attestation, encrypted communication, and memory isolation making it possible. 

  

### Benefits & Positive Impacts

  

The benefits of NVIDIA Confidential Computing are significant:

  

- Hardware-Based Security and Isolation – You can achieve full isolation of virtual machines (VMs) in any environment – on-premises, at the edge, or on the cloud. Your entire workload is secured with these built-in hardware firewalls that provide unprecedented protection. 
    

  

- Verifiability with Device Attestation – You will always rest assured that only authorized end users can deploy data and code for execution in the H200 or H100's TEE. Plus, through device attestation, you'll know that you're dealing with an authentic NVIDIA GPU, the firmware hasn't been modified, and updates have been done as expected. 
    

  

- Protection from Unauthorized Access – Your sensitive data, AI workloads, and intellectual property are protected, with confidentiality and integrity preserved at all times. All unauthorized entities (including the hypervisor, cloud provider, host operating system, and even anyone with physical access to the infrastructure) are blocked from viewing or modifying the AI application and data during execution. 
    

  

- No Application Code Change – As NVIDIA puts it, the new confidential computing feature "just works," without any code changes required for your GPU-accelerated workloads. 
    

  

As you can imagine, confidential computing-enabled GPUs usher in a whole host of use cases where security, privacy, and regulatory compliance are paramount. For instance, intellectual property protection is now possible for proprietary AI models, even when these models are distributed and deployed at scale on shared or remote infrastructure. 

  

Privacy and confidentiality can also now be preserved in AI training and inference – most notably in industries like healthcare, finance, and the public sector, where data may be sensitive and/or regulated. On top of that, collaboration between multiple parties can be done securely when building and improving AI models across participating sites, for use cases like medical imaging, drug development, and fraud detection. 

  

Another bonus: organizations, governments, and individuals can outsource AI workloads to a cloud provider while still ensuring protection from the underlying infrastructure. 

  

### Final Thoughts

  

It's exciting that confidential computing is now possible for even the most demanding workloads! Granted, for the vast majority of use cases, CC-enabled GPUs may arguably be overkill. But for those who need or prefer absolute confidentiality and security, NVIDIA Confidential Computing in the Hopper architecture can now provide that for you. 

  

At Vast.ai, we understand how important privacy and security can be for our customers. It is our hope that you'll find exactly what you need on our cost-effective, on-demand cloud GPU rental platform. 

  

In our global network of compute providers on Vast, some hosts are experimenting with NVIDIA Confidential Computing. Complete support for this feature is not yet available. Join our Discord for the latest product updates and up-to-date support.

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible 

Absolute Security with NVIDIA Confidential Computing


# Train a 70b language model on a 2X RTX 4090 with QLoRA and FSDP

## Overview

A new exciting [announcement from Answers.ai](https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html) demonstrated a way to train a larger model, such as Llama 2 70B on 48GB of GPU RAM. Furthermore it can run on multiple GPUS, so it is possible to train a model on a 2X 4090 instance!

The new [fsdp_qlora](https://github.com/AnswerDotAI/fsdp_qlora/tree/main) Open Source software has a number of training options and examples. The software builds on two big ideas:

**QLoRA (Quantized LoRA)** combines quantization (using fewer bits to store model weights) and LoRA (Low-Rank Adaptation, which adds small trainable matrices to a frozen base model). This allows training models larger than the GPU's memory by using a quantized base model with trainable LoRA adapters. However, QLoRA still has limitations, such as requiring expensive GPUs and restricting sequence lengths and batch sizes due to memory constraints.

**FSDP (Fully Sharded Data Parallel)** is a library developed by Meta that efficiently splits a large model across multiple GPUs, allowing them to be used simultaneously. It improves upon the previous gold standard, DDP (Distributed Data Parallel), which required the full model to fit on each GPU. FSDP enables training large models that exceed the memory of a single GPU by sharding the model parameters across multiple GPUs.

This guide will show you how to rent a 2X instance on Vast.ai to run this software on-demand. 

## How to run fsdp_qlora on a Vast.ai 2X 3090/4090 instance

Vast.ai has a large supply of RTX 3090 and RTX 4090 GPUs well suited to run fsdp_qlora. You will need a Vast.ai account setup with credits. This guide uses the [Vast Python CLI](https://github.com/vast-ai/vast-python). 

### Create your Vast account, add credits and install the Vast CLI on your local machine. 

Setup the Vast.ai account and verify your email address. Add some credits to your account for the GPU rental. Then [install the CLI](https://cloud.vast.ai/cli/). 

### Search the Vast.ai marketplace

We will need to find suitable GPUs. From my testing, for training Llama 70B it required 200GB of system RAM. This query will return on-demand GPU offers for suitable 2X 4090 instances sorted by download bandwidth

For 2X 4090 instances
```
vastai search offers "gpu_name=RTX_4090 cpu_ram>=130 disk_space>140 num_gpus=2" -o "inet_down-"
```

For 2X 3090 instances 
```
vastai search offers "gpu_name=RTX_3090 cpu_ram>=130 disk_space>140 num_gpus=2" -o "inet_down-"
```

#### Create the Vast.ai instance

The list of offers will have a lot of details about the machines that are available. To pick one, you will need the offer id. Then use that in the create command along with the Pytorch Cuda Devel template hash.

This command will create an instance on the offer supplied, using that template and allocating 200GB of disk.

Replace the XXXXXXX with the offer ID from the marketplace.

```
vastai create instance XXXXXXX --template_hash e4c5e88bc289f4eecb0c955c4fe7430d --disk 200
```

#### SSH into the Vast.ai instance

Wait a few minutes for the instance to boot up. You can use the [webui](https://cloud.vast.ai/instances/) to get the direct SSH command by clicking on the >_ button.

To connect via SSH, you will simply use the command copied from the instance card. It will look similar to this:

```
ssh -p <port> <sshURL> -L 8080:localhost:8080
```

#### Install fsdp_qlora and login to Hugging Face

```
git clone https://github.com/AnswerDotAI/fsdp_qlora
pip install llama-recipes fastcore --extra-index-url https://download.pytorch.org/whl/test/cu118
pip install bitsandbytes>=0.43.0
pip install wandb
```

Login to Hugging Face by pasting in your API key to download the model

```
huggingface-cli login
```

Optional: login to wandb to enable logging to Weights and Balances

```
wandb login
```

#### Optionally setup HQQ

HQQ is a fast and accurate model quantizer. It can be used instead of bitsandbytes. If you want to use it, you will need to install it first.

```
git clone https://github.com/mobiusml/hqq.git
```

Install HQQ

```
cd hqq && pip install .
```

#### Train the model

For complete options look at the [Github repo](https://github.com/AnswerDotAI/fsdp_qlora/tree/main). This example will finetune Llama 70B using hqq_lora.


#### Example: finetune Llama 2 70B with context length of 512

```
cd ~/fsdp_qlora
```

```
python train.py \
--model_name meta-llama/Llama-2-70b-hf \
--batch_size 2 \
--context_length 512 \
--precision bf16 \
--train_type qlora \
--use_gradient_checkpointing true \
--use_cpu_offload true \
--dataset alpaca \
--reentrant_checkpointing true
```

#### Troubleshooting

On some instances, the system seems to only run on the CPU. The GPUs might not being recognized. Monitor the [fsdp_qlora repo](https://github.com/AnswerDotAI/fsdp_qlora/tree/main) for help.

For any issues related to the instance or working with Vast.ai, click on the website support chat at the bottom right corner of [cloud.vast.ai](https://cloud.vast.ai/) for immediate support.

Sometimes it can help to export the cuda visible devices to help with an issue where the GPUs do not load and the CPU does all the work.

```
export CUDA_VISIBLE_DEVICES=0,1
```

A new approach to use both QLoRA and FSDP to do full training on a 2X 3090/4090

Industry

NVIDIA RTX 4090 vs. the Upcoming RTX 5090

May 2024 Product Update

Announcing AMD Support

Absolute Security with NVIDIA Confidential Computing

Train a 70b language model on a 2X RTX 3090/4090 with QLoRA and FSDP