
# Monthly Updates

We're pleased to share this month's updates, as we continuously refine our GPU rental platform to enhance the customer experience. At Vast.ai, our focus is on improving every day, ensuring that we meet your needs with the utmost efficiency and reliability at all times.

Though occasional bugs are inevitable, addressing them quickly is one of our top priorities. Here are some of the latest bug fixes we’ve rolled out since our last update:

- Fixed issue with template editor crashing
- Double reserve deposit warning
- Confirmation of host account conversion
- Fixed earnings page inconsistencies and display errors
- Resending verification email works again
- SSH regex accepts more valid key formats
- CLI geo location filter fixed

## Cloud GPU Rental Platform Updates

At Vast.ai, we're not just fixing bugs – we're also making strides in upgrading our platform to serve you better. This month, we've made some enhancements aimed at simplifying your experience and boosting performance. Here's a quick look at some key changes:

- Save template button added
- New GPU filter for the search page with UX improvements
- GPU select now saves local storage
- Added "today" button for selecting start date

As always, you can check out the full change log for our May 2024 product updates at the bottom of this page to discover all the enhancements we've made this month.

We’re committed to providing a top-tier, cost-effective GPU rental marketplace for both renters and hosts here at Vast.ai. Whether you’re just starting out or looking for ways to optimize your existing projects, we’re here to help every step of the way. Connect with our knowledgeable team via the chatbox at the bottom right, send us an email at [support@vast.ai](mailto:support@vast.ai), or join our engaging Discord community for additional support and resources. We'd be happy to assist you in making the most of what Vast.ai has to offer!

## Change Log

### Features:

- Save template button
- New GPU filter for the search page
- GPU select now saves local storage

### Bug Fixes:

- PDF invoice date range fix
- Fixed issue with template editor crashing
- Exported invoice dates, timezones, SSH key validation & cpu_ghz filter
- Cloud copy errors properly cleared
- Double reserve deposit warning
- Price breakdown disk value display corrected
- Confirm host account conversion
- Fixed earnings page inconsistencies and display errors
- Do not show reserved discounts on unverified search offers
- Resending verification email works again
- SSH regex accepts more valid key formats
- Instance IP modal scroll issues fixed
- Instance count now accurate
- Occupancy filter
- Price breakdown disk value
- Negative credit properly shows in header
- CLI geo location filter fixed
- CLI search offers "gpu_frac" unrecognized
- Removed newline from CLI command results show instances

### Miscellaneous Tasks:

- Today button for start date
- Typos
- Account key warning
- Instance label sort
- Clearer cloud copy errors
- UX improvements for GPU filter
- Formatting and warnings


May 2024 Product Update


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support! 

With this latest update, we're welcoming both the Radeon and Instinct product lines from AMD to the existing array of NVIDIA machines available for rent on our marketplace. As we widen the range of supported hardware, we can help connect even more customers with the specific GPU compute requirements they're looking for. 

Here are the [AMD GPUs we have available now](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76).

## Now Supporting AMD Instinct and Radeon GPUs

The Instinct line is AMD's brand of data center GPUs, intended to accelerate deep learning, artificial neural network, and high-performance computing tasks, while the Radeon line is aimed at gaming and consumer-grade applications. With both now supported on Vast, this expansion means more versatility and options for all of our users.

Travis Cannell, COO at Vast.ai, highlights the strategic advantage of the update: "We are interested in partnering with managed service providers who have the Instinct line and who want to increase the utilization of their clusters, as well as the Radeon line for as our users have needs for remote gaming, rendering and more." 

Ultimately, our platform will now be better equipped to meet the diverse needs and preferences of our user base – from data centers to gamers and everyone in between. Whether you're scaling up operations or just starting out, we're here to support your computing journey.

## How to Get Started

To browse the active AMD listings on cloud.vast.ai, you will need to select an AMD template which will then filter results for AMD GPUs. Renters looking to leverage these new GPU options on Vast should use the ROCm template. The two templates supported now include an Ubuntu 22.04 image and a PyTorch image.

To see the active results, select the desired template:

- [AMD Ubuntu 22.04 ROCm 6.02](https://cloud.vast.ai/?ref_id=43484&template_id=9e190d88664e20818ee75eb33dce943c)
- [AMD ROCm PyTorch](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76)

You can also find the latest AMD templates by looking for AMD on the [template page](https://cloud.vast.ai/templates/).

![Vast.ai AMD Templates](/uploads/amd-templates.PNG)


## How to Host AMD Equipment

For those interested in hosting their GPUs, Vast.ai provides comprehensive instructions in our [Machine Hosting Setup guide](https://cloud.vast.ai/host/setup/). 

Whether you're setting up an AMD or NVIDIA machine, the step-by-step guide gives you all the information you need to get started. From system requirements and networking configuration to troubleshooting and how to list your machine, learn to optimize your hosting setup and get up and running as soon as possible.

## Leading the Way in GPU Rental Innovation

The addition of support for AMD GPUs comes at a time when Vast.ai has announced an incredible 265% year-over-year growth since 2019. This growth illustrates the heavy demand in the GPU rental market. With more than 350 independent hosts supplying 16,000+ GPUs – and growing – Vast.ai efficiently matches GPU supply and demand, ensuring that every user finds the best deal for their specific compute needs.

As Vast.ai leads the charge in GPU rental innovation, the addition of AMD support is more than just an upgrade – it's also a reflection of our commitment to the democratization of AI. Join us as we continue to make powerful computing more accessible to everyone.


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support!

Announcing AMD Support


Just this past year, NVIDIA introduced a groundbreaking security feature in the Hopper architecture to protect sensitive data, AI models, and applications in use. The new feature, called NVIDIA Confidential Computing, offers a hardware-based solution for securely processing data and code that is in use, preventing unauthorized access and modification. 

  

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible – while you also get to enjoy the cutting-edge acceleration of NVIDIA H200 and H100 Tensor Core GPUs.

  

### What Is NVIDIA Confidential Computing?

  

To put it simply, "confidential computing" (CC) protects data-in-use by performing computation in a trusted execution environment (TEE) that is both hardware-based and attested. 

  

The NVIDIA H200 and H100 GPUs have a CVM (confidential virtual machine) TEE on the CPU that is anchored in an on-die root of trust (RoT). When it boots in CC-On mode, hardware protections that provide confidentiality and integrity of code and data are enabled. (In this context, "confidentiality" means that an attacker cannot access any data or code, and "integrity" means that they cannot modify the execution.) 

  

Here's how it works:

  

1. A chain of trust is established via a GPU boot sequence with a secure and measured boot.
    

  

2. An SPDM (Security Protocol and Data Model) session facilitates a secure connection to the driver in a CPU TEE. 
    

  

3. An attestation report with a cryptographically signed set of measurements is generated, verifying system integrity.
    

  

In order to run a confidential workload, the GPU must be authenticated as a legitimate NVIDIA GPU that supports confidential computing (a process involving a device-unique private key fused onto each GPU corresponding to a certified public key); the GPU must not have been revoked for CC; and the GPU attestation report must be verified. 

  

Upon successful verification, the NVIDIA driver in the CVM establishes a secure channel, via a session key, with the GPU hardware TEE in order to transfer data, perform computation, and retrieve results. The CVM and GPU communicate through a shared memory region outside the CVM since the CPU prevents the GPU from directly accessing a CVM's memory. They use AES-GCM – an advanced encryption standard – to prevent the host system from reading this communication. 

  

Finally, the GPU copies and decrypts all inputs to its internal memory, where everything runs in plaintext and direct access is blocked. Performance counters are disabled as well, since these could be used for side-channel attacks. 

  

In this way, a confidential-computing environment is extended from a CVM (or a secure enclave) to a GPU, with attestation, encrypted communication, and memory isolation making it possible. 

  

### Benefits & Positive Impacts

  

The benefits of NVIDIA Confidential Computing are significant:

  

- Hardware-Based Security and Isolation – You can achieve full isolation of virtual machines (VMs) in any environment – on-premises, at the edge, or on the cloud. Your entire workload is secured with these built-in hardware firewalls that provide unprecedented protection. 
    

  

- Verifiability with Device Attestation – You will always rest assured that only authorized end users can deploy data and code for execution in the H200 or H100's TEE. Plus, through device attestation, you'll know that you're dealing with an authentic NVIDIA GPU, the firmware hasn't been modified, and updates have been done as expected. 
    

  

- Protection from Unauthorized Access – Your sensitive data, AI workloads, and intellectual property are protected, with confidentiality and integrity preserved at all times. All unauthorized entities (including the hypervisor, cloud provider, host operating system, and even anyone with physical access to the infrastructure) are blocked from viewing or modifying the AI application and data during execution. 
    

  

- No Application Code Change – As NVIDIA puts it, the new confidential computing feature "just works," without any code changes required for your GPU-accelerated workloads. 
    

  

As you can imagine, confidential computing-enabled GPUs usher in a whole host of use cases where security, privacy, and regulatory compliance are paramount. For instance, intellectual property protection is now possible for proprietary AI models, even when these models are distributed and deployed at scale on shared or remote infrastructure. 

  

Privacy and confidentiality can also now be preserved in AI training and inference – most notably in industries like healthcare, finance, and the public sector, where data may be sensitive and/or regulated. On top of that, collaboration between multiple parties can be done securely when building and improving AI models across participating sites, for use cases like medical imaging, drug development, and fraud detection. 

  

Another bonus: organizations, governments, and individuals can outsource AI workloads to a cloud provider while still ensuring protection from the underlying infrastructure. 

  

### Final Thoughts

  

It's exciting that confidential computing is now possible for even the most demanding workloads! Granted, for the vast majority of use cases, CC-enabled GPUs may arguably be overkill. But for those who need or prefer absolute confidentiality and security, NVIDIA Confidential Computing in the Hopper architecture can now provide that for you. 

  

At Vast.ai, we understand how important privacy and security can be for our customers. It is our hope that you'll find exactly what you need on our cost-effective, on-demand cloud GPU rental platform. 

  

In our global network of compute providers on Vast, some hosts are experimenting with NVIDIA Confidential Computing. Complete support for this feature is not yet available. Join our Discord for the latest product updates and up-to-date support.

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible 

Absolute Security with NVIDIA Confidential Computing


# Train a 70b language model on a 2X RTX 4090 with QLoRA and FSDP

## Overview

A new exciting [announcement from Answers.ai](https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html) demonstrated a way to train a larger model, such as Llama 2 70B on 48GB of GPU RAM. Furthermore it can run on multiple GPUS, so it is possible to train a model on a 2X 4090 instance!

The new [fsdp_qlora](https://github.com/AnswerDotAI/fsdp_qlora/tree/main) Open Source software has a number of training options and examples. The software builds on two big ideas:

**QLoRA (Quantized LoRA)** combines quantization (using fewer bits to store model weights) and LoRA (Low-Rank Adaptation, which adds small trainable matrices to a frozen base model). This allows training models larger than the GPU's memory by using a quantized base model with trainable LoRA adapters. However, QLoRA still has limitations, such as requiring expensive GPUs and restricting sequence lengths and batch sizes due to memory constraints.

**FSDP (Fully Sharded Data Parallel)** is a library developed by Meta that efficiently splits a large model across multiple GPUs, allowing them to be used simultaneously. It improves upon the previous gold standard, DDP (Distributed Data Parallel), which required the full model to fit on each GPU. FSDP enables training large models that exceed the memory of a single GPU by sharding the model parameters across multiple GPUs.

This guide will show you how to rent a 2X instance on Vast.ai to run this software on-demand. 

## How to run fsdp_qlora on a Vast.ai 2X 3090/4090 instance

Vast.ai has a large supply of RTX 3090 and RTX 4090 GPUs well suited to run fsdp_qlora. You will need a Vast.ai account setup with credits. This guide uses the [Vast Python CLI](https://github.com/vast-ai/vast-python). 

### Create your Vast account, add credits and install the Vast CLI on your local machine. 

Setup the Vast.ai account and verify your email address. Add some credits to your account for the GPU rental. Then [install the CLI](https://cloud.vast.ai/cli/). 

### Search the Vast.ai marketplace

We will need to find suitable GPUs. From my testing, for training Llama 70B it required 200GB of system RAM. This query will return on-demand GPU offers for suitable 2X 4090 instances sorted by download bandwidth

For 2X 4090 instances
```
vastai search offers "gpu_name=RTX_4090 cpu_ram>=130 disk_space>140 num_gpus=2" -o "inet_down-"
```

For 2X 3090 instances 
```
vastai search offers "gpu_name=RTX_3090 cpu_ram>=130 disk_space>140 num_gpus=2" -o "inet_down-"
```

#### Create the Vast.ai instance

The list of offers will have a lot of details about the machines that are available. To pick one, you will need the offer id. Then use that in the create command along with the Pytorch Cuda Devel template hash.

This command will create an instance on the offer supplied, using that template and allocating 200GB of disk.

Replace the XXXXXXX with the offer ID from the marketplace.

```
vastai create instance XXXXXXX --template_hash e4c5e88bc289f4eecb0c955c4fe7430d --disk 200
```

#### SSH into the Vast.ai instance

Wait a few minutes for the instance to boot up. You can use the [webui](https://cloud.vast.ai/instances/) to get the direct SSH command by clicking on the >_ button.

To connect via SSH, you will simply use the command copied from the instance card. It will look similar to this:

```
ssh -p <port> <sshURL> -L 8080:localhost:8080
```

#### Install fsdp_qlora and login to Hugging Face

```
git clone https://github.com/AnswerDotAI/fsdp_qlora
pip install llama-recipes fastcore --extra-index-url https://download.pytorch.org/whl/test/cu118
pip install bitsandbytes>=0.43.0
pip install wandb
```

Login to Hugging Face by pasting in your API key to download the model

```
huggingface-cli login
```

Optional: login to wandb to enable logging to Weights and Balances

```
wandb login
```

#### Optionally setup HQQ

HQQ is a fast and accurate model quantizer. It can be used instead of bitsandbytes. If you want to use it, you will need to install it first.

```
git clone https://github.com/mobiusml/hqq.git
```

Install HQQ

```
cd hqq && pip install .
```

#### Train the model

For complete options look at the [Github repo](https://github.com/AnswerDotAI/fsdp_qlora/tree/main). This example will finetune Llama 70B using hqq_lora.


#### Example: finetune Llama 2 70B with context length of 512

```
cd ~/fsdp_qlora
```

```
python train.py \
--model_name meta-llama/Llama-2-70b-hf \
--batch_size 2 \
--context_length 512 \
--precision bf16 \
--train_type qlora \
--use_gradient_checkpointing true \
--use_cpu_offload true \
--dataset alpaca \
--reentrant_checkpointing true
```

#### Troubleshooting

On some instances, the system seems to only run on the CPU. The GPUs might not being recognized. Monitor the [fsdp_qlora repo](https://github.com/AnswerDotAI/fsdp_qlora/tree/main) for help.

For any issues related to the instance or working with Vast.ai, click on the website support chat at the bottom right corner of [cloud.vast.ai](https://cloud.vast.ai/) for immediate support.

Sometimes it can help to export the cuda visible devices to help with an issue where the GPUs do not load and the CPU does all the work.

```
export CUDA_VISIBLE_DEVICES=0,1
```

A new approach to use both QLoRA and FSDP to do full training on a 2X 3090/4090

Train a 70b language model on a 2X RTX 3090/4090 with QLoRA and FSDP


When you need high-powered resources to power complex calculations, the more you have the better. In a recent interview, Elon Musk claimed that he heard of a gigawatt class data center being built. Many responders think that Elon is referring to a rumor that –deep in the deserts of Kuwait–, the construction of a new data center is being built to compete with today’s most powerful compute power resources.

Elon Musk likely heard the rumor from [news](https://www.theinformation.com/articles/ai-agenda-the-mysterious-ai-data-center-startup-hiring-from-aws-azure-meta) of a startup poaching technology workers from AWS, Azure, Meta, and other large Silicon Valley companies. In 2023, the race to be a leader in artificial intelligence and machine learning became even more competitive. To be the best, organizations need immense GPU power, usually outsourced to the cloud for its quick deployment and scaling. Large cloud providers built the data centers to power calculations, and businesses can rent the compute resources necessary to power any project – from small personal projects to large enterprise-tier analytics.

The rumor of a Kuwait data center came in the summer of 2023. A company named Omniva –also known as Moneta Tech, and Moneta Systems– was interested in building the most powerful data center in the world. Their goal was to rival Google and Microsoft in technology. Originally, the data center idea was targeted towards cryptomining, but later became an idea to target businesses innovating for the latest artificial intelligence boom.

As more information became available, the gigawatt claims began to unravel. The facility was said to be about 430,000 square feet with total available power of 864 megawatts. A proposed budget of $200 million immediately sent red flags. As a benchmark, it’s reported that a Google data center costs between $1000 and $3000 per square foot. This means that at the very cheapest, a data center of Omniva’s size should cost $430 million, over double the proposed budget.

Cooling systems were ignored for manned spaces in exchange for employee cooling helmets. Fire systems were removed in exchange for fire extinguishers. Anyone with data center experience knows that the proposed planning and safety aren’t feasible.

Next is Omniva’s claim to offer gigawatts of power to customers. NVIDIA continues to dominate GPU performance and offer the best in compute power. To reach the compute power rumored to be available, Omniva would need the latest generation of GPUs, namely the new NVIDIA Blackwell B100. The B100s haven’t hit the market yet, but NVIDIA teased consumers with the promise that their next generation GPUs should launch in 2024.

As of today, any facility rumored to be built with gigawatt power has never been confirmed. The power promised in 2023 wasn’t available, and many of the plans presented to the public didn’t have the backing of professionals. The cloud GPU space still has the computational power to support artificial intelligence and cryptomining, but businesses and consumers should stick to a trusted provider.

For the absolute best in compute power with the latest in GPU technology, [Vast.ai](https://vast.ai/) has the resources you need for machine learning, artificial intelligence, and other high-powered analytics projects for your cloud GPU needs.


Coming up with just the right words, and keeping track of the ideas that you’re weaving together - this is difficult work. The difficulty can even make it seem that there might be better ways to be spending your time.

Is a Gigawatt Class Data Center Being Built in Kuwait?


The landscape of high-performance GPUs is continually evolving, particularly in the field of AI and ML. With the recent introduction of AMD's Radeon RX 7800 XT and Radeon RX 7700 XT, there has been significant interest in how these GPUs compare with established models like NVIDIA's RTX 4090, especially for AI applications. This article aims to provide a detailed comparison between these GPUs, focusing on their technical specifications, performance metrics, and suitability for AI tasks.

# AI Suitability of the AMD Radeon RX 7800 XT and Radeon RX 7700 XT

Unveiled in 2023, the Radeon RX 7800 XT and Radeon RX 7700 XT have piqued the interest of professionals in the AI field. While initially geared towards high-resolution gaming, these GPUs also hold potential for AI applications.

## AMD Radeon RX 7800 XT

This model boasts 60 AMD RDNA 3 unified compute units and offers significant performance-per-dollar value. Its high-speed 16GB GDDR6 VRAM and a game clock speed of 2124 MHz (boosting up to 2430 MHz) make it a competent choice for AI tasks that require substantial memory and computational power.

## AMD Radeon RX 7700 XT

With 54 AMD RDNA 3 compute units and 12GB of GDDR6 VRAM, the 7700 XT, while slightly less powerful than the 7800 XT, still offers robust performance for AI applications. Its game clock of 2171 MHz and boost clock up to 2544 MHz ensure efficient processing for complex algorithms.

Both GPUs leverage the advanced AMD RDNA 3 Architecture, including unified ray tracing and AI accelerators. This architecture is particularly beneficial for machine learning models that involve image processing and deep learning computations. However, given that Radeon cards primarily focus on gaming, the NVIDIA RTX 4090 is still a stronger candidate for AI Applications.

# The NVIDIA RTX 4090 Still Stands as a Versatile Choice

The [NVIDIA RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/) remains a solid choice for a variety of users for AI purposes, despite newer models like AMD's Radeon RX 7800 XT and RX 7700 XT. It is a well-established GPU in the AI community, known for its exceptional computational power and efficiency. It also features NVIDIA's Deep Learning Super Sampling (DLSS), an AI technology that enhances image quality without overburdening the GPU.

In terms of energy efficiency, the RTX 4090 is designed to provide high performance without excessive power consumption. Its effective cooling system also ensures it operates at optimal temperatures, which is key for long-term use. The GPU's compatibility with a wide range of software and hardware adds to its versatility, making it a suitable choice for a wide variety of applications.

Considering its advanced features and future-proof design, the RTX 4090 is a good long-term investment for those needing a GPU that can handle both current and upcoming software. While AMD’s Radeon RX 7800 XT and RX 7700 XT have their specific strengths, the RTX 4090's broad capabilities make it a reliable and efficient choice in the GPU market. However, these GPUs can be hard to find, or incredibly costly due to high demand and low supply.

# Leveraging Vast.ai for GPU Accessibility

While GPUs like NVIDIA’s RTX 4090 offer impressive capabilities, their availability and cost can be prohibitive for many users. This is where [Vast.ai](http://vast.ai) comes into play, providing a solution to these challenges.

## Vast.ai: A Solution for Easier GPU Access

Vast.ai is a platform that allows users to rent high-performance GPUs, like the RTX 4090, at more affordable rates. This service is particularly useful for those who need GPU power for specific projects or temporary tasks but find purchasing such hardware impractical due to cost or limited usage.

On the Vast.ai platform, users can find a range of GPUs available for rent, including the latest models. The process is straightforward: users search for the GPU that fits their needs and budget, and then rent it for the required period. This approach eliminates the need for a significant upfront investment in purchasing a high-end GPU.

## Why Use Vast.ai?

The primary advantage of using Vast.ai is cost-effectiveness. It allows individuals, small businesses, and researchers access to top-tier GPUs for their computational needs without the full cost of ownership. This flexibility is perfect for those who require advanced GPU capabilities for short-term projects or occasional use.

# Final Thoughts

In summary, when comparing NVIDIA's RTX 4090 with AMD's Radeon RX 7800 XT and 7700 XT for AI applications, NVIDIA is a more catered choice. The RTX 4090 stands out for its proven performance and advanced features, making it ideal for demanding AI tasks. The AMD options are more gaming-oriented, and while their potential for AI applications is solid, a dedicated AI GPU may be more appropriate. If you’re in need of a GPU for AI development, services like Vast.ai offer the RTX 4090, along with other high-end GPUs, allowing for the development of AI without the hefty investment in hardware.


High-Performance GPUs for AI: Nvidia RTX 4090 vs AMD Radeon RX 7800 XT and 7700 XT 