
# Harness the Power of Cloud GPUs with Vast.AI: Running the 70B LLama2 GPTQ

The rapid advancements in the AI domain have given rise to the need for powerful computational resources. If you're involved in data science or AI research, you're already aware of the immense processing capabilities required to train complex models like the 70B LLama2 GPTQ. Fortunately, with cloud GPU rental services like [Vast.AI](https://vast.ai/), accessing these resources has become easier and more efficient than ever.

## Quick Guide to Launch Oobabooga webUI on Vast.AI

For those interested in leveraging the groundbreaking [70B LLama2 GPTQ](https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ), TheBloke made this possible.

[We’ve created a template](https://cloud.vast.ai/?ref_id=43484&template_id=47c1a7f2cd646ecbb295e4491cd48134) to auto-launch the Oobabooga webUI. Notably, this template grants users the versatility of direct SSH and Jupyter (notebook) integration.

Users should note this particular model demands a staggering 40 GB of VRAM. To accommodate this, Vast.AI's interface has a **VRAM slider** to guide you in selecting suitable machines like the A6000, A40, and A100.

For the 70B Model, we recommend the A6000 (_currently $0.50/hr_) or A40 (_currently $0.40/hr_). To run the 13B model, we recommend either a 3090 (_currently $0.20/hr_) or 4090 (_currently $0.48/hr_). To see all current pricing, refer to our dynamic [pricing page](https://vast.ai/pricing), or head to our [search console](https://cloud.vast.ai/).

## Steps to Download and Execute the Model in text-generation-webui:

1. Navigate to the Model tab.
2. In the section labeled Download custom model or LoRA, input TheBloke/Llama-2-70B-chat-GPTQ.
3. For downloading from specific branches, for instance, TheBloke/Llama-2-70B-chat-GPTQ:gptq-4bit-32g-actorder_True, refer to the Provided Files section for a comprehensive list of branches for each choice.
4. Click Download. Wait for the model to download, and once it's completed, a "Done" message will appear.
5. Set the Loader to ExLlama if you're planning to utilize a 4-bit file. Alternatively, opt for AutoGPTQ or GPTQ-for-LLaMA.
6. Using AutoGPTQ? Ensure that the "No inject fused attention" option is selected.
7. On the top left, click the refresh icon adjacent to the Model label.
8. From the Model dropdown, select the freshly downloaded model: **TheBloke/Llama-2-70B-chat-GPTQ**.
9. Once the model automatically loads, it's set and ready for deployment!
10. Secure your settings by clicking Save settings for this model. Then, click Reload the Model in the top right to ensure your configurations are retained.
11. Now, head over to the Text Generation tab, input your desired prompt, and watch the magic unfold!

For those considering running LLama2 on GPUs like the 4090s and 3090s, **TheBloke/Llama-2-13B-GPTQ** is the model you'd want. Vast.AI's platform is diverse, offering a plethora of options tailored to meet your project's requirements.

For additional details and to delve deeper, please visit the official [github page for ooba](https://github.com/oobabooga/text-generation-webui).

In today's competitive digital environment, it's crucial to have scalable and reliable computational resources at your fingertips. Vast.AI is revolutionizing the way researchers and developers access and utilize GPU power, making AI model training seamless and efficient. Whether you're an AI novice or an established researcher, Vast.AI is your go-to solution for all your cloud GPU rental needs.


Harness the Power of Cloud GPUs with Vast.AI: Running the 70B LLama2 GPTQ.

Harness the Power of Cloud GPUs with Vast.AI: Running the 70B LLama2 GPTQ


## Introduction

Meta has recently launched Llama 2, the latest edition of the widely recognized Llama models, which has been trained with a 40% increase in data. The announcement of this new model is quite thrilling, considering the meteoric rise in popularity of open-source large language models (LLMs) like Llama. The Llama LLM bares some similarities to other large language models, but it’s commercially available for free, which levels the playing field. Llama 2 was pre-trained on an enormous dataset of publicly available online text and code. The fine-tuned model, Llama-2-chat, was trained on this dataset as well as over 1 million human annotations. These models have given rise to popular offspring such as Vicuna and Falcon.

Concurrently, the open-source community has launched an abundance of utilities designed to fine-tune and deploy these language models. Tools such as Peft, Bitsandbytes, and TRL allow for fine-tuning of LLMs on machines that don't have the capacity to hold the full precision model in their GPU RAM.

The aim of this blog post is to guide you on how to fine-tune Llama 2 models on the Vast platform. We have benchmarked this on an RTX 3090, RTX 4090, and A100 SMX4 80GB. See the [latest pricing](https://vast.ai/pricing) on Vast for up to the minute on-demand rental prices.

## Benchmarks

The RTX 4090 demonstrates an impressive 1.5 8-bit samples/sec with a batch size of 8. This is almost twice as fast as running this on an A100 when accounting for batch size! Considering that the RTX 4090 is $0.50/hr, the price for performance is about 6X when compared to an A100 for $1.50/hr.

<table className="llamaTable mx-auto table-fixed" style={{color: "white"}}>
<tr className="llamaTableRow" style={{ padding: '10px', textAlign:'left',     borderBottom: "1px solid lightGray"}}>
<th style={{  width: '4.5rem' }}>Model</th>
<th style={{  width: '10rem' }}>GPU</th>
<th style={{  width: '7rem' }}>Batch Size</th>
<th style={{ width: '12rem'}}>4bit samples/sec[1]</th>
<th style={{ width: '11rem'}}>8bit samples/sec[1]</th></tr>

<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>RTX 4090</th>
  <th>8</th>
  <th>1</th>
  <th>1.5</th>
</tr>
<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>A100 SMX4 80GB</th>
  <th>32</th>
  <th>0.5</th>
  <th>0.8</th>
</tr>
<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>A100 RTX 3090</th>
  <th>8</th>
  <th>0.5</th>
  <th>0.9</th>
</tr>

</table>

[1] Here samples/sec is calculated by multiplying batch size by the inverse of s/iter that the sft_trainer script reports. All training runs had gradient accumulation steps equal to 1.

## How to run this yourself on Vast

Follow the rest of this post for a guide on how to fine-tune LLaMA 2 on Vast using TRL.

## Sign up with Meta and Hugging Face for access:

1- Request access from Meta here: [https://ai.meta.com/resources/models-and-libraries/llama-downloads/](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)

2- Request access from Hugging Face on any of the model pages: [https://huggingface.co/meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b)

3 - Set up an auth token with Hugging Face here: [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)

You will need to do both 1 and 2 in order to get access to LLaMA 2.

## Rent a powerful GPU on Vast.ai

Vast has RTX 3090s, RTX 4090s and A100s for on-demand rentals. Our pricing is typically the best you can find online.

To run LLaMA 2 fine tuning, you will want to use a Pytorch image on the machine you pick. To do that, click on [this Vast console link](https://cloud.vast.ai/?ref_id=43484&template_id=2d307d4c1ba166b8e4bafcd40d3033e2), which will select our recommended Pytorch template with SSH enabled and other settings enabled. If you don’t have an account set up, no problem! You will need to first quickly set up a Vast account by registering your email, verifying your email, and then purchasing credits.

To rent a machine, pick an RTX 3090 or RTX 4090 by selecting that filter in the upper right. Move the storage slider over on the left of the interface to ~30GB so your instance will have enough storage to download the model weights.

Hit the rent button to start the instance. Once it is done loading, hit the blue <\_ button to get the SSH details and SSH into the instance by copying and pasting that SSH command to your command line. You will now be logged into your Vast instance via SSH. If you have trouble setting up SSH, [read our SSH docs](https://vast.ai/docs/instance-setup/ssh).

## Set up environment

Setup the packages you will need:

`pip install transformers peft trl bitsandbytes scipy`

Clone the TRL repo for the training script

`git clone https://github.com/lvwerra/trl`

Log into HuggingFace on CLI

`huggingface-cli login`

Copy the auth token you created earlier (from https://huggingface.co/settings/tokens) and paste it into the prompt when asked. You can decline adding the token to your git credentials.

Fine tune!

`python trl/examples/scripts/sft_trainer.py --model_name meta-llama/Llama-2-7b-hf --dataset_name timdettmers/openassistant-guanaco --load_in_8bit --use_peft --batch_size 8 --gradient_accumulation_steps 1`

This will download the model weights automatically, so the first time you run, it will take a bit to actually start training. The dataset is specified there and it is using the openassistant-guanaco set. If you want to try loading in the 4bit version or change the batch size, modify that command.

You should end up seeing output like this:

```
{'loss': 1.6493, 'learning_rate': 1.4096181965881397e-05, 'epoch': 0.0}
{'loss': 1.3571, 'learning_rate': 1.4092363931762796e-05, 'epoch': 0.0}
{'loss': 1.5853, 'learning_rate': 1.4088545897644193e-05, 'epoch': 0.0}
{'loss': 1.4237, 'learning_rate': 1.408472786352559e-05, 'epoch': 0.0}
{'loss': 1.7098, 'learning_rate': 1.4080909829406987e-05, 'epoch': 0.0}
{'loss': 1.4348, 'learning_rate': 1.4077091795288384e-05, 'epoch': 0.0}
{'loss': 1.6022, 'learning_rate': 1.407327376116978e-05, 'epoch': 0.01}
{'loss': 1.3352, 'learning_rate': 1.4069455727051177e-05, 'epoch': 0.01}
```

## Summary

The price for performance on Vast when using an RTX 4090 is quite impressive when compared to an A100. In this post we have shown to easy it is to spin up a very low cost GPU ($0.20 per hour) and fine-tune the LLaMA 2 models.


LLaMA 2 fine-tuning on a Vast RTX 4090 for $0.50/hr outperforms an A100.

Fine-tuning the LLaMA 2 model on RTX 4090


In the rapidly evolving world of cloud computing, Vast.ai remains steadfast in our commitment to being the leader in cost-effective cloud GPU rental. Today, we are thrilled to announce our latest feature – Reserved Instance Discounts – aimed at helping our users get even better pricing for longer term reservations.

## On-Demand vs Reserved: Striking the Right Balance

In the on-demand world of cloud computing, users only pay for what they use and have the flexibility to end a contract at any time. While this offers unrivaled flexibility, the occupancy rate – or the fraction of time GPUs are rented – often clocks in at less than 100%, sometimes even as low as 50%. The resulting impact on provider revenue is noticeable as their hardware sits idle.

Clients willing to commit upfront to a longer rental term can now leverage our Reserved Instance Discounts feature which solves this idle problem. With this, clients can commit to an extended rental contract by pre-paying in advance. While they give up the on-demand liquidity, they receive a hefty discount in return. The provider controls the discount percentage, so each offer will vary. 

## How Reserved Instance Discounts Work

Here's the breakdown:

1. Users can deposit credits into a specific instance using either the [CLI](https://cloud.vast.ai/cli/) or website GUI (look for the **green tag**). The CLI gives users a detailed control over deposit amounts, while the GUI provides a user-friendly interface. The clickable green tag in the GUI is shown for participating machines in the bottom right of the instance card.

![green tag](/uploads/prepaidicon.png)

2. The discount schedule is a function of the time duration of the balance. Larger deposits unlock greater discounts, albeit with diminishing returns. The default discount schedule offers a 20% discount for a 1-month commitment, 30% for a 3-month commitment, and 40% for a 6-month commitment.

![](/uploads/reservedmodal.png)

3. To give an example, if an instance usually costs $1/hr on-demand, a 1-month prepaid rental with the default discount schedule would cost $576 instead of $720. That's a savings of 20%!

4. When additional funds are deposited into an existing reserved instance, everything is recalculated according to the new total balance, leading to a new discount rate.

The specific discount schedule can vary from provider to provider, with 40% at 6 months being the default setting.

## Link to the Documentation

For the complete description of how these works, read our documentation for [Reserved Instance Discounts](https://vast.ai/docs/instance-rental/prepaid-plans).

## Seize the Savings

Our new Reserved Instance Discounts feature is here to change the way you use cloud computing, providing an avenue to commit to your long-term compute needs while reaping the benefits of considerable discounts. So whether you are a start-up scaling your operations or an established business with predictable computing demands, let Vast.ai help you make the most of your cloud computing budget. Your savings start today with Reserved Instance Discounts. Take control of your cloud computing journey and optimize your budget like never before.





Our on-demand rentals provide incredible value. For hosts that enable this feature, you can get further discounts by reserving an instance for months.

Stretching Your Cloud Computing Budget: Introducing Reserved Instance Discounts


The Nvidia RTX 4090 is a highly reliable and powerful GPU released to the PC gaming market. However it is also suitable for machine learning and deep learning jobs. Whether you're a data scientist, AI researcher, or developer looking for a GPU with high deep learning performance to help take your projects to the next level, the RTX 4090 is an excellent choice.

The RTX 4090 is a high-end GPU powered by the Ada Lovelace architecture. Equipped with 16,384 CUDA cores and 512 Turing Tensor Cores, this GPU is a monster of computing power and can easily tackle the most complex deep learning tasks. This makes it the ideal choice for tackling difficult projects such as facial recognition, natural language processing, and computer vision.

Vast.ai has on-demand [cloud RTX 4090](https://cloud.vast.ai/?gpu_option=RTX%204090) rentals available now.

## NVLink Phased Out

Nvidia is opting for PCIe 4.0 instead of NVLink as a bridge for those who desire a multi-GPU setup with the latest RTX 4000 series, catering to gamers, creators, and workstation users. It will use PCIe 4.0 even when inserted in a PCIe 5.0 slot, due to backward compatibility.

Before launch, Nvidia CEO Jensen Huang announced to reporters that the company will be discontinuing its NVLink bridge and introducing PCIe Gen 5 as its replacement. Huang and Nvidia stated that this new connection will be fast enough for linking multiple GPUs and that the space previously used by the NVLink connector will now be dedicated to additional AI computing.

"The reason why we took [NVLink] out was because we needed the we needed the I/Os for something else, and so, so we use the I/O area to cram in as much as much AI processing as we could," Huang confirmed and explained of NVLink's absence.

## CUDA Library Support on 4090

The RTX 4090 also supports the latest version of Nvidia's CUDA-X AI Library. This library provides data scientists and developers with a rich set of optimized algorithms for deep learning applications. With its help, they can quickly and easily build complex projects which require high-performance computing, and add GPU acceleration to their projects build complex deep-learning models in a shorter time.

To put it simply, the GeForce RTX 4090 is an excellent choice for those interested in affordable deep learning. This is especially true for students, researchers, and creators on a budget. While it's faster the previous flagship GPU, the GeForce RTX 3090, it's also more cost-effective in terms of training throughput.

## Additional Benefits for Deep Learning

The 4090 is also equipped with 24 GB of GDDR6 VRAM, allowing it to store and retrieve large amounts of data quickly and effectively. It's well suited for developers and data scientists who need to quickly perform significantly higher computation tasks in the world of machine and deep learning.

The GPU also contains DLSS AI upscaling, which can improve the performance of your deep learning models by 200%. This makes the 4090 a great choice for both training and serving models.

## Final Thoughts on the RTX 4090 for Deep Learning

Overall, the RTX 4090 is an impressive piece of technology for deep learning. The 4090 is an ideal choice for those looking to take their deep learning and machine learning projects to the next level. With its powerful computing capabilities and easy integration with Nvidia's CUDA libraries, it's designed to help you tackle any task quickly and effectively.

With its powerful architecture and generous CUDA core count, it has the capability to tackle difficult projects with ease. If you're looking for a reliable GPU that can power your deep learning projects, the RTX 4090 is a great choice.


We dig deeper into the Nvidia RTX 4090 and its uses in deep learning.

Nvidia RTX 4090 as a Deep Learning GPU


The rapid rise in the demand for high-performance computing has led to an increasing need for efficient, cost-effective solutions. One such solution is GPU rental. These services offer organizations the flexibility to access powerful processing units without the hefty upfront investment. Let's look at what exactly renting a GPU entails, their application in Deep Learning (and cryptocurrency), and why Vast.AI stands out as the best rental GPU source.

## Understanding GPU Rentals

A Graphics Processing Unit (GPU) is a powerful electronic circuit designed to quickly manipulate and render images, 3D graphics, and other complex calculations. The parallel processing capabilities of GPUs make them an ideal choice for tasks that demand high computational power.

In recent years, GPU rentals have been on the rise. This service enables users to rent powerful GPUs on a pay-as-you-go basis, significantly reducing the cost associated with purchasing and maintaining high-end GPUs. One such service provider that has gained prominence in the GPU rental market is Vast.AI.

## How Does Vast.AI Work?

Vast.AI is a global GPU marketplace that aims to reduce the cost of cloud GPU rentals for AI/ML and GPU-related compute needs. By leveraging a sharing economy approach, Vast.AI offers GPU rentals at prices significantly lower than its competitors.

Hosts with professional-grade machines can list their devices on Vast.AI and set pricing for various rental components, such as storage and bandwidth. Users looking to rent GPUs can search through the available offers, filtering by GPU type, GPU RAM, CPU, and system RAM. Pricing is transparently displayed for the GPU rental, storage, and any bandwidth charges.

Only at Vast.AI, payments can be made either by credit card or with crypto through either Coinbase or Crypto.com.

Vast.AI uses Linux docker images, and users can select any private or public repo with a Linux docker image to load onto a rented machine. A powerful CLI tool allows for a programmatic interface to automate GPU server rentals.

## Unleashing the Power of Cloud GPUs in Deep Learning

Deep learning, a subset of machine learning, relies on artificial neural networks to mimic the human brain's functioning. It enables the system to learn from vast amounts of data, making it a critical component of artificial intelligence.

GPUs are designed to perform parallel computations, making them an excellent choice for deep learning models. They can expedite large computational tasks due to their many cores and high memory bandwidth. Whether it's training AI models, rendering CAD models, or playing graphics-rich video games, GPUs have become an indispensable tool in the world of technology.

When it comes to deep learning, GPUs can offer performance speeds up to 250 times faster than traditional CPUs. This incredible speed makes them an ideal choice for training neural networks associated with deep learning.

## The Benefits of Cloud GPUs

The advent of cloud computing has taken the capabilities of GPUs to new heights. Cloud GPUs offer several advantages over traditional physical GPUs.

### Scalability

One of the significant benefits of cloud GPUs is their scalability. As an organization grows, its computational needs also increase. Cloud GPUs can easily scale up or down to meet these changing needs, offering flexibility that physical GPUs can't match.

### Cost-Effective

Cloud GPUs also offer a more cost-effective solution. Instead of making a substantial upfront investment in high-powered GPUs, organizations can rent cloud GPUs on an hourly basis. This pay-as-you-go model can significantly reduce costs, especially for organizations that don't need constant access to high-powered GPUs.

### Space and Resource Efficiency

Cloud GPUs don't consume local resources, unlike physical GPUs that take up significant space on your computer. This means your computer can be used for other tasks while the heavy computational work is offloaded to the cloud.

### Time-Saving

Cloud GPUs can save a significant amount of time. Tasks that once took hours or even days to complete can now be finished in minutes, thanks to the power of cloud GPUs. This increased efficiency can lead to higher productivity and more time for innovation.

## Why Vast.AI is the Best Rental GPU Source

Vast.AI stands out in the GPU rental market due to its unique approach. Transparently, Vast.AI offers GPU rentals at significantly lower prices than its competitors.

But cost isn't the only factor that sets Vast.AI apart. The platform also provides a user-friendly interface with detailed information about each available machine, making it easy for users to find the perfect GPU for their needs.

Furthermore, Vast.AI's flexibility of payment options makes it easier for users to pay for their rentals in the way that works best for them.

## Conclusion

In conclusion, GPU rental has emerged as a game-changing solution in the world of high-performance computing. Whether it's deep learning, crypto mining, or any other task that requires significant computational power, GPU rental can provide a cost-effective, flexible solution.

Vast.AI, with its unique approach and user-friendly platform, stands out as the best rental GPU source. Whether you're an AI researcher looking for a cost-effective way to train your models or a crypto miner looking to maximize your efficiency, Vast.AI can provide the powerful, affordable GPU rental solution you need.


We take a look at what renting a GPU entails, their application in Deep Learning, and why Vast.AI stands out as the best rental GPU source.

Gpu

Harness the Power of Cloud GPUs with Vast.AI: Running the 70B LLama2 GPTQ

Fine-tuning the LLaMA 2 model on RTX 4090

Stretching Your Cloud Computing Budget: Introducing Reserved Instance Discounts

Nvidia RTX 4090 as a Deep Learning GPU

Revolutionizing Cloud GPU Rentals for Deep Learning with Vast.ai