
## Introduction

Meta has recently launched Llama 2, the latest edition of the widely recognized Llama models, which has been trained with a 40% increase in data. The announcement of this new model is quite thrilling, considering the meteoric rise in popularity of open-source large language models (LLMs) like Llama. The Llama LLM bares some similarities to other large language models, but it’s commercially available for free, which levels the playing field. Llama 2 was pre-trained on an enormous dataset of publicly available online text and code. The fine-tuned model, Llama-2-chat, was trained on this dataset as well as over 1 million human annotations. These models have given rise to popular offspring such as Vicuna and Falcon.

Concurrently, the open-source community has launched an abundance of utilities designed to fine-tune and deploy these language models. Tools such as Peft, Bitsandbytes, and TRL allow for fine-tuning of LLMs on machines that don't have the capacity to hold the full precision model in their GPU RAM.

The aim of this blog post is to guide you on how to fine-tune Llama 2 models on the Vast platform. We have benchmarked this on an RTX 3090, RTX 4090, and A100 SMX4 80GB. See the [latest pricing](https://vast.ai/pricing) on Vast for up to the minute on-demand rental prices.

## Benchmarks

The RTX 4090 demonstrates an impressive 1.5 8-bit samples/sec with a batch size of 8. This is almost twice as fast as running this on an A100 when accounting for batch size! Considering that the RTX 4090 is $0.50/hr, the price for performance is about 6X when compared to an A100 for $1.50/hr.

<table className="llamaTable mx-auto table-fixed" style={{color: "white"}}>
<tr className="llamaTableRow" style={{ padding: '10px', textAlign:'left',     borderBottom: "1px solid lightGray"}}>
<th style={{  width: '4.5rem' }}>Model</th>
<th style={{  width: '10rem' }}>GPU</th>
<th style={{  width: '7rem' }}>Batch Size</th>
<th style={{ width: '12rem'}}>4bit samples/sec[1]</th>
<th style={{ width: '11rem'}}>8bit samples/sec[1]</th></tr>

<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>RTX 4090</th>
  <th>8</th>
  <th>1</th>
  <th>1.5</th>
</tr>
<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>A100 SMX4 80GB</th>
  <th>32</th>
  <th>0.5</th>
  <th>0.8</th>
</tr>
<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>A100 RTX 3090</th>
  <th>8</th>
  <th>0.5</th>
  <th>0.9</th>
</tr>

</table>

[1] Here samples/sec is calculated by multiplying batch size by the inverse of s/iter that the sft_trainer script reports. All training runs had gradient accumulation steps equal to 1.

## How to run this yourself on Vast

Follow the rest of this post for a guide on how to fine-tune LLaMA 2 on Vast using TRL.

## Sign up with Meta and Hugging Face for access:

1- Request access from Meta here: [https://ai.meta.com/resources/models-and-libraries/llama-downloads/](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)

2- Request access from Hugging Face on any of the model pages: [https://huggingface.co/meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b)

3 - Set up an auth token with Hugging Face here: [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)

You will need to do both 1 and 2 in order to get access to LLaMA 2.

## Rent a powerful GPU on Vast.ai

Vast has RTX 3090s, RTX 4090s and A100s for on-demand rentals. Our pricing is typically the best you can find online.

To run LLaMA 2 fine tuning, you will want to use a Pytorch image on the machine you pick. To do that, click on [this Vast console link](https://cloud.vast.ai/?ref_id=43484&template_id=2d307d4c1ba166b8e4bafcd40d3033e2), which will select our recommended Pytorch template with SSH enabled and other settings enabled. If you don’t have an account set up, no problem! You will need to first quickly set up a Vast account by registering your email, verifying your email, and then purchasing credits.

To rent a machine, pick an RTX 3090 or RTX 4090 by selecting that filter in the upper right. Move the storage slider over on the left of the interface to ~30GB so your instance will have enough storage to download the model weights.

Hit the rent button to start the instance. Once it is done loading, hit the blue <\_ button to get the SSH details and SSH into the instance by copying and pasting that SSH command to your command line. You will now be logged into your Vast instance via SSH. If you have trouble setting up SSH, [read our SSH docs](https://vast.ai/docs/gpu-instances/ssh).

## Set up environment

Setup the packages you will need:

`pip install transformers peft trl bitsandbytes scipy`

Clone the TRL repo for the training script

`git clone https://github.com/lvwerra/trl`

Log into HuggingFace on CLI

`huggingface-cli login`

Copy the auth token you created earlier (from https://huggingface.co/settings/tokens) and paste it into the prompt when asked. You can decline adding the token to your git credentials.

Fine tune!

`python trl/examples/scripts/sft_trainer.py --model_name meta-llama/Llama-2-7b-hf --dataset_name timdettmers/openassistant-guanaco --load_in_8bit --use_peft --batch_size 8 --gradient_accumulation_steps 1`

This will download the model weights automatically, so the first time you run, it will take a bit to actually start training. The dataset is specified there and it is using the openassistant-guanaco set. If you want to try loading in the 4bit version or change the batch size, modify that command.

You should end up seeing output like this:

```
{'loss': 1.6493, 'learning_rate': 1.4096181965881397e-05, 'epoch': 0.0}
{'loss': 1.3571, 'learning_rate': 1.4092363931762796e-05, 'epoch': 0.0}
{'loss': 1.5853, 'learning_rate': 1.4088545897644193e-05, 'epoch': 0.0}
{'loss': 1.4237, 'learning_rate': 1.408472786352559e-05, 'epoch': 0.0}
{'loss': 1.7098, 'learning_rate': 1.4080909829406987e-05, 'epoch': 0.0}
{'loss': 1.4348, 'learning_rate': 1.4077091795288384e-05, 'epoch': 0.0}
{'loss': 1.6022, 'learning_rate': 1.407327376116978e-05, 'epoch': 0.01}
{'loss': 1.3352, 'learning_rate': 1.4069455727051177e-05, 'epoch': 0.01}
```

## Summary

The price for performance on Vast when using an RTX 4090 is quite impressive when compared to an A100. In this post we have shown to easy it is to spin up a very low cost GPU ($0.20 per hour) and fine-tune the LLaMA 2 models.


LLaMA 2 fine-tuning on a Vast RTX 4090 for $0.50/hr outperforms an A100.

Fine-tuning the LLaMA 2 model on RTX 4090


In the rapidly evolving world of cloud computing, Vast.ai remains steadfast in our commitment to being the leader in cost-effective cloud GPU rental. Today, we are thrilled to announce our latest feature – Reserved Instance Discounts – aimed at helping our users get even better pricing for longer term reservations.

## On-Demand vs Reserved: Striking the Right Balance

In the on-demand world of cloud computing, users only pay for what they use and have the flexibility to end a contract at any time. While this offers unrivaled flexibility, the occupancy rate – or the fraction of time GPUs are rented – often clocks in at less than 100%, sometimes even as low as 50%. The resulting impact on provider revenue is noticeable as their hardware sits idle.

Clients willing to commit upfront to a longer rental term can now leverage our Reserved Instance Discounts feature which solves this idle problem. With this, clients can commit to an extended rental contract by pre-paying in advance. While they give up the on-demand liquidity, they receive a hefty discount in return. The provider controls the discount percentage, so each offer will vary. 

## How Reserved Instance Discounts Work

Here's the breakdown:

1. Users can deposit credits into a specific instance using either the [CLI](https://cloud.vast.ai/cli/) or website GUI (look for the **green tag**). The CLI gives users a detailed control over deposit amounts, while the GUI provides a user-friendly interface. The clickable green tag in the GUI is shown for participating machines in the bottom right of the instance card.

![green tag](/uploads/prepaidicon.png)

2. The discount schedule is a function of the time duration of the balance. Larger deposits unlock greater discounts, albeit with diminishing returns. The default discount schedule offers a 20% discount for a 1-month commitment, 30% for a 3-month commitment, and 40% for a 6-month commitment.

![](/uploads/reservedmodal.png)

3. To give an example, if an instance usually costs $1/hr on-demand, a 1-month prepaid rental with the default discount schedule would cost $576 instead of $720. That's a savings of 20%!

4. When additional funds are deposited into an existing reserved instance, everything is recalculated according to the new total balance, leading to a new discount rate.

The specific discount schedule can vary from provider to provider, with 40% at 6 months being the default setting.

## Link to the Documentation

For the complete description of how these works, read our documentation for [Reserved Instance Discounts](https://vast.ai/docs/gpu-instances/21-prepaid-plans).

## Seize the Savings

Our new Reserved Instance Discounts feature is here to change the way you use cloud computing, providing an avenue to commit to your long-term compute needs while reaping the benefits of considerable discounts. So whether you are a start-up scaling your operations or an established business with predictable computing demands, let Vast.ai help you make the most of your cloud computing budget. Your savings start today with Reserved Instance Discounts. Take control of your cloud computing journey and optimize your budget like never before.





Our on-demand rentals provide incredible value. For hosts that enable this feature, you can get further discounts by reserving an instance for months.

Stretching Your Cloud Computing Budget: Introducing Reserved Instance Discounts


As we delve deeper into the realm of artificial intelligence, one of the most impactful developments is the rise of large language models. These sophisticated tools, designed to understand and generate human-like text, have a myriad of applications spanning diverse fields. This article provides a comprehensive analysis of the key factors impacting their major use cases, including customer service, education, healthcare, legal, and marketing. We will also discuss the trade-offs and challenges associated with these different approaches, highlighting the importance of informed decision-making when choosing a language model.

## An Overview of Large Language Models

Large language models, such as OpenAI's GPT-3 and Google's BERT, are AI models trained on vast amounts of text data. Their capabilities range from understanding context in a conversation, translating languages, to even generating human-like text. The implications of these capabilities are far-reaching, with applications in various sectors.

## Major Use Cases: From Customer Service to Marketing

One of the significant applications of large language models is in customer service. These models can power chatbots and virtual assistants, providing fast, accurate, and personalized responses to customer inquiries. However, the trade-off here is the lack of human touch and potential misunderstandings due to the limitations in understanding complex human emotions and intentions.

In education, large language models can help develop intelligent tutoring systems, providing personalized learning experiences for students. While this promises a more scalable education solution, the risk lies in ensuring the accuracy and appropriateness of the content generated.

In the healthcare sector, large language models can assist in analyzing patient records and medical literature, providing recommendations or aiding in diagnoses. However, the stakes are high, as inaccurate or misinterpreted information can have serious consequences.

Legal professionals can use large language models to analyze legal documents, contracts, and case laws, which can automate some aspects of legal research. But the nuanced and critical nature of legal work means that outputs must be thoroughly reviewed for accuracy.

Lastly, in marketing, language models can generate creative marketing content and analyze consumer sentiment. The challenge here lies in ensuring that the content remains authentic and resonates with the target audience.

## Challenges and Considerations

Despite the impressive capabilities of large language models, their use is accompanied by several challenges. One notable issue is the risk of bias, as these models can inadvertently reproduce biases present in their training data. This risk underscores the importance of carefully curating training data and monitoring model outputs.

The selection of an appropriate language model often depends on the user's specific needs and constraints. While more powerful models can generate higher-quality outputs, they also require more computational resources, thus increasing costs.

## Conclusion

Large language models offer transformative potential across various sectors, from customer service to marketing. However, their implementation is not without challenges, and users must carefully weigh the trade-offs. By considering factors such as performance, cost, and the potential for bias, users can make informed decisions about the best model to use. Despite the complexities, the world of large language models holds immense promise and potential for advancement.

We take a deep dive into the pros and cons of large language models like OpenAI's GPT-4 and Google's BERT systems.

The Versatile Power of Large Language Models: Exploring Major Use Cases and Their Trade-offs


The Nvidia RTX 4090 is a highly reliable and powerful GPU released to the PC gaming market. However it is also suitable for machine learning and deep learning jobs. Whether you're a data scientist, AI researcher, or developer looking for a GPU with high deep learning performance to help take your projects to the next level, the RTX 4090 is an excellent choice.

The RTX 4090 is a high-end GPU powered by the Ada Lovelace architecture. Equipped with 16,384 CUDA cores and 512 Turing Tensor Cores, this GPU is a monster of computing power and can easily tackle the most complex deep learning tasks. This makes it the ideal choice for tackling difficult projects such as facial recognition, natural language processing, and computer vision.

Vast.ai has on-demand [cloud RTX 4090](https://cloud.vast.ai/?gpu_option=RTX%204090) rentals available now.

## NVLink Phased Out

Nvidia is opting for PCIe 4.0 instead of NVLink as a bridge for those who desire a multi-GPU setup with the latest RTX 4000 series, catering to gamers, creators, and workstation users. It will use PCIe 4.0 even when inserted in a PCIe 5.0 slot, due to backward compatibility.

Before launch, Nvidia CEO Jensen Huang announced to reporters that the company will be discontinuing its NVLink bridge and introducing PCIe Gen 5 as its replacement. Huang and Nvidia stated that this new connection will be fast enough for linking multiple GPUs and that the space previously used by the NVLink connector will now be dedicated to additional AI computing.

"The reason why we took [NVLink] out was because we needed the we needed the I/Os for something else, and so, so we use the I/O area to cram in as much as much AI processing as we could," Huang confirmed and explained of NVLink's absence.

## CUDA Library Support on 4090

The RTX 4090 also supports the latest version of Nvidia's CUDA-X AI Library. This library provides data scientists and developers with a rich set of optimized algorithms for deep learning applications. With its help, they can quickly and easily build complex projects which require high-performance computing, and add GPU acceleration to their projects build complex deep-learning models in a shorter time.

To put it simply, the GeForce RTX 4090 is an excellent choice for those interested in affordable deep learning. This is especially true for students, researchers, and creators on a budget. While it's faster the previous flagship GPU, the GeForce RTX 3090, it's also more cost-effective in terms of training throughput.

## Additional Benefits for Deep Learning

The 4090 is also equipped with 24 GB of GDDR6 VRAM, allowing it to store and retrieve large amounts of data quickly and effectively. It's well suited for developers and data scientists who need to quickly perform significantly higher computation tasks in the world of machine and deep learning.

The GPU also contains DLSS AI upscaling, which can improve the performance of your deep learning models by 200%. This makes the 4090 a great choice for both training and serving models.

## Final Thoughts on the RTX 4090 for Deep Learning

Overall, the RTX 4090 is an impressive piece of technology for deep learning. The 4090 is an ideal choice for those looking to take their deep learning and machine learning projects to the next level. With its powerful computing capabilities and easy integration with Nvidia's CUDA libraries, it's designed to help you tackle any task quickly and effectively.

With its powerful architecture and generous CUDA core count, it has the capability to tackle difficult projects with ease. If you're looking for a reliable GPU that can power your deep learning projects, the RTX 4090 is a great choice.


We dig deeper into the Nvidia RTX 4090 and its uses in deep learning.

Nvidia RTX 4090 as a Deep Learning GPU


The rapid rise in the demand for high-performance computing has led to an increasing need for efficient, cost-effective solutions. One such solution is GPU rental. These services offer organizations the flexibility to access powerful processing units without the hefty upfront investment. Let's look at what exactly renting a GPU entails, their application in Deep Learning (and cryptocurrency), and why Vast.AI stands out as the best rental GPU source.

## Understanding GPU Rentals

A Graphics Processing Unit (GPU) is a powerful electronic circuit designed to quickly manipulate and render images, 3D graphics, and other complex calculations. The parallel processing capabilities of GPUs make them an ideal choice for tasks that demand high computational power.

In recent years, GPU rentals have been on the rise. This service enables users to rent powerful GPUs on a pay-as-you-go basis, significantly reducing the cost associated with purchasing and maintaining high-end GPUs. One such service provider that has gained prominence in the GPU rental market is Vast.AI.

## How Does Vast.AI Work?

Vast.AI is a global GPU marketplace that aims to reduce the cost of cloud GPU rentals for AI/ML and GPU-related compute needs. By leveraging a sharing economy approach, Vast.AI offers GPU rentals at prices significantly lower than its competitors.

Hosts with professional-grade machines can list their devices on Vast.AI and set pricing for various rental components, such as storage and bandwidth. Users looking to rent GPUs can search through the available offers, filtering by GPU type, GPU RAM, CPU, and system RAM. Pricing is transparently displayed for the GPU rental, storage, and any bandwidth charges.

Only at Vast.AI, payments can be made either by credit card or with crypto through either Coinbase or Crypto.com.

Vast.AI uses Linux docker images, and users can select any private or public repo with a Linux docker image to load onto a rented machine. A powerful CLI tool allows for a programmatic interface to automate GPU server rentals.

## Unleashing the Power of Cloud GPUs in Deep Learning

Deep learning, a subset of machine learning, relies on artificial neural networks to mimic the human brain's functioning. It enables the system to learn from vast amounts of data, making it a critical component of artificial intelligence.

GPUs are designed to perform parallel computations, making them an excellent choice for deep learning models. They can expedite large computational tasks due to their many cores and high memory bandwidth. Whether it's training AI models, rendering CAD models, or playing graphics-rich video games, GPUs have become an indispensable tool in the world of technology.

When it comes to deep learning, GPUs can offer performance speeds up to 250 times faster than traditional CPUs. This incredible speed makes them an ideal choice for training neural networks associated with deep learning.

## The Benefits of Cloud GPUs

The advent of cloud computing has taken the capabilities of GPUs to new heights. Cloud GPUs offer several advantages over traditional physical GPUs.

### Scalability

One of the significant benefits of cloud GPUs is their scalability. As an organization grows, its computational needs also increase. Cloud GPUs can easily scale up or down to meet these changing needs, offering flexibility that physical GPUs can't match.

### Cost-Effective

Cloud GPUs also offer a more cost-effective solution. Instead of making a substantial upfront investment in high-powered GPUs, organizations can rent cloud GPUs on an hourly basis. This pay-as-you-go model can significantly reduce costs, especially for organizations that don't need constant access to high-powered GPUs.

### Space and Resource Efficiency

Cloud GPUs don't consume local resources, unlike physical GPUs that take up significant space on your computer. This means your computer can be used for other tasks while the heavy computational work is offloaded to the cloud.

### Time-Saving

Cloud GPUs can save a significant amount of time. Tasks that once took hours or even days to complete can now be finished in minutes, thanks to the power of cloud GPUs. This increased efficiency can lead to higher productivity and more time for innovation.

## Why Vast.AI is the Best Rental GPU Source

Vast.AI stands out in the GPU rental market due to its unique approach. Transparently, Vast.AI offers GPU rentals at significantly lower prices than its competitors.

But cost isn't the only factor that sets Vast.AI apart. The platform also provides a user-friendly interface with detailed information about each available machine, making it easy for users to find the perfect GPU for their needs.

Furthermore, Vast.AI's flexibility of payment options makes it easier for users to pay for their rentals in the way that works best for them.

## Conclusion

In conclusion, GPU rental has emerged as a game-changing solution in the world of high-performance computing. Whether it's deep learning, crypto mining, or any other task that requires significant computational power, GPU rental can provide a cost-effective, flexible solution.

Vast.AI, with its unique approach and user-friendly platform, stands out as the best rental GPU source. Whether you're an AI researcher looking for a cost-effective way to train your models or a crypto miner looking to maximize your efficiency, Vast.AI can provide the powerful, affordable GPU rental solution you need.


We take a look at what renting a GPU entails, their application in Deep Learning, and why Vast.AI stands out as the best rental GPU source.

Revolutionizing Cloud GPU Rentals for Deep Learning with Vast.ai


In the rapidly evolving world of machine learning, the ability to leverage powerful computing resources on-demand has become crucial. The need to run large language models such as LLaMA, GPT-J, Pythia, and Falcon has grown more pronounced, as these models are increasingly important in various AI applications. However, these models require significant computational power, typically in the form of high-performance graphics processing units (GPUs). Many of the models require 40+ GB of GPU RAM to load the models.

This is where Vast and Oobabooga come in.

Vast offers on-demand GPU rentals, providing users with the raw power needed to run these models. It's known for being the most cost-effective offering on the market, making it an ideal choice for those who are budget-conscious or just starting out with AI and machine learning.

[Oobabooga](https://github.com/oobabooga/text-generation-webui) is a web user interface (UI) designed specifically for running large language models. It's simple and intuitive to use, requiring no advanced technical skills or extensive prior knowledge of AI and machine learning.

In this post, we will guide you through the process of deploying Oobabooga on Vast, showing you just how easy it is to run large language models on a budget.

## Getting Started with Vast

The first step is to sign up for an account on Vast. You will need to add credits before you can rent. Go to the billing page and choose a payment method. Larger language models will require more expensive GPUs, so be sure to add enough credits.

## Selecting the Oobabooga template

Vast rents out Linux Docker instances on-demand. We have a pre-built Oobabooga template that you will need to load. Load the template by clicking on [this link](https://cloud.vast.ai/?template_id=d8339b706edae2f312c29fb3940a4b17). 

## Add enough storage

Underneath the Edit image and config options there is a slider which will allocate storage for your instance. Large language models are typically 70+ GB, so make sure to move the slider to 80 GB so your instance will have enough storage to download the model you want to run.

## Pick a GPU

Now comes the fun part! Vast is a marketplace of on-demand cloud GPUs. You can easily select a powerful GPU that supports your selected configuration. Many large language models require the absolute best GPU right now. Read about how much GPU RAM your model needs to run. The top of the line GPU is the A100 SMX4 80GB or A100 PCIE 80GB. To ensure your instance will have enough GPU RAM, use the GPU RAM slider in the interface.

Once you find a suitable GPU, click RENT. 

## Running Your Models

The instance will need to load the Docker image and complete setup. Once this is complete, a blue OPEN button will appear. Click on that to open the Oobabooga web UI interface. 

Read the Oobabooga documentation on their project website [here](https://github.com/oobabooga/text-generation-webui). You will need to download an LLM to the instance which can take some time depending on the internet connection speed of the machine you picked and the place you are downloading the model from. HuggingFace is typically pretty fast. 

Once you have the model loaded, you can start interacting with it. You'll be able to monitor the progress in real-time through the Oobabooga interface, and once the computations are complete, the results will be displayed directly on the screen. The whole process is seamless, and thanks to Vast and Oobabooga, you can run large language models without breaking the bank or needing advanced technical knowledge.

## Conclusion
With the power of Vast's affordable GPU rentals and the simplicity of Oobabooga's user interface, running large language models has never been easier or more accessible. Whether you're a student, a researcher, or an AI enthusiast, this combination of tools provides an excellent way to delve into the world of AI and machine learning without the usual barriers of cost and complexity.

An easy, cost-effective method for deploying LLaMa, GPT-J, Pythia and Falcon

Deploying Oobabooga for Running Large Language Models


Ongoing product updates are an important part of what we offer at Vast.ai. Our customers get excellent performance and reliability in large part because of the consistency with which we work to improve our service. As a Vast.ai customer, you can not only expect to get outstanding NVIDIA GPU cloud performance on day one, but you can expect that experience to continue to improve over time.

The starting point for every product update is to address any bugs that have popped up along the way. Our latest update is no different, including the fixes listed below --

- Ensuring all instances show on instance page
- Correct port allocation that caused instances to fail to start
- Improve disk speed test rates
- Solve error on 4090s causing NVLINK to show erroneously
- Fix billing page error that showed ratelimit error
- Disable extra filters with the show incompatible filter

If you are an existing user, you will notice that these fixes streamline your experience and make the product more enjoyable to use day after day. For potential new customers, you can rest assured that these fixes were made promptly and the save level of service will be delivered on future corrections that need to be made.

## NVIDIA Cloud GPU Fixes

Fixes are one thing, but we also utilize updates to deliver added capabilities and flexibility to the platform. Here are some of the highlights --

- The Juypter open button will run a test to confirm it is running before displaying
- Updated the mobile UX to improve info bar issues
- GUI filter settings now persist
- Hosts receive email notifications of expired machines
- Add ability to delete templates from recent tab
- Added 4080 and 4070 search filter
- Added function for hosts to disconnect Stripe connect for payouts

To confirm the many features and capabilities offered by Vast.ai, you are welcome to view our full documentation, or you can simply reach out directly to our team for more information. You can chat with us directly here on our site, where we will respond quickly and clear up any confusion you may have about what we offer. We also offer a Discord server that you can join to learn more, or you can drop us a message at support@vast.ai. Either way, we would be happy to assist you with your GPU rental needs.

## Change Log

-bugfix with the instance page to ensure all instances show

-bugfix with port allocation causing instances failing to start

-bugfix on the slow rate of disk speed tests

-bugfix on 4090s causing NVLINK to erroneously show

-bugfix on the billing page which sometimes showed a ratelimit error

-bugfix with the show incompatible filter to disable extra filters

-total instances cost per hour/day meter added in the billing page

-Juypter open button now tests to ensure Juypter is running before displaying

-Filters can be passed into the create page via URL

-Updated the mobile UX to fix login/user infobar issues

-Added the ability for hosts to disconnect Stripe connect for payouts

-Added a new recommended template for Bittensor 3.7.0 with readme (edited)

-Relisting offers will only extend the end date of existing contracts when the new price is the same or less

-CLI command to cancel rsync added

-Added ability to delete templates from recent tab

-Added the ability to create a readme file for templates

-Docs updated to reflect template system

-GUI filter settings now persist

-Hosts now receive email notification of expired machines

-Added 4080 and 4070 to search filter (edited)


Ongoing product updates are an important part of what we offer at Vast.ai...

Vast Product Update | April 2023


Cloud GPU rental services are becoming increasingly popular in the past few years for those who want to train deep learning models. Deep learning is a subset of machine learning that utilizes neural networks with multiple layers to analyze and process data. It has been a rapidly growing field in recent years, with many breakthroughs and advancements in the areas of image recognition, natural language processing, and self-driving cars.

Renting GPUs for training can be more cost-effective than purchasing them outright. It also gives users the ability to access the latest and greatest hardware. With cloud-based solutions, organizations can easily access these new GPUs without having to go through the process of purchasing and installing new hardware.

## GPUs for Deep Learning

Deep learning models, such as neural networks, are composed of many layers of interconnected nodes, or neurons. Each neuron performs a simple mathematical operation, such as a dot product or an activation function, on its input data. However, when these operations are applied to millions of neurons across many layers, the computational requirements become extremely large.

This is where GPUs come in. They are specifically designed to perform massive amounts of parallel computation, making them well suited for deep learning tasks. In contrast, traditional CPUs (Central Processing Units) are not optimized for parallel computation and are not able to keep up with the computational demands of deep learning models.

Another reason why GPUs are used for deep learning is that they can be used to perform both the training and the inference of deep learning models. The training process of a deep learning model requires a large amount of data and computational power. This is because the model needs to adjust the values of its parameters, or weights, to minimize the difference between the predicted outputs and the actual outputs. Inference, on the other hand, is the process of applying the trained model to new data. This also requires a significant amount of computational power.

As deep learning models continue to become more complex and data-intensive, the use of GPUs will likely become even more prevalent in the field.

## Choosing a Deep Learning GPU provider

There are a number of cloud GPU rental services available. Some of the most popular cloud GPU rental providers include Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. Each cloud provider's GPU cloud instances have different performance and cost characteristics, so it's important to do your research to find the best instance for your deep learning application.

Vast.ai is a decentralized compute marketplace that allows users to rent cloud GPUs at much lower prices when compared to major providers. The company utilizes a peer-to-peer network of individual and corporate GPU owners to provide users with access to powerful GPU resources at a fraction of the cost of traditional cloud providers. Hosts are both semi-professional and professional datacenters.

No matter which cloud provider you choose, cloud GPUs are ideal for running large-scale deep learning applications such as computer vision, natural language processing (NLP), and machine learning algorithms. You can also use cloud GPUs to build and test deep learning models in less time than with traditional CPUs.

## Choosing the Right GPUs for Deep Learning

When choosing a cloud GPU rental service, it is important to consider the needs of your deep learning applications when you pick a particular GPU, such as an A100, A6000 or RTX 4090. Factors to consider include the size of your training dataset, the number of epochs you need to train your model, the training time for each epoch, and the computational and memory resources required.

Based on these factors, you can compare the prices of different cloud GPU providers and choose the one that best suits your needs.

## Reduce Expenses for Large Datasets and Faster Training

The expense of cloud GPU rental services can differ depending on the company and the particular instance type being utilized. In general, though, renting GPUs from the cloud is much more reasonably priced than purchasing them outright - especially when you take into account all of the other benefits that come with using a cloud service.

If you're training deep learning models and are looking for a cost-effective way to do so, cloud GPU rental services are definitely worth considering. The cost of consumer-grade GPUs makes purchasing them less than stellar.

Cloud providers offer reliable and powerful cloud GPU instances that can help you reduce expenses for large datasets and faster training. With cloud GPU rental services, you can focus on building better models without breaking the bank, and access the best GPUs right away.

So cloud GPUs are the way to go if you have large datasets and need faster training times. Cloud GPU rental services can provide the necessary computing resources without breaking the bank.


GPU prices continue to increase, but there exists a happy and unexpected middle ground for AI applications.

Blog