
## Background
vLLM is an open source framework for Large Language model inference. It specifically focuses on throughput for serving and batch workloads. This is important for building apps for multiple users and at scale.

vLLM provides an OpenAI compatible server, which means that you can integrate it into chatbots, and other applications

As companies build out their AI products, they often hit roadblocks like rate limits and cost for using these models. With vLLM on Vast, you can run your own models in the form factor you need, but with much more affordable compute. As inference grows in demand with agents and complicated workflows, vLLM on Vast shines for performance and affordability where you need it the most.


This guide will show you how to setup vLLM to serve an LLM on Vast. We reference a notebook that you can use [here](/uploads/serving_vllm_on_vast.ipynb)

## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account and add credits, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving an LLM, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU with decent RAM since we're going to serve a single small model. `vLLM` also requires Cuda version 12.1 or higher, so we will filter for that as well. We will query the vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.1' 
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 


```bash 
vastai create instance <instance-id> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 40 --args --model stabilityai/stablelm-2-zephyr-1_6b
```

## Connecting and Testing:
To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view.png)

We will copy over the IP address and the port into the cell below.
```bash
# This request assumes you haven't changed the model. If you did, fill it in the "model" value in the payload json below
curl -X POST http://<IP-Address>:<Port>/v1/completions -H "Content-Type: application/json"  -d '{"model" : "stabilityai/stablelm-2-zephyr-1_6b", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

## Advanced Usage: Serving a Quantized Llama-3-70b Model:
Now that we've spun up a model on vLLM, we can get into more complicated deployments. We'll work on serving this specific quantized Llama-3 70B [model](https://huggingface.co/casperhansen/llama-3-70b-instruct-awq).

With this quantized model, we can easilly serve this model on on 4 4090 GPU's.

### What is different this time around:
1. The model string - we need to use the new model id.
2. We're going to use 4 GPU's instead of just 1.
3. We need to provision much more space on our system to be able to download the full set of weights. 100 GB in this case should be fine.
4. We need to set up tensor parallelism inside vLLM to split up the model across these 4 gpus. 
5. We need to let vLLM know that this is a quantized model

First, we will search for instances that match our needs
``` bash
vastai search offers 'compute_cap >= 800 gpu_ram >= 24 num_gpus = 4 static_ip=true direct_port_count > 1 cuda_vers >= 12.1' 
```

In our instance creation, we will increase our disk usage to 100GB.

Then, we will tell vllm to: 1. use the specific model, 2. split across 4 GPU's, and 3. Let it know that it is in fact a quantized model.

``` bash
vastai create instance <Instance-ID> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 100 --args --model casperhansen/llama-3-70b-instruct-awq --tensor-parallel-size 4  --quantization awq 
```

### Other things to look out for with other configurations:

If you are downloading a model that needs authentication from the huggingface hub, passing `-e HF_token=<Your-Read-Only-Token>` within vast's `--env` variable string should help.

Sometimes the full context of a model can't be used given the space allocated for vLLM on the GPU + the models size. In those cases, you might want to increase `--gpu-memory-utilization`, or decrease the `max-model-len`. Increasing `--gpu-memory-utilization` does come with CUDA OutOfMemory Issues that can be hard to predict ahead of time.

We won't need either of these for this specific model and GPU configuration.

### Testing:

Copy the IP address from your instance once it is ready, and then we can use the following code to call it. Note that while your server might have ports ready, the model might not have downloaded yet as it is much larger this time. You can check the status of this via the logs to see when it has started serving.

```bash
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'casperhansen/llama-3-70b-instruct-awq',
    'prompt': 'Hello, how are you?',
    'max_tokens': 50,
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/completions', headers=headers, json=json_data)
print(response.content)
```


## Conclusions
Model inference is expensive, and leveraging more affordable compute/models makes a huge difference for engineering teams in terms of margins, and shipping velocity.

Using vLLM on Vast is perfect for this, pairing Vast's access to affordable compute with the simplicity and State of the Art throughput of the vLLM backend.

vLLM is a great beginning to building Generative AI Apps. We will continue to explore using this tool more with Vast in future posts.

Llama-3 is already ready to go on Vast to start experimenting and building!


Serving Online Inference with vLLM API on Vast.ai


NVIDIA's highly anticipated next-generation GH200 Grace Hopper Superchip platform is finally available. Perfectly suited for the era of accelerated computing and generative AI, it's essentially the world's most powerful GPU paired with the most versatile CPU. 

  

With its scalable design, the NVIDIA GH200 can handle the most complex generative AI workloads, from large language models (LLMs) and recommender systems to vector databases, graph neural networks (GNNs), and beyond. 

  

"To meet surging demand for generative AI, data centers require accelerated computing platforms with specialized needs," [said](https://nvidianews.nvidia.com/news/gh200-grace-hopper-superchip-with-hbm3e-memory) NVIDIA founder and CEO Jensen Huang. This is precisely the gap that the GH200 aims to fill. 

  

Huang added, "[It] delivers this with exceptional memory technology and bandwidth to improve throughput, the ability to connect GPUs to aggregate performance without compromise, and a server design that can be easily deployed across the entire data center."

  

### Groundbreaking Memory and Bandwidth

  

The GH200's architecture redefines high-performance computing (HPC). It integrates the cutting-edge Hopper GPU and the versatile Grace CPU into a single superchip – bridged by the lightning-fast, memory-coherent NVIDIA NVLink Chip-2-Chip (C2C) interconnect. 

  

The NVLink-C2C interconnect is the heart of the GH200 Grace Hopper Superchip. The power of its coherent memory is pretty incredible; the interconnect delivers 900GB/s of bidirectional CPU-GPU bandwidth. That's seven times the performance compared to the PCIe Gen5 connections found in conventional accelerated systems! Plus, interconnect power consumption is slashed by more than 5X.

  

With NVLink-C2C, applications can oversubscribe the GPU's memory, harnessing the high-bandwidth memory of the Grace CPU directly. The GH200 offers up to 480GB of LPDDR5X CPU memory, meaning that – combined with 96GB HBM3 or 144GB HBM3e – the Hopper GPU can directly access up to 624GB of fast memory. 




![](/uploads/wxoZpDKUOFRqBvNJkCLco6gy3-gBofvjPwirVSYuHDdHVBbFVT-JDjSOQHIYWTSV6VTnVj4U5kFJ9lhfUeHalOA5N0QRdDAV0TxByVawrhTyJID0yActvuTghOYzziRHm9xDCI0xIwY3ubuD5nRSEas.png)


(Image credit: NVIDIA)

  

Key features of the NVIDIA Grace CPU include:

  

- 2X the performance per watt of conventional x86-64 platforms
    
- 72 Neoverse V2 Armv9 cores with up to 480GB of server-class LPDDR5X memory with error-correction code (ECC)
    
- Up to 53% more bandwidth at one-eighth the power per GB per second, compared to an eight-channel DDR5 design
    

  

Then there's the H100 Tensor Core GPU, based on the new Hopper GPU architecture, which showcases several innovative highlights:

  

- Unprecedentedly fast matrix computations by the new fourth-generation Tensor Cores, enabling a wider range of AI and HPC tasks
    
- Up to 9X faster AI training and up to 30X faster AI inference with its new Transformer Engine compared to the prior-generation NVIDIA A100
    
- Maximized quality of service for smaller workloads thanks to secure Multi-Instance GPU (MIG) partitioning the GPU into isolated, right-size instances
    

  

In a nutshell, if you need raw power, you've got it with the GH200. Unfortunately, it's so new that there hasn't really been much comprehensive benchmarking. But let's take a look anyway. 

### Benchmarks: Pitting the GH200 Against its Main Rivals

  

On the Linux benchmark site Phoronix, author Michael Larabel [published](https://www.phoronix.com/review/nvidia-gh200-gptshop-benchmark) some initial HPC benchmarks run on a GH200 workstation sourced from GPTshop.ai. Testing was done on the Grace CPU performance only.

  

The GH200 system in question was tested with 72 cores, a Quanta S74G motherboard, 480GB of RAM, and 960GB + 1920GB SAMSUNG SSD drives. (A complete description of specs and environment can be found on the Phoronix site at the above link.) Again, this benchmarking was merely preliminary, as it focused on CPU performance, and no power consumption numbers are available – but there were some interesting results nonetheless. 

  

According to the standard [HPCG](https://www.hpcg-benchmark.org/) memory bandwidth benchmark, the GH200 Grace CPU came in at a solid 41.7 GFLOPS. 

  

  ![](/uploads/OatdrZqcQK_Bp41PV_DBcUFPeSRnvYSJpF4O1uc1Q6D_dLyr4ESSPbeS5HzrYaK92ybZSAVi3ma0rKQ0FmlpXcgDBrhGdY06dP5C980mThA-hIlAexWMgn0yiXt3GXod1eAklyI8p1HdjCGyRpoEw-U.png)

Results for NVIDIA GH200 running HPCG benchmark. (Image credit: Phoronix)

5vFBmCRUcbfgb9PR7NpFoXpSeRSFqr47lr9V1YtOG-k2Aus1jPY1T217XD_EnBKKZKnn8MTFUs-Jx2lMYVR_8Ek4Q9eQcM-kml5l0C2B4wfzwdytunAxtEcGZNfi_pp-GQKy8jTPCrEpqRUtlrk8OXw

Another notable result came from the [NWChem](https://www.nwchem-sw.org/) benchmark, with the GH200 coming in second place at 1403.5 seconds.

  

![](/uploads/5vFBmCRUcbfgb9PR7NpFoXpSeRSFqr47lr9V1YtOG-k2Aus1jPY1T217XD_EnBKKZKnn8MTFUs-Jx2lMYVR_8Ek4Q9eQcM-kml5l0C2B4wfzwdytunAxtEcGZNfi_pp-GQKy8jTPCrEpqRUtlrk8OXw.png)

Results for NVIDIA GH200 running NWChem benchmark. (Image credit: Phoronix)

  

On a geometric mean basis across all the benchmarks, the GH200 Grace CPU performance was perfectly respectable overall. 

  
![](/uploads/O_VgErmTsszfyrnSV20nbY-BdZb4QvnVJe7LOp55l33JKKI3JFoQSRpWU2lia3g_QlqKTwi4_1VXjUQ2gE1iHKNYy_A4JkgtMsPnn5uzc7TBf9JjVzH1rifiNFA_n17d4TI_t7MB6bsjIMCMYGMcrXo.png)

(Image credit: Phoronix)

  

A set of GPU benchmarks run by researcher Simon Butcher [compared](https://blog.hpc.qmul.ac.uk/benchmarking-grace-hopper-nodes/) performance of the PyTorch ResNet50 [training recipes](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnet50v1.5) published by NVIDIA. The ImageNet 2012 dataset (150GB) was used, and training was run for 90 epochs, or about an hour. The GH200 showed excellent performance here.

  

![](/uploads/qlwGIbeeRNXXquUTSoF6EDJmoss443fwqWUEQg9PtgAVPwIHTNTbi0U5BsEFx3Wx2dX10b0gfTdlgMfQxU_kHulwTU5QaOGlz7Oy9U8GBpdkvj4pCE7wjW7PJzfU_lTK-mdd3VUlbSjUaXEx2RVtloA.png)

(Image credit: Simon Butcher, Queen Mary University of London)

  

NVIDIA also [published](https://resources.nvidia.com/en-us-dgx-gh200/nvidia-grace-hopper-superchip-datasheet) some performance comparisons that may be of interest.

  

![](/uploads/dXjLeW3nCbOSpjMxABHSZ6LP84CLuzszoNLvi2m9CD1Dxf9GgTYk17wr2YlxWxqNM5xF3U6GXV-E6JFILwTqmj4zEmdd9Rv5MNAUNK9xaa6YeTfM9krmcLeMRaUQ4eF8qt_G0-xW81JoRiaEz4157xQ.png)

(Image credit: NVIDIA)

  

### Final Thoughts

  

The NVIDIA GH200 Grace Hopper Superchip delivers the performance required for truly giant-scale AI and HPC applications running terabytes of data. Whether you're a scientist, an engineer, or a massive data center, you'll find it up to the task. 

As for what the future holds? Nvidia [recently announced](https://www.tomshardware.com/pc-components/gpus/nvidias-next-gen-ai-gpu-revealed-blackwell-b200-gpu-delivers-up-to-20-petaflops-of-compute-and-massive-improvements-over-hopper-h100) the GH200's successor: Grace Blackwell B200, the next-generation data center and AI GPU.

  

Here at Vast.ai, we're proud to say that we recently added ARM64 and GH200 support to our lineup. On our GPU rental platform, just about anyone can get access to top-tier GPUs at affordable rates for a variety of computational needs. You can skip the cost-prohibitive investment in hardware and get right to work today – we'd love to help you discover the possibilities!

NVIDIA's highly anticipated next-generation GH200 Grace Hopper Superchip platform is finally available. Perfectly suited for the era of accelerated computing and generative AI, it's essentially the world's most powerful GPU paired with the most versatile CPU.

Examining the NVIDIA GH200


Just this past year, NVIDIA introduced a groundbreaking security feature in the Hopper architecture to protect sensitive data, AI models, and applications in use. The new feature, called NVIDIA Confidential Computing, offers a hardware-based solution for securely processing data and code that is in use, preventing unauthorized access and modification. 

  

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible – while you also get to enjoy the cutting-edge acceleration of NVIDIA H200 and H100 Tensor Core GPUs.

  

### What Is NVIDIA Confidential Computing?

  

To put it simply, "confidential computing" (CC) protects data-in-use by performing computation in a trusted execution environment (TEE) that is both hardware-based and attested. 

  

The NVIDIA H200 and H100 GPUs have a CVM (confidential virtual machine) TEE on the CPU that is anchored in an on-die root of trust (RoT). When it boots in CC-On mode, hardware protections that provide confidentiality and integrity of code and data are enabled. (In this context, "confidentiality" means that an attacker cannot access any data or code, and "integrity" means that they cannot modify the execution.) 

  

Here's how it works:

  

1. A chain of trust is established via a GPU boot sequence with a secure and measured boot.
    

  

2. An SPDM (Security Protocol and Data Model) session facilitates a secure connection to the driver in a CPU TEE. 
    

  

3. An attestation report with a cryptographically signed set of measurements is generated, verifying system integrity.
    

  

In order to run a confidential workload, the GPU must be authenticated as a legitimate NVIDIA GPU that supports confidential computing (a process involving a device-unique private key fused onto each GPU corresponding to a certified public key); the GPU must not have been revoked for CC; and the GPU attestation report must be verified. 

  

Upon successful verification, the NVIDIA driver in the CVM establishes a secure channel, via a session key, with the GPU hardware TEE in order to transfer data, perform computation, and retrieve results. The CVM and GPU communicate through a shared memory region outside the CVM since the CPU prevents the GPU from directly accessing a CVM's memory. They use AES-GCM – an advanced encryption standard – to prevent the host system from reading this communication. 

  

Finally, the GPU copies and decrypts all inputs to its internal memory, where everything runs in plaintext and direct access is blocked. Performance counters are disabled as well, since these could be used for side-channel attacks. 

  

In this way, a confidential-computing environment is extended from a CVM (or a secure enclave) to a GPU, with attestation, encrypted communication, and memory isolation making it possible. 

  

### Benefits & Positive Impacts

  

The benefits of NVIDIA Confidential Computing are significant:

  

- Hardware-Based Security and Isolation – You can achieve full isolation of virtual machines (VMs) in any environment – on-premises, at the edge, or on the cloud. Your entire workload is secured with these built-in hardware firewalls that provide unprecedented protection. 
    

  

- Verifiability with Device Attestation – You will always rest assured that only authorized end users can deploy data and code for execution in the H200 or H100's TEE. Plus, through device attestation, you'll know that you're dealing with an authentic NVIDIA GPU, the firmware hasn't been modified, and updates have been done as expected. 
    

  

- Protection from Unauthorized Access – Your sensitive data, AI workloads, and intellectual property are protected, with confidentiality and integrity preserved at all times. All unauthorized entities (including the hypervisor, cloud provider, host operating system, and even anyone with physical access to the infrastructure) are blocked from viewing or modifying the AI application and data during execution. 
    

  

- No Application Code Change – As NVIDIA puts it, the new confidential computing feature "just works," without any code changes required for your GPU-accelerated workloads. 
    

  

As you can imagine, confidential computing-enabled GPUs usher in a whole host of use cases where security, privacy, and regulatory compliance are paramount. For instance, intellectual property protection is now possible for proprietary AI models, even when these models are distributed and deployed at scale on shared or remote infrastructure. 

  

Privacy and confidentiality can also now be preserved in AI training and inference – most notably in industries like healthcare, finance, and the public sector, where data may be sensitive and/or regulated. On top of that, collaboration between multiple parties can be done securely when building and improving AI models across participating sites, for use cases like medical imaging, drug development, and fraud detection. 

  

Another bonus: organizations, governments, and individuals can outsource AI workloads to a cloud provider while still ensuring protection from the underlying infrastructure. 

  

### Final Thoughts

  

It's exciting that confidential computing is now possible for even the most demanding workloads! Granted, for the vast majority of use cases, CC-enabled GPUs may arguably be overkill. But for those who need or prefer absolute confidentiality and security, NVIDIA Confidential Computing in the Hopper architecture can now provide that for you. 

  

At Vast.ai, we understand how important privacy and security can be for our customers. It is our hope that you'll find exactly what you need on our cost-effective, on-demand cloud GPU rental platform. 

  

In our global network of compute providers on Vast, some hosts are experimenting with NVIDIA Confidential Computing. Complete support for this feature is not yet available. Join our Discord for the latest product updates and up-to-date support.

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible 

Absolute Security with NVIDIA Confidential Computing


# Train a 70b language model on a 2X RTX 4090 with QLoRA and FSDP

## Overview

A new exciting [announcement from Answers.ai](https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html) demonstrated a way to train a larger model, such as Llama 2 70B on 48GB of GPU RAM. Furthermore it can run on multiple GPUS, so it is possible to train a model on a 2X 4090 instance!

The new [fsdp_qlora](https://github.com/AnswerDotAI/fsdp_qlora/tree/main) Open Source software has a number of training options and examples. The software builds on two big ideas:

**QLoRA (Quantized LoRA)** combines quantization (using fewer bits to store model weights) and LoRA (Low-Rank Adaptation, which adds small trainable matrices to a frozen base model). This allows training models larger than the GPU's memory by using a quantized base model with trainable LoRA adapters. However, QLoRA still has limitations, such as requiring expensive GPUs and restricting sequence lengths and batch sizes due to memory constraints.

**FSDP (Fully Sharded Data Parallel)** is a library developed by Meta that efficiently splits a large model across multiple GPUs, allowing them to be used simultaneously. It improves upon the previous gold standard, DDP (Distributed Data Parallel), which required the full model to fit on each GPU. FSDP enables training large models that exceed the memory of a single GPU by sharding the model parameters across multiple GPUs.

This guide will show you how to rent a 2X instance on Vast.ai to run this software on-demand. 

## How to run fsdp_qlora on a Vast.ai 2X 3090/4090 instance

Vast.ai has a large supply of RTX 3090 and RTX 4090 GPUs well suited to run fsdp_qlora. You will need a Vast.ai account setup with credits. This guide uses the [Vast Python CLI](https://github.com/vast-ai/vast-python). 

### Create your Vast account, add credits and install the Vast CLI on your local machine. 

Setup the Vast.ai account and verify your email address. Add some credits to your account for the GPU rental. Then [install the CLI](https://cloud.vast.ai/cli/). 

### Search the Vast.ai marketplace

We will need to find suitable GPUs. From my testing, for training Llama 70B it required 200GB of system RAM. This query will return on-demand GPU offers for suitable 2X 4090 instances sorted by download bandwidth

For 2X 4090 instances
```
vastai search offers "gpu_name=RTX_4090 cpu_ram>=130 disk_space>140 num_gpus=2" -o "inet_down-"
```

For 2X 3090 instances 
```
vastai search offers "gpu_name=RTX_3090 cpu_ram>=130 disk_space>140 num_gpus=2" -o "inet_down-"
```

#### Create the Vast.ai instance

The list of offers will have a lot of details about the machines that are available. To pick one, you will need the offer id. Then use that in the create command along with the Pytorch Cuda Devel template hash.

This command will create an instance on the offer supplied, using that template and allocating 200GB of disk.

Replace the XXXXXXX with the offer ID from the marketplace.

```
vastai create instance XXXXXXX --template_hash e4c5e88bc289f4eecb0c955c4fe7430d --disk 200
```

#### SSH into the Vast.ai instance

Wait a few minutes for the instance to boot up. You can use the [webui](https://cloud.vast.ai/instances/) to get the direct SSH command by clicking on the >_ button.

To connect via SSH, you will simply use the command copied from the instance card. It will look similar to this:

```
ssh -p <port> <sshURL> -L 8080:localhost:8080
```

#### Install fsdp_qlora and login to Hugging Face

```
git clone https://github.com/AnswerDotAI/fsdp_qlora
pip install llama-recipes fastcore --extra-index-url https://download.pytorch.org/whl/test/cu118
pip install bitsandbytes>=0.43.0
pip install wandb
```

Login to Hugging Face by pasting in your API key to download the model

```
huggingface-cli login
```

Optional: login to wandb to enable logging to Weights and Balances

```
wandb login
```

#### Optionally setup HQQ

HQQ is a fast and accurate model quantizer. It can be used instead of bitsandbytes. If you want to use it, you will need to install it first.

```
git clone https://github.com/mobiusml/hqq.git
```

Install HQQ

```
cd hqq && pip install .
```

#### Train the model

For complete options look at the [Github repo](https://github.com/AnswerDotAI/fsdp_qlora/tree/main). This example will finetune Llama 70B using hqq_lora.


#### Example: finetune Llama 2 70B with context length of 512

```
cd ~/fsdp_qlora
```

```
python train.py \
--model_name meta-llama/Llama-2-70b-hf \
--batch_size 2 \
--context_length 512 \
--precision bf16 \
--train_type qlora \
--use_gradient_checkpointing true \
--use_cpu_offload true \
--dataset alpaca \
--reentrant_checkpointing true
```

#### Troubleshooting

On some instances, the system seems to only run on the CPU. The GPUs might not being recognized. Monitor the [fsdp_qlora repo](https://github.com/AnswerDotAI/fsdp_qlora/tree/main) for help.

For any issues related to the instance or working with Vast.ai, click on the website support chat at the bottom right corner of [cloud.vast.ai](https://cloud.vast.ai/) for immediate support.

Sometimes it can help to export the cuda visible devices to help with an issue where the GPUs do not load and the CPU does all the work.

```
export CUDA_VISIBLE_DEVICES=0,1
```

A new approach to use both QLoRA and FSDP to do full training on a 2X 3090/4090

Train a 70b language model on a 2X RTX 3090/4090 with QLoRA and FSDP


We're pleased to reveal the latest round of updates here at Vast.ai! Our mission is to provide our customers with the best possible GPU rental platform, and we believe that every update is an opportunity to make our product shine that much more. For instance, we've recently added ARM64 and GH200 support – so we're now able to support ARM64 platforms including the Grace Hopper superchip!

But before we delve into some of our bigger updates below, first up on the agenda are bug fixes. We strive to address every minor hiccup and bug as quickly as possible, so that you can enjoy a seamless, uninterrupted experience on Vast. Here are a few of our latest bug fixes:

- Linking to docs now aligns correctly
- Random logouts from the webUI have been fixed
- Users don't see the host setup command unless they are in a hosting account
- Fixed an off-by-1 error with the CLI command ssh-url
- Everywhere a template name is mentioned, it is now linkable with a hash ID
- An issue with number of referrals not displaying has been corrected
- Fixed issue with template editor in search interface

![](/uploads/feb24Image.png)

### NVIDIA Cloud GPU Updates

Over the past few weeks, we've made some important enhancements to our platform. It's our hope that these updates will help save you time and effort, as well as maximize your goals and results on Vast. The following are some highlights of the latest changes:

- As mentioned, we've added ARM64 and GH200 support; we now support ARM64 platforms including the Grace Hopper superchip
- Added a template for NVIDIA RAPIDS - [https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593](https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593)
- Changed Autoscaler CLI commands to work with template linking
- Added a search feature for templates and hover help (tooltip) to the template page
- Implemented flexible date format for CLI so it isn't just Unix epoch time
- Improved search API for performance
- Added a DLPerf filter

Please check out the full change log for our February 2024 product updates at the bottom of this page. Getting to know the latest capabilities will help you make the most of your time on Vast!

If you're looking for affordable, on-demand cloud GPU rental, you've come to the right place – and if you need some assistance getting started, we have the experience and know-how to guide you through whatever your project requires. Join our thriving Discord community for support or reach out to our customer service team to get all your questions answered! We would love to help you explore the possibilities that Vast holds for you today. Please feel free to contact us through the chatbox at the bottom right or drop us a line at [support@vast.ai](mailto:support@vast.ai) whenever you'd like!

### Change Log

- Added ARM64 and GH200 support; we now support ARM64 platforms including the Grace Hopper superchip
- Added a CLI parameter on create instance for a template hash; i.e. vastai create instance --template_hash HASH
- WebGUI uses cancel_unavail by default (instances are not created stopped)
- Added a template for NVIDIA RAPIDS - [https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593](https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593)
- Upgraded our SD guides with ai-dock template and info on setting a strong password
- Improved Autoscaler to use template info more completely
- Changed Autoscaler CLI commands to work with template linking
- Bugfix: args launch now doesn't accept an onstart script
- Bugfix: linking to docs now aligns correctly
- Bugfix: random logouts from the webUI fixed
- Bugfix: users don't see host setup command unless they are in a hosting account
- Bugfix: fixed an issue with number of referrals not displaying
- Bugfix: fixed host link to Stripe connect earnings dashboard
- Bugfix: on instance to instance copy, copy to /root is fixed
- Bugfix: fixed issue with template editor in search interface
- Added a search feature for templates
- Added a "Create new template" feature on templates
- Improved search API for performance
- Added the team name and member name for the API/team contexts on the account page
- Added hover help to the template page
- Added DLPerf filter
- Added flexible date format for CLI so it isn't just Unix epoch time
- Bugfix: everywhere a template name is mentioned, it is now linkable with a hash ID
- Bugfix: loading bar was showing before transfer commands, it is now fixed
- Bugfix: template tag dropdown selector is fixed, as it wasn't updating correctly
- Bugfix: fixed an off-by-1 error with the CLI command ssh-url


February 2024 Product Update


When you need high-powered resources to power complex calculations, the more you have the better. In a recent interview, Elon Musk claimed that he heard of a gigawatt class data center being built. Many responders think that Elon is referring to a rumor that –deep in the deserts of Kuwait–, the construction of a new data center is being built to compete with today’s most powerful compute power resources.

Elon Musk likely heard the rumor from [news](https://www.theinformation.com/articles/ai-agenda-the-mysterious-ai-data-center-startup-hiring-from-aws-azure-meta) of a startup poaching technology workers from AWS, Azure, Meta, and other large Silicon Valley companies. In 2023, the race to be a leader in artificial intelligence and machine learning became even more competitive. To be the best, organizations need immense GPU power, usually outsourced to the cloud for its quick deployment and scaling. Large cloud providers built the data centers to power calculations, and businesses can rent the compute resources necessary to power any project – from small personal projects to large enterprise-tier analytics.

The rumor of a Kuwait data center came in the summer of 2023. A company named Omniva –also known as Moneta Tech, and Moneta Systems– was interested in building the most powerful data center in the world. Their goal was to rival Google and Microsoft in technology. Originally, the data center idea was targeted towards cryptomining, but later became an idea to target businesses innovating for the latest artificial intelligence boom.

As more information became available, the gigawatt claims began to unravel. The facility was said to be about 430,000 square feet with total available power of 864 megawatts. A proposed budget of $200 million immediately sent red flags. As a benchmark, it’s reported that a Google data center costs between $1000 and $3000 per square foot. This means that at the very cheapest, a data center of Omniva’s size should cost $430 million, over double the proposed budget.

Cooling systems were ignored for manned spaces in exchange for employee cooling helmets. Fire systems were removed in exchange for fire extinguishers. Anyone with data center experience knows that the proposed planning and safety aren’t feasible.

Next is Omniva’s claim to offer gigawatts of power to customers. NVIDIA continues to dominate GPU performance and offer the best in compute power. To reach the compute power rumored to be available, Omniva would need the latest generation of GPUs, namely the new NVIDIA Blackwell B100. The B100s haven’t hit the market yet, but NVIDIA teased consumers with the promise that their next generation GPUs should launch in 2024.

As of today, any facility rumored to be built with gigawatt power has never been confirmed. The power promised in 2023 wasn’t available, and many of the plans presented to the public didn’t have the backing of professionals. The cloud GPU space still has the computational power to support artificial intelligence and cryptomining, but businesses and consumers should stick to a trusted provider.

For the absolute best in compute power with the latest in GPU technology, [Vast.ai](https://vast.ai/) has the resources you need for machine learning, artificial intelligence, and other high-powered analytics projects for your cloud GPU needs.


Coming up with just the right words, and keeping track of the ideas that you’re weaving together - this is difficult work. The difficulty can even make it seem that there might be better ways to be spending your time.

Is a Gigawatt Class Data Center Being Built in Kuwait?


The landscape of high-performance GPUs is continually evolving, particularly in the field of AI and ML. With the recent introduction of AMD's Radeon RX 7800 XT and Radeon RX 7700 XT, there has been significant interest in how these GPUs compare with established models like NVIDIA's RTX 4090, especially for AI applications. This article aims to provide a detailed comparison between these GPUs, focusing on their technical specifications, performance metrics, and suitability for AI tasks.

# AI Suitability of the AMD Radeon RX 7800 XT and Radeon RX 7700 XT

Unveiled in 2023, the Radeon RX 7800 XT and Radeon RX 7700 XT have piqued the interest of professionals in the AI field. While initially geared towards high-resolution gaming, these GPUs also hold potential for AI applications.

## AMD Radeon RX 7800 XT

This model boasts 60 AMD RDNA 3 unified compute units and offers significant performance-per-dollar value. Its high-speed 16GB GDDR6 VRAM and a game clock speed of 2124 MHz (boosting up to 2430 MHz) make it a competent choice for AI tasks that require substantial memory and computational power.

## AMD Radeon RX 7700 XT

With 54 AMD RDNA 3 compute units and 12GB of GDDR6 VRAM, the 7700 XT, while slightly less powerful than the 7800 XT, still offers robust performance for AI applications. Its game clock of 2171 MHz and boost clock up to 2544 MHz ensure efficient processing for complex algorithms.

Both GPUs leverage the advanced AMD RDNA 3 Architecture, including unified ray tracing and AI accelerators. This architecture is particularly beneficial for machine learning models that involve image processing and deep learning computations. However, given that Radeon cards primarily focus on gaming, the NVIDIA RTX 4090 is still a stronger candidate for AI Applications.

# The NVIDIA RTX 4090 Still Stands as a Versatile Choice

The [NVIDIA RTX 4090](https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/) remains a solid choice for a variety of users for AI purposes, despite newer models like AMD's Radeon RX 7800 XT and RX 7700 XT. It is a well-established GPU in the AI community, known for its exceptional computational power and efficiency. It also features NVIDIA's Deep Learning Super Sampling (DLSS), an AI technology that enhances image quality without overburdening the GPU.

In terms of energy efficiency, the RTX 4090 is designed to provide high performance without excessive power consumption. Its effective cooling system also ensures it operates at optimal temperatures, which is key for long-term use. The GPU's compatibility with a wide range of software and hardware adds to its versatility, making it a suitable choice for a wide variety of applications.

Considering its advanced features and future-proof design, the RTX 4090 is a good long-term investment for those needing a GPU that can handle both current and upcoming software. While AMD’s Radeon RX 7800 XT and RX 7700 XT have their specific strengths, the RTX 4090's broad capabilities make it a reliable and efficient choice in the GPU market. However, these GPUs can be hard to find, or incredibly costly due to high demand and low supply.

# Leveraging Vast.ai for GPU Accessibility

While GPUs like NVIDIA’s RTX 4090 offer impressive capabilities, their availability and cost can be prohibitive for many users. This is where [Vast.ai](http://vast.ai) comes into play, providing a solution to these challenges.

## Vast.ai: A Solution for Easier GPU Access

Vast.ai is a platform that allows users to rent high-performance GPUs, like the RTX 4090, at more affordable rates. This service is particularly useful for those who need GPU power for specific projects or temporary tasks but find purchasing such hardware impractical due to cost or limited usage.

On the Vast.ai platform, users can find a range of GPUs available for rent, including the latest models. The process is straightforward: users search for the GPU that fits their needs and budget, and then rent it for the required period. This approach eliminates the need for a significant upfront investment in purchasing a high-end GPU.

## Why Use Vast.ai?

The primary advantage of using Vast.ai is cost-effectiveness. It allows individuals, small businesses, and researchers access to top-tier GPUs for their computational needs without the full cost of ownership. This flexibility is perfect for those who require advanced GPU capabilities for short-term projects or occasional use.

# Final Thoughts

In summary, when comparing NVIDIA's RTX 4090 with AMD's Radeon RX 7800 XT and 7700 XT for AI applications, NVIDIA is a more catered choice. The RTX 4090 stands out for its proven performance and advanced features, making it ideal for demanding AI tasks. The AMD options are more gaming-oriented, and while their potential for AI applications is solid, a dedicated AI GPU may be more appropriate. If you’re in need of a GPU for AI development, services like Vast.ai offer the RTX 4090, along with other high-end GPUs, allowing for the development of AI without the hefty investment in hardware.


High-Performance GPUs for AI: Nvidia RTX 4090 vs AMD Radeon RX 7800 XT and 7700 XT 


# January 2024 Product Update

It's a new year and we're back with a bunch of new and exciting updates! Our goal at Vast.ai is to continue to bring our customers the best-in-class service they've come to expect – and always look for ways to elevate our product.

To that end, we've recently added a Teams feature so that multiple users can manage a single account! Roles and permissions are fully customizable to give you the flexibility and security you need. Teams is a beta feature and CLI-only for now, but available to anyone who's interested; see more details below.

Some other recent changes, of course, include bug fixes to keep things running smoothly at all times. The following are a few of the latest fixes we've implemented:

- Templates load now for logged-out users
- Unlisting and relisting a machine with new date correctly extends current contracts
- Host machine page now shows the correct current reserve price factor
- Fixed issue with instance bandwidth tracking breaking after 1TB is used
- Deposit Refunds is now based on when the money was deposited
- Stopping offline instances now stops them when they come back online

### NVIDIA Cloud GPU Updates

Here at Vast.ai, our dedicated team works hard to enhance the user experience for our valued customers. Every month we provide a log of the improvements and adjustments we've made to the platform, to help you make the most of your time with Vast. Below are some highlights of our most recent product updates:

- Oauth has been added for Google and GitHub!
- Launched teams (beta) with fully customizable roles & permissions - [https://cloud.vast.ai/teams/](https://cloud.vast.ai/teams/)

- Read the team documentation here - [https://vast.ai/docs/team/introduction](https://vast.ai/docs/team/introduction)

- Major overhaul of our authorization/permission system, now based on fine-grained API keys with fully customizable fine-grained permissions - [https://vast.ai/docs/cli/roles-and-permissions](https://vast.ai/docs/cli/roles-and-permissions)
- Added a tax field for VAT (or similar) to invoices
- Added 2FA for account authentication via SMS
- Improvements to mobile viewing and improved contained startup time
- Added recycle as a command to pull new image, destroy and recreate instance (keeps the volumes)

For a complete list of changes and bug fixes made since our last update, please check out the January 2024 change log below.

As always, we're delighted to serve new and old customers as a reliable provider of cost-effective, instant-access GPU rental services. Our user-friendly platform is supported by an active Discord community and a dedicated customer support team ready to address your queries. Please feel free to get in touch through our chatbox at the bottom right, join our Discord server for a whole host of helpful resources, or email us at [support@vast.ai](mailto:support@vast.ai). We'd be happy to assist you in any way we can!

### Change Log

- Added Oauth for Google and GitHub!
- Bugfix: templates load now for logged-out users
- Bugfix: reset password edge cases fixed
- Added Teams feature for multiple users to manage a single account! (Beta and CLI only for now.) Here is the documentation: [https://vast.ai/docs/team/introduction](https://vast.ai/docs/team/introduction)
- Updated our API docs, fixed some issues and included a section for teams
- The reserve instance deposit confirmation modal now includes amount deposited and more info
- Added a tax field for VAT (or similar) to invoices
- Added A100X to search filter
- Removed min_cuda, max_cuda from the templates (get those from the Docker image instead)
- Bugfix: unlisting and relisting a machine with new date correctly extends current contracts
- Bugfix: internet speedtests on recent Ubuntu 22 machines fixed
- Bugfix: scroll lock issues fixed
- Bugfix: host machine page now shows the correct current reserve price factor
- Bugfix: earnings and payout invoices now showing for client referral earnings
- Bugfix: fixed issue with instance bandwidth tracking breaking after 1TB used
- Launched teams (beta) with fully customizable roles & permissions - [https://cloud.vast.ai/teams/](https://cloud.vast.ai/teams/)
- Major overhaul of our authorization/permission system, now based on fine-grained API keys with fully customizable fine-grained permissions - [https://vast.ai/docs/cli/roles-and-permissions](https://vast.ai/docs/cli/roles-and-permissions)
- Added API key management in the GUI on the account page
- Added 2FA for account authentication via SMS
- New CLI option 'cancel_unavail' added to fail new instance creation if it can't immediately schedule running
- New CLI option 'recycle' to destroy and recreate instance in place
- New CLI command to fetch machine reports
- Updated our invoicing to better display instance pre-pay
- Improvements to mobile viewing
- Added ubuntu_version search filter to GUI
- Improved container startup time
- Bugfix: on the instance button the auto web port is removed (only map open/connecting if OPEN_BUTTON_PORT is set)
- Bugfix: Deposit Refunds now based on when the money was deposited
- Bugfix: terminating a reserved instance now records the penalty (any deposit credit lost) as a charge
- Bugfix: host maintenance window date is now in standard format
- Bugfix: Docker login repo text clarified
- Added new team page in the GUI to manage team membership & access (requires active team)
- Added recycle as a command to pull new image, destroy and recreate instance (keeps the volumes)
- Added option on instance page to group by label, sort by label, and/or have tabs for labels with a number of instances or autoscaler endpoints
- Account page has a UI panel for API key management
- Added DATA_DIRECTORY to templates which is used by instance-to-instance copy and cloud sync as the default directory for copy commands
- Referrals now show number of referred users
- Improved 2FA flow
- Unified the earnings and billing date picker UI
- Bugfix: fixed some issues with NVMe SSD product name detection in Daemon
- Bugfix: fixed issue where slow blocking nvidia-smi call was causing timeouts on some machines/VMs in Daemon
- Bugfix: fixed rare port allocation conflict bug in Daemon
- Bugfix: teams CLI now works for inviting new team members
- Bugfix: stopping offline instances now stops them when they come back online
- Bugfix: earnings PDF now shows correct sign for the platform fee
- Bugfix: add credit now shows up for prepaid instances


We are constantly looking for ways to improve and bring you a service that continues to innovate.

January Product Update


In the complex landscape of data center operations, understanding and adhering to various compliance standards is crucial. This article will explore the differences and similarities between Tier 2/3 data center classifications and compliance standards like HIPAA and ISO 27001, providing insights for businesses navigating these regulations.

# Understanding Data Center Tiers

Data centers are classified into four tiers by the Uptime Institute, based on their uptime and redundancy levels. Tier 2 and Tier 3 data centers are particularly relevant for growing businesses and large organizations.

Tier 2 Data Centers offer a single path for power and cooling but include some redundant and backup components. They promise an uptime of 99.741% per year, translating to up to 22 hours of downtime annually​​.

Tier 3 Data Centers provide multiple paths for power and cooling, with redundant systems that allow maintenance without downtime. They boast an uptime of 99.982% per year, equating to less than 1.6 hours of downtime annually​​.

These tiers offer a balance between performance, reliability, and cost, making them suitable for businesses that require consistent uptime without the need for the extensive redundancies of Tier 4.

## Compliance Standards: HIPAA and ISO 27001

Compliance standards like HIPAA and ISO 27001 are essential for data centers, especially those handling sensitive information.

HIPAA (Health Insurance Portability and Accountability Act) is a U.S. legislation focused on protecting sensitive patient information managed by healthcare organizations. It requires entities to implement physical, administrative, and technical safeguards for Protected Health Information (PHI), which can be supported by the infrastructure and redundancy offered by Tier 2 and Tier 3 facilities​​.

ISO 27001 is an international standard for information security management. It includes 114 controls in areas like access control, physical and environmental security, and incident management. While ISO 27001 certification does not equate to HIPAA compliance, its controls significantly overlap with HIPAA requirements​. Tier 2 and Tier 3 data centers can support ISO 27001 compliance through their security measures, operational sustainability, and staff expertise​​.

# Vast.ai's Datacenter Compliance 

Vast.ai has a network of dozens of datacenters that have ISO 27001 and TIER 2 or 3 ratings. Vast.ai has done due diligence and confirmed that these provider’s equipment is in a facility with an up to date compliance certificate.

To find Vast.ai providers, use the “Secure Cloud (Only Trusted Datacenters)” filter on the search page at https://cloud.vast.ai/create/

Vast.ai secure cloud has providers with at least one of the following certifications:

- ISO 27001
- TIER 2
- TIER 3

For production services where uptime and data security are vital, use the trusted datacenter filter and look for the blue datacenter label.

# Final Thoughts

Choosing a data center that aligns with compliance needs is crucial for businesses. Tier 2 and Tier 3 data centers offer balanced and cost-effective solutions for organizations requiring consistent uptime and robust security measures. Vast.ai's secure cloud offering empowers businesses to navigate this landscape efficiently, ensuring compliance and security. As the digital landscape evolves, staying compliant and resilient remains a top priority, a challenge that Vast.ai is well-equipped to meet.


Navigating Data Center Compliance: Understanding Tier 2/3 and HIPAA/ISO 27001 Standards


As the year comes to a close, we're back one more time with the latest updates here at Vast.ai! At the moment, we're primarily working on a few major features that will be unveiled in due course – but for now, we're pleased to announce the smaller ongoing product updates that continue to make our GPU rental platform better and better over time.

First up are bug fixes. We're proud of our team's response time in getting every hiccup dealt with as quickly as possible, which ensures we're able to deliver a seamless experience for our valued customers. The following are some of our latest fixes:

- Fixed internet speedtest script for Ubuntu 23+
- All Coinbase payments now display a link to the transaction
- Fixed issues with Google Drive token expiration
- Cuda version detection is now fixed for TensorFlow
- Fixed some de-verification issues
- Template editors now accept > and <
- Fixed issues with Dropbox connections

#### NVIDIA Cloud GPU Updates

As mentioned, more changes are on the way! But in the past few weeks, we've implemented some enhancements to our platform that should make Vast even more user-friendly and intuitive for both current and new customers. Here are a few highlights of the latest updates:

- Added RTX 5000 Ada to search page
- Updated CLI documentation - https://vast.ai/docs/cli/commands
- Updated the displayed text for Docker image in the template editor
- Discount factor now visible for reserved instance discounts
- Added machine OS version filter to CLI/API
- Updated recommended templates to use AI Dock's ComfyUI template

Plus, we've updated our [Terms of Service](https://vast.ai/terms) to reflect that credits purchased via credit card cannot be used for cryptocurrency mining. The refund policy has also been clarified. And as usual, you can find the full change log for our December 2023 product updates below for your convenience.

We're proud to be the market leader in low-cost, on-demand cloud GPU rental. With Vast, you can save 5-6X on GPU compute compared to other solutions – all in one simple interface with a supportive Discord community and a responsive customer service team to answer all your questions! We'd love to chat and help you discover what Vast can do for you today. You're welcome to contact us any time via the chatbox at the bottom right or email us at [support@vast.ai](mailto:support@vast.ai). In the meantime, happy holidays!

#### Change Log

- Added RTX 5000 Ada to search page
- Schedule maintenance doesn't require a machine that is listed
- Added strong warning about copying / or /root or /root/.ssh
- Updated our docs to include using the CLI from inside an instance
- Bugfix: fixed internet speedtest script for Ubuntu 23+
- Bugfix: all Coinbase payments now display a link to the transaction
- Bugfix: host end date fix
- Bugfix: Cuda version detection fixed for TensorFlow
- Updated the displayed text for Docker image in the template editor
- Bugfix: host UI schedule maintenance UI button fixed
- Bugfix: display on billing warning message
- Bugfix: fixed some de-verification issues
- Bugfix: template editors now accept > and <
- Bugfix: in the template editor extra filters can be overwritten even if connected to the Docker repo
- Terms of Service has been updated (no mining with credits purchased via credit card; refund policy clarified)
- Reset API key now asks for confirmation
- CLI has more explicit error messages for syntax errors
- Updated CLI documentation - https://vast.ai/docs/cli/commands
- Discount factor now visible for reserved instance discounts
- Added machine OS version filter to CLI/API
- Updated destroyed instance cleanup
- Updated recommended templates to use AI Dock's ComfyUI template
- Bugfix: fixed issues with Google Drive token expiration
- Bugfix: earnings page should not allow you to see charges and payments for export
- Bugfix: kaalia: auto-detection of NVML and bad GPU errors
- Bugfix: kaalia: fix issue with update_scripts not installing on new machines
- Bugfix: fixed issues with Dropbox connections


December product updates


# How to Run Mixtral 8X7B

Mixtral is the latest open source model from Mistral AI. The model was release with no documentation, explanation or other comment from Mistral AI on Dec 8th via a single Tweet of the [magnet link](https://x.com/MistralAI/status/1733150512395038967?s=20). Mistral AI has several open source LLM models that are popular including Mistral 7B. 

Mixtral 8X7B is notable in that it is a mixture of experts (MoE) model with exceptional ability. This guide uses some hacky implementations to get it to run. Once the model is out for a few months, it will certainly gain more support with open source tools.

To run it on Vast.ai, use a 8X 4090/3090 or a 4X A6000/A40 instance. You will need 120GB of total GPU RAM on the instance for the model. We use the typical development Pytorch recommended template and a modified version of Illama to run inference. This is a base model with no instruction fine tuning, so proper prompt techniques are helpful. 

## Steps:

1. **Select Template**:
   - Choose the [Devel template](https://cloud.vast.ai/?ref_id=43484&template_id=e4c5e88bc289f4eecb0c955c4fe7430d) for Pytorch by clicking that link or selecting it in Templates.

2. **Rent a Server**:
   - Select a 8X 4090/3090 or 4X A40/A6000
   - Add at least 120GB of disk space
   - Click the Rent button
   - Purchase credits if needed.
   - Refer to the [quickstart guide](https://vast.ai/docs/overview/quickstart) for help.

3. **SSH into the Machine**:
   - Obtain SSH information from the instance card by clicking on the >_ button 
   - Connect to the instance via the SSH command. It will look like this but have the port and server IP info of your instance:
     ```
     ssh -p <yourport> root@<yourserverip> -L 8080:localhost:8080
     ```

4. **Download Mixtral 8X7B Model Weights**:
   - Use a torrent file. This typically takes about 15 minutes, depending on server download speed.
   - To download the torrent file, run:
     ```
     apt install transmission-cli
     transmission-cli magnet:?xt=urn:btih:5546272da9065eddeb6fcd7ffddeef5b75be79a7&dn=mixtral-8x7b-32kseqlen&tr=udp%3A%2F%http://2Fopentracker.i2p.rocks%3A6969%2Fannounce&tr=http%3A%2F%http://2Ftracker.openbittorrent.com%3A80%2Fannounce
     ```

5. **Install Illama for Inference**:
   - Clone the repository:
     ```
     git clone https://github.com/dzhulgakov/llama-mistral
     ```
   - Install dependencies:
     ```
     pip install fire sentencepiece
     ```

6. **Run Mixtral 8X7B!**:
   - Load the model weights and run the prompts in `example_text_completion.py`. If you use 8 GPUS, append "--num-gpus 8" at the end of the run command like in the example below. If you use 2 GPUS, remove that line.:
     ```
     cd llama-mistral
     python example_text_completion.py ../Downloads/mixtral-8x7b-32kseqlen/ ../Downloads/mixtral-8x7b-32kseqlen/tokenizer.model --num-gpus 8
     ```
	 And there you go! You will get the output of the prompts that are currently listed 

7. **Modify Prompts**:
   - Install Nano:
     ```
     apt install nano
     ```
   - Edit the file:
     ```
     nano example_text_completion.py
     ```
   - Find and modify the prompts at the bottom.
   - Re-run the python file to get new responses.

Thank you for using Vast.ai to run the latest open source LLM, Mixtral 8X7B. Drop us a line and let us know what you think of the model.


Running the latest open source model from Mistral AI called Mixtral 8x7b on Vast.ai using 8X 4090s or 8X 3090s

How to run Mixtral 8X7B


The rise of GPU-centric tasks in Python environments means a balanced computing setup is as important as ever. While a powerful GPU handles parallel processing tasks efficiently, a good CPU is necessary to prevent bottlenecks and ensure smooth performance. A common misconception is that a strong GPU can make up for a weak CPU, but this is not the case. A weak CPU can stall, creating delays that lower performance, regardless of the GPU's strength.

The real issue lies in the CPU's role in managing a program’s workflow, including memory management, I/O operations, and providing instructions to the GPU. When the CPU lags, it can't feed instructions to the GPU quickly enough, causing bottlenecks. This problem is especially true in Python environments, whose language limitations can worsen the bottleneck.

# Why Does a Weak CPU Lead to Bottleneck?

In computing, the CPU and GPU work together. The CPU directs operations and sends instructions to the GPU for processing. However, when a CPU is weak or slow, it can't keep up with the GPU, causing a bottleneck. This means the GPU has to wait for the CPU to send instructions, which slows down the entire process.

Python environments can exacerbate this issue due to certain limitations. For instance, Python's Global Interpreter Lock (GIL) can further slow down the CPU, making the bottleneck issue more noticeable. The efficiency of running Python tasks largely depends on the CPU's capabilities. A stronger CPU can send instructions to the GPU faster, keeping the process running smoothly and ensuring better performance.

# Introducing AMD’s Threadripper: A More Powerful CPU

The [AMD Ryzen Threadripper 7000 Series and PRO 7000 WX-Series](https://www.amd.com/en/newsroom/press-releases/2023-10-19-amd-introduces-new-amd-ryzen-threadripper-7000-ser.html), built on the advanced "Zen 4" architecture, offer high core and thread counts. This design significantly alleviates CPU bottlenecks, especially in Python environments where the CPU often stalls due to the language's inefficiencies.

Threadripper's superior multi-core performance allows for more efficient instruction management, reducing the wait time for the GPU. This results in significantly better Python execution times and smoother operations across various computational scenarios.

# Bridging the Gap with Vast.ai

Many individuals lacking access to high-performance CPUs like Threadripper may experience bottlenecks in their GPU-accelerated tasks. The inefficiencies of a weaker CPU can significantly slow down processing, making certain computational tasks a slog.

[Vast.ai](https://vast.ai/) presents a viable solution to this problem. It provides a platform where users can rent high-performance GPU resources, which can be paired with superior CPU capabilities. By offering a gateway to better hardware resources, Vast.ai helps in overcoming the limitations imposed by less capable CPUs. Users can thus experience smoother, more efficient processing, enabling them to execute their tasks more effectively and in a timely manner. This way, Vast.ai plays a crucial role in alleviating the hardware constraints that many individuals face, bridging the gap between their current setup and the ideal computational environment.

# Final Thoughts

A robust CPU is crucial for maximizing GPU utilization, particularly in Python environments where a weaker CPU can cause bottlenecks. [The AMD Ryzen Threadripper](https://www.amd.com/en/processors/ryzen-threadripper-pro) significantly alleviates this issue with its high core and thread counts. For those without access to such high-performance CPUs, [platforms like Vast.ai](https://vast.ai/) offer the opportunity to rent superior GPU and CPU resources, providing a practical solution to hardware limitations. Exploring these options can lead to better performance in GPU-centric tasks and a smoother computing experience overall.


Introducing AMD’s Threadripper: A More Powerful CPU

Threadripper: Balancing CPU and GPU for Optimized Performance in Python


We're pleased to unveil the latest updates here at Vast.ai, geared toward achieving the best possible product experience for new and existing customers alike! Our aim is to provide you with a GPU rental platform that offers best-in-class service at a fraction of the price.

To that end, we work tirelessly to stay on top of the game when it comes to delivering improvements and enhancements. Bug fixes are always first on the agenda so we can ensure our customers enjoy a seamless experience at all times. Here are some of the latest fixes we've implemented:

- Refreshing now no longer deselects the selected template
- Fixed issue with setting internet pricing on host machine page
- Billing page now correctly displays no charges for offline instances
- GPU filters now working for all GPU types; added L40 and A6000ada
- Fixed incorrect invoice charges summation
- Scroll issues on instance page fixed
- Client earnings page now shows payouts for referrals

#### NVIDIA Cloud GPU Updates

Updating our product isn't just about bug fixes – we also extend our focus to implementing meaningful enhancements in a variety of ways. We want all of our customers to be able to make the most of their time on Vast! Below are some highlights of the latest changes:

- Documents and guides for Autoscaler (in beta) are now available – https://vast.ai/docs/autoscaler/introduction

- Contact us (on website chat or via email at contact@vast.ai) if you want to try out the Autoscaler for automating the provisioning of GPU workers for dynamic workloads.

- Added support for Oobabooga and Stable Diffusion for the Autoscaler beta
- Added warning to user for contract expiration
- Added example CLI autoscale script
- Added in PDF and CSV billing reports for clients with a date range picker
- Added a clean modal for hosts to manage machines in the GUI with a beta flag

We've also recently updated our Terms of Service, which can be read [here](https://vast.ai/terms). The full change log for our November 2023 product updates is included at the bottom of this page for your convenience – there are plenty of additional improvements we've made in recent weeks that you can check out below.

If you're new to Vast and looking for a cost-effective cloud GPU rental solution for any number of potential use cases (a few of which we outlined in a [recent blog post](https://vast.ai/article/GPU-Rental-Use-Cases)) we invite you to discover what our platform can do for you! Please reach out at any time via the chatbox at the bottom right or email us at [support@vast.ai](mailto:support@vast.ai) with any questions – we're happy to help!

#### Change Log

- Templates can now specify recommended disk size
- Added docs for Autoscaler (in beta) - https://vast.ai/docs/autoscaler/introduction

- Contact us (on website chat or email at contact@vast.ai) if you want to try out the Autoscaler for automating the provisioning of GPU workers for dynamic workloads.

- Added links to the txn for Crypto payments in billing
- Added confirmation on reboot
- Copy modal now has a pop-up with indication of success or failure
- For reserved instances, a pop-up now appears if there's not enough credit
- Updated our hosting landing page with info on how to become a datacenter partner ([https://vast.ai/hosting](https://vast.ai/hosting))
- Bugfix: issues with de-verification email to hosts
- Bugfix: issue with setting internet pricing on host machine page
- Bugfix: refreshing now no longer deselects the selected template
- Bugfix: payment information on billing page fixed
- Bugfix: fixes for templates with filters, such as GLX desktop
- Bugfix: SSH proxy port was wrong, now fixed
- Added in equivalent CLI command in each template
- /Host/setup is now the only place an account can be converted to a host account
- Added better code snippet text in docs
- Reworked the email notification system for when balance is zero
- Added warning to user for contract expiration
- Added example CLI autoscale script
- Bugfix: args for templates now filter machines properly
- Bugfix: autoscroll issues fixed
- Added in PDF and CSV billing reports for clients with a date range picker
- Added in multi-region support in the geo filter
- Added more clear arrows for the cloud copy UI to make it clear you can swap directions
- Added a warning pop-up for when you set your autobilling threshold above 0 (which could result in an immediate charge)
- Fixed the client referral earnings page to be more clear
- Instance deposits for reserved discounts now show instance ID in billing page
- Add or change invoices CLI/API endpoint to get charge history for a set of instances with contract-based filters (like instance tag for autoscaling groups)
- Bugfix: billing page now correctly displays no charges for offline instances
- Bugfix: fixed overflow/spacing issues on mobile browsers on the homepage
- Bugfix: GPU filters now working for all GPU types; added L40 and A6000ada
- Bugfix: fixed issue with payouts appearing on client billing pages
- Bugfix: invoices now use start date and end date properly
- Added support for Oobabooga and Stable Diffusion for the Autoscaler beta
- Added additional documentation for the Autoscaler
- Added in an easy switch link to the login & register modal
- Added feedback notifications and spinners on PDF exports for invoices
- Bugfix: fixed incorrect invoice charges summation
- Bugfix: fixed scroll issues on instance page
- Bugfix: date range is now properly set for hosts on the invoice/earnings page
- Bugfix: full invoice address now in the PDF invoices
- We have updated our Terms of Service ([https://vast.ai/terms](https://vast.ai/terms))
- Added a clean modal for hosts to manage machines in the GUI with a beta flag
- Added a South America filter
- Added a verification email re-sent button on account page
- Added additional documentation and guides for the Autoscaler beta
- Template name and image are now displayed when available in the UI
- You can now override the entrypoint for args launch instances using the option --onstart-cmd (or aliased as --entrypoint). For example: vastai create instance 7277200 --image pytorch/pytorch --disk 40 --onstart-cmd 'bash' --args -c 'echo hello; sleep infinity;'
- Bugfix: client earnings page now shows payouts for referrals
- Bugfix: frontend GUI now enforcing 8GB minimum disk size
- Bugfix: added additional detections and deletions for instances that failed to load


November product updates


Vast.ai provides a platform where developers can access a wide selection of GPUs from various hosts and data centers, offering competitive pricing to fit any budget. The cost and availability of GPUs are critical to the success and speed of generative AI projects. We’re excited to announce that [dstack](https://dstack.ai/) now includes integration with our [GPU marketplace](https://vast.ai/). dstack is an open-source cloud platform for training and deploying gen AI models. This integration enables our users to access an extensive selection of affordable GPUs, breaking down the barriers to entry for developers and ensuring that high costs do not stifle innovation.

# Diverse GPU Options at Your Fingertips

Vast.ai’s marketplace stands out by offering a wide selection of GPUs, ensuring that developers can select the best fit for their projects without cost constraints. The platform recommends GPU instances based on a comprehensive quality score but also gives you the flexibility to set a maximum price, ensuring that you stay within your budget.

# Empowering Developers with dstack and Vast.ai

With dstack 0.12.3, accessing Vast.ai's marketplace becomes a seamless part of the development workflow. Now, dstack’s intuitive CLI and API are equipped to tap into Vast.ai's resources, allowing you to train and deploy generative AI models with greater ease and less financial strain. This integration streamlines your development process, enabling you to utilize the cost benefits of Vast.ai without the hassle of complex cloud infrastructure management.

![](/uploads/dstack-top.png)

# Effortless Setup and Deployment

Using Vast.ai with dstack is easy and straightforward:

1. Log into your Vast.ai account and navigate to the 'Account' section to find your API Key.
2. With your API Key, configure the dstack backend according to the provided steps.

![](/uploads/dstack-bottom.png)

3. Once configured, begin using the robust features of dstack’s CLI and API.

This integration not only facilitates [fine-tuning](https://dstack.ai/docs/guides/fine-tuning/) and [deployment](https://dstack.ai/docs/guides/text-generation/) of models but also the creation of [development environments](https://dstack.ai/docs/guides/dev-environments/) that support interactive coding, making it a perfect fit for developers who prefer to work within their favorite IDEs.

# Seamless Switching Between Cloud Providers

The flexibility that dstack offers is unparalleled. You can now effortlessly switch to Vast.ai when it suits your project needs while retaining the option to use other cloud providers as required. This ensures that your workflow is as efficient and cost-effective as possible.

# Streamline Your AI Projects with dstack and Vast.ai

If you're looking for a simple and cost-effective way to train and deploy generative AI models, the combination of dstack and Vast.ai is ready for you. It's designed to save you time and money. With dstack’s new preview versions of the [fine-tuning](https://dstack.ai/docs/guides/fine-tuning/) and [text generation](https://dstack.ai/docs/guides/text-generation/) APIs, your development process just got even better.

Need a hand? Our [community](https://discord.gg/hSuEbSQ4X8) as well as dstack’s [community on Discord](https://discord.gg/u8SmfwPpMd) is available to offer support and answer your questions. We're committed to helping you make the most of what Vast.ai and dstack have to offer.


We’re excited to announce that dstack now includes integration with our GPU marketplace.

Accessing the GPU marketplace with Vast.ai and dstack


# Understanding the GPU Supply Crunch

Graphics Processing Units (GPUs) have evolved from their original use in video game graphics to become essential in fields like artificial intelligence, data processing, and advanced computing. Their ability to efficiently handle complex calculations has made them a key component in today's technology landscape. Despite their importance, there's a significant problem: a shortage of GPUs, known as the GPU supply crunch. This shortage is affecting many industries, leading to delays in projects and rising costs. In this article, we'll explore the reasons behind this shortage, which include problems in global supply chains, production challenges, and a sudden increase in demand for GPUs. We'll also look at its impact and how [services like Vast.ai](https://vast.ai/), a GPU rental platform, are providing solutions during this challenging time.

# What Caused The GPU Supply Crunch?

The cause of the GPU supply crunch can be attributed to several key factors. A major driver is the increased demand for GPUs, especially due to the rise of generative AI technologies in 2023. As AI becomes more integral to sectors like healthcare, automotive, finance, and entertainment, the competition for these powerful GPUs has intensified. Additionally, the COVID-19 pandemic caused significant disruptions in manufacturing and supply chains, leading to factory closures and production slowdowns. This situation was worsened by [shortages of crucial components like CoWoS packaging](https://asia.nikkei.com/Business/Tech/Semiconductors/TSMC-sees-AI-chip-output-constraints-lasting-1.5-years), which is essential for integrating memory and chip components.

Furthermore, geopolitical factors, such as the [U.S. restrictions on Nvidia's GPU shipments to China](https://www.trendforce.com/presscenter/news/20231019-11891.html), have impacted manufacturing capacities and market dynamics. In response, companies like Nvidia are adapting by increasing the availability of GPUs through cloud-based rental services, marking a shift in how these resources are distributed and used.

# Consequences of the Crunch

The GPU supply crunch has significantly affected various sectors, with the professional computing and AI research areas facing notable challenges. The growing demand for GPUs, particularly for AI applications like training models such as ChatGPT, has been remarkable. For instance, the [ChatGPT model was initially trained on 10,000 NVIDIA GPUs, but current demands have risen to about 25,000 NVIDIA GPUs](https://www.techgenyz.com/2023/02/13/nvidias-gpus-to-face-shortage-supply/). This increase has put a strain on GPU supplies, impacting not only AI research but also professional computing sectors. As generative AI continues to evolve, there's a possibility that companies like Nvidia might focus more on AI applications, potentially affecting GPU supply and pricing across other sectors.

The crunch has resulted in increased GPU prices and limited availability, affecting both consumers and businesses. They face challenges in affording and acquiring the needed hardware. Although the situation is slowly improving after the COVID-19 pandemic, the market is still not back to its pre-pandemic levels, and prices are higher than previously

# Industry Responses

The industry has been actively addressing the GPU supply crunch with several initiatives. Major players, such as Nvidia, are significantly increasing their production capacities. For example, [Nvidia plans to double its CoWoS packaging capacity by the end of 2024 with a $2.9 billion investment in Taiwan](https://www.irrgame.com/news/nvidia-gpu-shortage/), a step aimed at easing the GPU shortage and potentially lowering prices. Other companies like TSMC, Intel, and Samsung are also focusing on improving chip packaging, a key bottleneck in production. Intel, for instance, plans to quadruple its capacity by 2025. These measures demonstrate the industry's dedication to tackling the current production issues and meeting the growing demand, especially in AI and professional computing sectors.

# Government and Regulatory Responses 

On the government and regulatory side, there's been a notable effort to boost domestic chip manufacturing. The U.S. Commerce Secretary, Gina Raimondo, [has been vocal in urging Congress to act and provide funding for this initiative](https://www.industryweek.com/supply-chain/planning-forecasting/article/21243009/global-chip-shortage-likely-to-last-through-2023-us-official). Key legislative initiatives include the CHIPS Act and the America COMPETES Act, both proposing an investment of $52 billion in domestic chip research and manufacturing. While the final agreements on these bills are still pending, discussions among top chipmakers and government officials suggest that substantial relief from the shortage may not be seen until 2024. This highlights the necessity for global collaboration and long-term planning in the semiconductor industry.

# Vast.ai: Addressing the GPU Market Needs

During the current GPU supply crunch, [Vast.ai has become a key player in the GPU rental market](https://vast.ai/). This platform connects those who need GPUs for tasks such as AI training, data analysis, or graphics rendering with those who have spare computational power to rent. This model provides an affordable alternative to buying expensive hardware, particularly useful when prices are high and availability is low.

Vast.ai's system enables GPU owners to offer their processing power for rent, making high-end computing resources more accessible and affordable. This is particularly beneficial during the supply crunch, as it eases the burden on companies and individuals who are struggling to find available GPUs or who cannot afford their high costs.

The advantages of Vast.ai's rental model are evident in the current situation. It's a cost-effective option for those requiring GPUs for short-term projects or for those who can't meet the high initial costs of purchasing. The platform also offers flexibility, allowing users to adjust their computing resources according to their needs without the need for a long-term commitment or large investment. Vast.ai won’t just be useful during the supply crunch, though; it's a forward-thinking approach to managing the allocation and accessibility of resources in the increasingly demanding world of advanced computing.

# Wrapping Up: Navigating the GPU Supply Crunch

The GPU supply crunch presents a multifaceted challenge, but it's met with dynamic solutions and strategies. From the surge in demand for GPUs driven by the rise of generative AI to the disruptions caused by the COVID-19 pandemic and geopolitical factors, this shortage has far-reaching implications. However, the tech industry's response, highlighted by significant investments from companies like Nvidia, TSMC, and Intel, shows a strong commitment to overcoming these obstacles. Similarly, government initiatives like the CHIPS Act indicate a strategic approach to revitalizing domestic chip manufacturing.

[Vast.ai's emergence as a resourceful player in the GPU rental space](https://vast.ai/) illustrates an innovative response to the crunch, offering accessible and flexible computing power to those in need. This model not only addresses the immediate challenge but also sets a precedent for future resource distribution in the world of advanced computing.

As we move forward, these collective efforts from industry leaders, governments, and innovative platforms like Vast.ai will be instrumental in navigating the current crisis. They pave the way for a more resilient and efficient tech industry, capable of adapting to the growing demands and challenges of the future.


Despite their importance, there's a significant problem: a shortage of GPUs, known as the GPU supply crunch.

Quieting The Chip Crunch: Understanding the GPU Supply Shortage


# PyTorch vs TensorFlow: Which One Is Right For You? 

[PyTorch](https://pytorch.org/) and [TensorFlow](https://www.tensorflow.org/) are two of the most widely used deep learning libraries in the field of artificial intelligence. Both are supported on Vast.ai with easy to use templates. PyTorch, initially developed by Meta, offers an intuitive approach to building neural networks and is favored for its flexibility and ease of use in research. On the other hand, TensorFlow, developed by Google Brain, is designed for large-scale and complex machine learning models, with strong support for production environments. This article will compare PyTorch and TensorFlow, helping you understand which library may be more suitable for your project's needs and computational requirements.

# PyTorch Overview

PyTorch, initially developed by Meta, has quickly become a top choice for deep learning enthusiasts and professionals thanks to its dynamic computational graphing and deep integration with the Python language. It's specifically designed to facilitate rapid development and iteration of deep learning models, resonating with the workflow of many developers.

The true power of PyTorch comes when combined with NVIDIA's CUDA, a parallel computing platform that allows dramatic increases in computing performance by harnessing the power of GPUs. With CUDA, PyTorch can perform data-intensive computations at a fraction of the time it would take on traditional CPUs. This capability is very important when training complex models or dealing with massive datasets, a common task in AI research and development.

PyTorch also supports production needs with tools like TorchScript for model exporting and TorchServe for deployment. Its ecosystem is full of specialized libraries, fostering a collaborative environment where developers share tools and pre-built models, enhancing the PyTorch experience for both research and industry use.

# TensorFlow Overview

TensorFlow, developed by the Google Brain team, has also become one of the most widely used frameworks in the industry due to its scalable and flexible architecture. Its architecture centers around a static computational graph that simplifies and accelerates data processing. This feature, combined with its capacity for distributed computing, makes TensorFlow ideal for large-scale projects. In addition, transitioning models from the lab to the field is seamless with TensorFlow, thanks to tools like TensorFlow Serving for servers and TensorFlow Lite for mobile devices.

Furthermore, TensorFlow's integration with TensorFlow Extended (TFX) eases the creation of end-to-end machine learning workflows, making it more suitable for production use. The platform also includes comprehensive tools such as TensorBoard, which simplifies model analysis and debugging with its visualization capabilities.

The framework has fostered a massive community ranging from academic researchers to industry professionals. This diverse user base benefits from a large source of educational materials, add-ons from third parties, and forums for troubleshooting. Its widespread use in the corporate sector attests to TensorFlow's dependability and advanced capabilities. Ongoing input from this community contributes to TensorFlow's growth, keeping it at the forefront of AI application development.

# Comparing PyTorch and TensorFlow Metrics

### Performance Comparison

Both PyTorch and TensorFlow offer fast performance, but they do come with their own set of advantages and disadvantages. [A benchmark comparison revealed](https://viso.ai/deep-learning/pytorch-vs-tensorflow/) that PyTorch had a better performance compared to TensorFlow, particularly when offloading most of the computation to the cuDNN and cuBLAS libraries, which are essential components for GPU-accelerated computing​. However, performance can vary depending on the specific use case and hardware, with both frameworks capable of achieving high performance when properly optimized​​.

### Training Time and Memory Usage

[In a direct comparison](https://viso.ai/deep-learning/pytorch-vs-tensorflow/) utilizing CUDA, PyTorch outperforms TensorFlow in training speed, completing tasks in an average of 7.67 seconds against TensorFlow's 11.19 seconds. However, TensorFlow is more memory-efficient, using 1.7 GB of RAM during training compared to PyTorch’s 3.5 GB. Therefore, for quicker training, PyTorch is favorable, but for lower memory usage, TensorFlow is the better choice​.

### Ease of Use

PyTorch is often favored for its intuitive, Pythonic interface, making it easier for rapid prototyping, especially when utilizing CUDA for GPU acceleration. On the other hand, TensorFlow, although having a steeper learning curve, offers a more structured environment suitable for large-scale or commercial projects. Over time, TensorFlow has improved its usability, making it more approachable, but PyTorch is still considered more usable. In the end, PyTorch is simpler for research and development, while TensorFlow is structured for larger, production-oriented projects. However, regardless of which framework fits your project’s requirements, the computational requirements remain a substantial factor that can influence your choice.

# Addressing Computational Demands with Vast.ai

The computational intensity of working with PyTorch or TensorFlow can present challenges, especially when processing large datasets or training complex models. Both frameworks perform much better with high-performance GPUs, but they are not always readily accessible or economically feasible for every developer. [Vast.ai](https://vast.ai/) steps in to fill this gap by offering affordable and scalable GPU resources. With Vast.ai, developers and researchers can expedite training times and scale computing resources in tandem with their project demands, effectively leveraging the strengths of either framework without hardware limitations.

# Conclusion

In the end, your choice between PyTorch and TensorFlow should align with your project requirements: PyTorch for its user-friendly nature in research and development, and TensorFlow for its robustness in large-scale, production-level projects. Both frameworks have their merits and can be significantly enhanced by overcoming hardware limitations, a service that [Vast.ai](http://vast.ai/pricing) provides effectively. With the right tools and resources, you can ensure that your machine learning projects are not only innovative but also computationally feasible.


PyTorch vs TensorFlow: Which One Is Right For You? 

PyTorch vs TensorFlow: Which One Is Right For You


## GPU Rental: A Closer Look at Use Cases

Over the past few years, cloud GPU rental has emerged as a game-changer for innovators everywhere who want to power next-generation capabilities for their projects in various industries – without the hassle and big price tag associated with maintaining their own hardware.

The use cases for GPU rental are so diverse and wide-ranging that many people aren't aware of just how much of a difference it can make in both their personal and professional lives. With Vast.ai, anyone can access high-end computing power, on demand, while transforming their project's performance, scalability, and affordability.

Let's take a look at some specific use cases to explore the benefits and advantages of renting GPU resources on the cloud.

#### In-Depth Look: Podcast Transcription

A great example is **podcast transcription**. GPU rental through a platform like Vast.ai can power large language models that are able to perform transcription services with incredible speed and accuracy. Trained on massive datasets, these models can handle different accents, complex terminology, and industry-specific jargon, all while sifting through background noise.

[Whisper ASR Webservice](https://github.com/ahmetoner/whisper-asr-webservice/) is a simple to use transcription service that can harness the power of Vast GPUs. Simply select the template for Whisper by clicking this [link](https://cloud.vast.ai/?ref_id=43484&template_id=e795f6239ba0236393d61d892c3f4152), select a GPU and launch the service. It will then be accessible from your web browser and utilize the GPU on Vast for video/audio transcription to multiple languages.

Not only that, but GPU-accelerated transcription services can also provide near-instantaneous speech-to-text conversion for live broadcasts, enabling real-time closed captions or transcriptions for viewers.

Renting GPUs only when needed optimizes expenses and ensures you pay for computing resources only during active transcription tasks. And if you need to transcribe a high volume of podcasts at once (e.g. if you're working through a backlog or you're providing transcription as a service), GPU rental allows you to scale your operations as needed. You can flexibly increase or decrease GPU resources at any time.

Any podcaster, content creator, business owner, or even college student needing transcription assistance can take advantage of the speed, accuracy, real-time capability, cost-efficiency, and scalability that GPU rental offers, whenever and wherever they need it.

But there's so much more.

#### Other GPU Rental Use Cases

Renting cloud GPUs gives you an incredibly powerful workstation without the overhead and maintenance that normally go along with it. This frees you up to focus on accelerating diverse workloads, from the mundane to the most mission-critical.

Below are just a few of the possibilities.

**Translation services**: Similar to transcription, GPU-accelerated translation rapidly processes text and even speech via deep learning and neural networks, delivering speed and accuracy for high-quality multilingual communication. Real-time translation is also possible for applications like live broadcasts and video conferencing.

**Video rendering and editing**: Content creators can quickly render high-definition videos and easily edit multimedia projects using the immense processing power of a rented GPU server. The same is true for **3D rendering and animation**, speeding up a process that might otherwise take many hours.

**Writing and storytelling**: Seasoned authors and novice writers alike can make use of AI models to spark creativity in their own writing or to help craft novels, articles, text-based adventure games, and dynamic and immersive storytelling experiences. Right here on Vast.ai, we offer access to [KoboldAI](https://vast.ai/article/exploring-ai-based-writing) – an AI-powered writing assistant – via a ready-to-use template suitable for even the most non-tech-savvy user.

**Automated customer service**: From text-based chatbots and virtual assistants to voice-driven customer service through automated phone systems, GPU rental can provide the computational power needed for cutting-edge natural language processing. Images and videos attached to customer inquiries can also be analyzed and identified, improving the ability to provide relevant responses.

**Virtual desktop infrastructure**: Businesses that are embracing remote work can employ GPU rental to provide employees with high-performance virtual workstations, enabling dynamic collaboration from anywhere in the world.

**AI-generated artwork**: Whatever your artistic needs may be, deep learning AI tools like Stable Diffusion – based on latent diffusion processes – bring image generation to your fingertips with the press of a button. Frameworks like [ComfyUI](https://vast.ai/article/getting-started-with-comfy-UI), available as a template right here on Vast, allow for unprecedented flexibility and speed in AI-powered art, whether through text-to-image generation, image-to-image translation, or image interpolation.

#### The Bottom Line

The above-mentioned use cases are just the beginning. From **high-performance computing** and **AI model training** to **scientific simulations** and **advanced medical imaging**, the solution to a wide range of complex computing and data-intensive tasks is just a few clicks away! Renting the GPU resources you need, with built-in scalability, minimizes OpEx and maximizes ROI.

As the market leader in low-cost cloud GPU rental, we invite you to explore what Vast.ai can offer you and your organization today. We would be pleased to help you discover the powerful and affordable GPU rental solution you've been waiting for.


use cases to explore the benefits and advantages of renting GPU resources on the cloud

GPU Rental: A Closer Look at Use Cases


## Introduction to Oobabooga

Oobabooga is a tool designed for interacting with Large Language Models (LLMs), capable of understanding and generating text. With its user-friendly web interface, it's perfect for non-technical users interested in harnessing the power of LLMs. [Oobabooga supports a wide range of models](https://vast.ai/article/deploying-oobabooga-for-running-large-language-models), excelling in various use-cases like prompting, loading, and managing models.

### Why Run Oobabooga in API Mode vs Standard Mode?

Choosing API Mode for Oobabooga on Vast.ai offers several advantages, including enhanced scalability for multitasking, seamless tool integration via user-friendly endpoints, and streamlined workflow automation. Given these benefits, let's dive into how to actually run Oobabooga on Vast.ai.

### Why Choose Vast.ai for Oobabooga Deployment?

Running Oobabooga can be GPU-intensive, posing challenges for those without high-end GPUs or the financial resources to acquire them. That's where Vast.ai, a cloud-based GPU rental platform, comes in. By deploying Oobabooga on Vast.ai, you gain access to powerful, on-demand GPU resources and pay only for what you use, enabling complex text-generation tasks without hardware limitations.

### Launching Oobabooga on Vast.ai: A Step-by-Step Guide

#### Logging In to Vast.ai:

1. [Log into your Vast.ai account](https://vast.ai/).
2. Ensure you have enough credits for the task.

#### Selecting Oobabooga Template:

1. Click into the console.
2. Click ‘EDIT IMAGE & CONFIG.’

![](/uploads/Instance-Configuration.png)

3. Select the recommended template for Oobabooga.

![](/uploads/kO9lpdJwnxVSVjFnuprmaBQ3IJ6K2ALmORoSIyyWt3fG2gdPRIqrrFAVT8O6uCVqQwdTgPm6tRvIgbJFHmeaS8G3hIZxsKPBqC18fklDy62VIw8cf7OmhXiVkUcA.png)

4. Click on ‘EDIT’

![](/uploads/of-these-templates.png)

5. Scroll down to ‘On-start Script’ and type in ‘--api’ before the final semicolon

![](/uploads/vast.ai.png)

6. Click on ‘Select And Save’

![](/uploads/SELECT-AND-SAVE.png)

#### Allocating Disk Storage:

1. Adjust disk storage to a minimum of 80GB.

![](/uploads/EDIT-IMAGE-&-CONFIG.png)

#### Choosing a Suitable GPU:

1. [Review the Large Language Model (LLM) leaderboard on Hugging Face](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) to understand the GPU RAM requirements of your chosen model.
2. Filter for the total GPU RAM on the machine. Oobabooga can combine the GPU RAM on multi-GPU instances to load the model.

![](/uploads/GPU-Resources.png)

#### Launching Oobabooga Instance:

1. Click ‘Rent’.

![](/uploads/Auto-Sort.png)

2. Go to ‘Instances’

![](/uploads/cgNf2u4ui_EUkbfjH86kj1qltiHu7-CP32MKL-z5YTFjvc5fYPAPDnC06HXeuzG8R4YczJhTNVKkPB1LBqmxpt7g9nw_l9IpfYPFUaahDz3fYLMA3kQLw199j3Qt.png)

3. Click ‘Open’ on your newly created instance.

![](/uploads/entation.png)

## Final Thoughts

Oobabooga's user-friendly interface and versatile support for various LLMs make it an invaluable tool for text generation tasks. When considering deployment options, Vast.ai emerges as a cost-effective solution, providing access to powerful GPU resources on-demand. Following this guide, users can easily set up Oobabooga on Vast.ai, bypassing any hardware hurdles, and making using Oobabooga straightforward and simple.


Guide to deploying Oobabooga in API Mode on Vast.ai

Running Oobabooga in API Mode on Vast.ai


It's that time again! We're pleased to announce the latest round of updates at Vast.ai. Our goal is to continually identify ways to improve, in order to elevate our service to new heights and bring you a product experience that is second to none.

An important aspect of our ongoing commitment to excellence is to fix bugs as quickly as we can. This ensures that our customers enjoy the best possible performance at all times. Some of our latest bug fixes include the following:

- Google Drive verification is complete and now works for all users!
- Reliability score has been added back to host machine page
- Made fixes to host machine page for setting price and date
- Reserved instances cannot be reserved for longer than they exist
- Earnings dashboards no longer display internal transfers
- Instance prepay items now show correctly in billing tab
- Interruptible bid pricing now includes storage cost and keeps the minimum in the make bid modal

### NVIDIA Cloud GPU Updates

At Vast.ai, we are committed to enhancing the user experience. Whether you're a current or future customer, you can expect continual improvements as we make adjustments that boost usability and performance. Below are some of the latest changes to help you make the most of our product:

- Templates now have a permalink and a version link
- Added NCCL test to host installer
  - For any hosts with AMD EPYC systems, that hardware has compatibility issues/problems with the core NVIDIA NCCL library that distributed PyTorch (and other frameworks) use, which may make some machines unusable. You can read more and find workarounds if you Google (or join our Discord server and search the history for EPYC), but here are a few links:
    - [PyTorch Discussion](https://discuss.pytorch.org/t/torch-distributed-init-process-group-hangs-with-4-gpus-with-backend-nccl-but-not-gloo/149061?u=ptrblck)
    - [NVIDIA NCCL Troubleshooting](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs)
- Added filters for max CUDA and driver version
- Added a "rented" option in the CLI so you can filter offers with GPUs overlapping your current stopped instances on the same machine
- Added filter for total system GPU RAM
- Added display of args for args launch instances
- Improved docs for copying and moving data

For full details, please check out the change log for our October 2023 product updates below. We've implemented a host of additional improvements in recent weeks that we believe our valued customers will appreciate.

If you'd like to learn more about the services we offer or explore how we can help you with your GPU rental needs, you are welcome to contact us at any time! Feel free to reach out via the chatbox at the bottom right, email us at [support@vast.ai](mailto:support@vast.ai), or join our Discord server today to enjoy a helpful community and a library of resources available at all times.

## Change Log

- Reformatted the stop instance modal for clarity
- Templates now have a permalink and a version link
- Complete status msg now displayed in expanded instance display
- Bugfix: fixed scroll jump after stop instance button pushed
- Bugfix: WebGUI Docker login validation now matches Daemon
- Bugfix: earnings dashboards no longer display internal transfers
- Bugfix: reserved instances cannot be reserved for longer than they exist
- Bugfix: host dashboard client end date no longer displays anything when there are no active contracts
- Added NCCL test to host installer
  - For any hosts with AMD EPYC systems, that hardware has compatibility issues/problems with the core NVIDIA NCCL library that distributed PyTorch (and other frameworks) use, which may make some machines unusable. You can read more and find workarounds if you Google (or join our Discord server and search the history for EPYC), but here are a few links:
    - [PyTorch Discussion](https://discuss.pytorch.org/t/torch-distributed-init-process-group-hangs-with-4-gpus-with-backend-nccl-but-not-gloo/149061?u=ptrblck)
    - [NVIDIA NCCL Troubleshooting](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs)
- Added filters for max CUDA and driver version
- Autobilling threshold now sets the low balance email threshold
- Updated email warning for all billing attempts
- All accounts are client accounts unless changed in the hosting directions
- Added a "rented" option in the CLI so you can filter offers with GPUs overlapping your current stopped instances on the same machine
- Bugfix: remove the Docker login/password after instance is created
- Bugfix: Coinbase payment improvements
- Bugfix: web UI direct port filter fixed
- Bugfix: added back reliability score to host machine page
- Bugfix: Jupyter button appears if JUPYTER_TOKEN or JUPYTER_PORT is defined
- Bugfix: client end date now handles no/null end date contracts
- Added filter for total system GPU RAM
- Added react code blocks in documentation
- SSH/Jupyter launch mode does not show entrypoint args
- Added feedback for password reset
- Added display of args for args launch instances
- Improved docs for copying and moving data
- Kaalia – upgraded disk IO testing to use Fio
- Bugfix: host machine page fixes for setting price and date
- Bugfix: interruptible bid pricing now includes storage cost and keeps the minimum in the make bid modal
- Bugfix: image CUDA version detection now properly detects the right CUDA version
- Bugfix: logging in refreshes templates
- Bugfix: Oobabooga template now opens the webUI upon clicking open
- Bugfix: instance prepay items now show correctly in billing tab
- Bugfix: email notifications for hosts fixes
- Bugfix: Firefox issue no longer blocking setting name in invoice
- Bugfix: pending crypto transactions no longer appear on pdf billing statements
- Google Drive verification is complete and now works for all users!


Discover Vast.ai's October updates: enhanced GPU features, vital bug fixes, and platform improvements for an optimal user experience.

October Product Update


Writing a book is a daunting task.

Coming up with just the right words, and keeping track of the ideas that you’re weaving together - this is difficult work. The difficulty can even make it seem that there might be better ways to be spending your time.

A new generation of tools seeks to solve this challenge though - to allow you to bring your powerful intuition and vision to life, while leaving the management of the details to the machines.

This may sound too good to be true - but it’s not. Try the tools out for yourself and see. We’ll help you get started.

Setting some context is important, though - so we’ll go quickly over some of the basics, and then show you step-by-step how to get started.

KoboldAI is a tool that provides functionality specifically related to writing. KoboldAI in turn can leverage any number of models while producing its assistance. The one that we will use here is called Pygmalion.

And finally the magic dust that brings all of this technical wizardry together is Vast.ai. We will install the KoboldAI tooling with the Pygmalion model on the powerful Vast.ai GPUs - which we can rent for pennies-per-hour. This not only gives us a cost-effective way to put these tools to use - it also allows us to only pay for what we need.

## Step 1: Create a Vast.ai Account

If you don't already have a Vast.ai account, you'll need to create one. Visit [Vast.ai](Vast.ai), sign up, and follow the on-screen instructions to create your account.

## Step 2: Add Credits

Before renting a virtual machine on Vast.ai, you'll need to add credits to your account. This is how you pay for the GPU resources you use. Simply click on your account balance and follow the prompts to add funds securely. Vast accepts credit card payments through Stripe and crypto through Crypto.com and Coinbase.

## Step 3: Running the Template

1. Browse Available Templates: Within the Vast.ai platform, take advantage of their pre-configured templates, which include the Pygmalion model. Visit the [Vast.ai Templates](https://cloud.vast.ai/templates/) section to explore these ready-to-use configurations.

2. Select the [KoboldAI + Pygmalion Template](https://cloud.vast.ai/?ref_id=43484&template_id=0930ce52c987ce85e981371fadacd7b1): Click that link, or locate and select the Pygmalion template from the list of available templates. These templates come pre-packaged with the necessary settings, eliminating the need for manual setup.
   [![ScreenShot](/uploads/1DCF197C-F94C-4EE0-9A60-DBD3ED8AB42B.jpeg)](/uploads/1DCF197C-F94C-4EE0-9A60-DBD3ED8AB42B.jpeg)

3. Customize as Needed: While the templates streamline the process, you still have the flexibility to customize certain parameters to suit your specific requirements, such as GPU type or resource allocation. Select your preferred storage size to find a machine for your needs.

4. Rent the Instance: Clicking the “Rent” button will start the instance.
   [![ScreenShot2](/uploads/B87F9218-AEA0-450C-BEFF-D5468A6C4212_4_5005_c.jpeg)](/uploads/B87F9218-AEA0-450C-BEFF-D5468A6C4212_4_5005_c.jpeg)

## Step 4: Write Your Next Great Novel!

Once your instance has started (this will take a few minutes), the final entries in the logs will show the URLs that can be used to access the system. They are labeled according to the type of the interface. Take one of these and plug it into your browser, and you can begin your work.

You will notice that one of these URLs is for use as an API - so if you are looking to programmatically include the capabilities of this tool into your own application, this is how you would do it.

### The Sky Is The Limit

This is just a small taste of the capability that might be provided by this and by similar tools. The flexible compute power offered by Vast.ai offers a cost-effective and straightforward way to get started - either with experimenting with workflows or with putting them to real use.


Exploring AI-Based Writing


Imagine you are working on a data-intensive machine learning project and require a Linux environment optimized for deep learning frameworks. How can you easily and efficiently obtain one and still enjoy endless customizability? We recommend using templates to create Linux Docker instances with predefined settings.

This is where Vast.ai comes in. We provide containerized Linux Docker instances for rent along with a wide range of [templates](https://cloud.vast.ai/templates/) to suit numerous use cases for our customers.

## Templates – the Building Blocks of Customization

Think of a template as a blueprint for your instance. Each template is a saved set of initialization information that specifies what Docker image to load and includes settings that can only be configured when the instance is created.

So in the example project above, where you would like to make use of a deep learning framework, you could select a PyTorch or TensorFlow template on Vast. Then, with just a few clicks, you can create a new instance with your preferred framework, preset environment variables, and networking configurations ready to go.

With Vast templates, you have a multitude of these powerful tools in an easy-to-use format at your fingertips – from AI-powered image generation with [ComfyUI](https://vast.ai/article/getting-started-with-comfy-UI) to running large language models (LLM) via the pre-built [Oobabooga](https://vast.ai/article/deploying-oobabooga-for-running-large-language-models) template, and much more.

## How to Use Templates on Vast.ai

To get started, access the primary instance configuration menu by clicking on the [Templates link](https://cloud.vast.ai/templates/) in the navigation panel on the Vast console.

There are three different template categories for you to choose from:

- **Recommended** – templates that will load a smart set of options for typical use cases.

- **Popular** – a list of the current most popular templates used on Vast.

- **Recent** – where your customized and selected templates are saved.

To use any template as is, simply click “Select” to pick your desired template. After that, you will be directed to the search interface where you can find a suitable GPU offer and move forward from there!

If you prefer to customize a template for your own specific needs, click “Edit” to bring up the template editor. Editing any template creates a new version of that template, which will then appear in your Recent section after being saved.

You can also customize your template within the search interface via a separate configuration menu, although the settings are the same either way. For more information about template customization and selecting launch mode and disk size, see our detailed template guide [here](https://vast.ai/docs/instance-setup/templates).

## Making the Most of Your Templates

Vast also offers a [referral reward program](https://vast.ai/docs/account/referral-program). You can use a simple referral link, found on your account page, or drive traffic to a specific public template that you have created. Here’s how template referral works:

1. Make your own Vast template configured for the use case you want to promote and set the template to public.

2. Refer people to your template link (found next to the name of the template), which will autoload the Docker image and predefined settings for your use case, making it much easier for new users to get started on Vast.

3. Earn 3% of the credits purchased by anyone who registers a new client account via your public template referral link – and continue to earn 3% for the lifetime of the account!

4. Keep your referral credits as a positive balance in your account for your own use on Vast, or convert up to 75% of referred credits into cash that can be withdrawn via Stripe Connect, PayPal, or Wipe.

For the nitty-gritty details about our referral program, check out the full guide [here](https://vast.ai/docs/account/referral-program).

## Conclusion

With the help of our easy-to-use templates (as well as our referral program, should you choose to make use of it), we hope you will discover an even more accessible and cost-effective way to leverage the power of GPU rentals. At Vast.ai, we’re proud to be able to bring you a world of resources while minimizing the barriers of cost and complexity.

If you have any questions or would like further assistance, please feel free to contact us via the chat box below, or drop us a line at [support@vast.ai](mailto:support@vast.ai). We look forward to hearing from you.


Unlocking the Power of templates - Linux Docker Instances


When it comes to Falcon 180B, the epic new LLM set loose on the world by the folks at The Technology Innovation Institute (TII), you can’t be shy with the hardware!

This model is a beast - it is trained on more than 3 trillion tokens - mostly coming from TII’s “RefinedWeb” dataset.

The “180B” in the name hints at the incredible size and power of this model. While the B doesn’t stand for beast - it does reflect the 180 billion neural network parameters it uses.

That’s a lot. And it takes serious GPU RAM and computing power to operate.

This is where Vast.ai comes in. With Vast.ai you can rent the GPU power you need to run immense models like Falcon 180B.

So to get you up and running we’ll walk step by step through just how to get Falcon 180B running with Vast.ai. We will show you how to do it with the Web UI as well as directly with SSH.

In both cases - you’ll want to purchase some credits prior to starting, so that you will be able to run your instances.

## Use the Web UI to run the model!

### Step 1: Select the template you would like to use.

Click [templates](https://cloud.vast.ai/templates/) on the menu on the left hand side of the screen (expanding it with the “hamburger” icon on the top left if necessary).

From the template list, find the “Oobabooga” template and select it. This will run a simple webui for downloading, loading and chatting with the Falcon 180B model.

![](</uploads/OobaboogaLLMWebUI(LLaMA2).png>)

### Step 2: Select enough storage

Move the storage slider over to allocate storage for the instance. Storage can only be set before the instance is created, so this is an important step.

Falcon 180B requires differing amounts of storage depending on the quantization method you select.

We recommend at least 100GB for the model we recommend.

### Step 3: Find an appropriate machine - and click the Rent button to start it.

Falcon 180B requires a lot of GPU VRAM. With no quantization, the LLM requires ~640GB of GPU VRAM!

With quantization, the amount of GPU VRAM is reduced. However, the model quality degrades when more quantization is used. It is a balance. We recommend the Bloke’s quantized models - https://huggingface.co/TheBloke/Falcon-180B-Chat-GPTQ#provided-files-and-gptq-parameters

The linked chart shows the tradeoff between more quantization and quality. At the high end, the server will need 94.25 GB of GPU VRAM. On the low end, 70.54 GB.We recommend the main branch which uses 4 bit quantization and requires 95 GB of GPU VRAM along with 100 GB of disk space.

This will fit in a variety of configurations. You could use a 2X A6000/A40 or a 4X 3090/4090/A5000. The total GPU RAM needs to be over 95GB.

![](/uploads/C5FUug48n7mMT1m_aZPFEQalyeux2a8C-rthdQDdmuDFAVwApD-Q510pGghYczjegkYkMqOiVbmylzJ39FVt1Ve-TyW1L7qd56S3cBEof0eU3fbZI-PQAYTxp9GK.jpg)

Click the blue rent button to create and launch your instance.

### Step 3: Open the Web UI & download the model.

On the instance screen, once your instance has started, there will be a button labeled “Open” - this will take you to the Web UI.

Once the Web UI has loaded, on the top of the screen there will be a menu. Select “Model” from that menu.

On the right hand side of the screen there will then be a text box for downloading the model.

Enter the following: TheBloke/Falcon-180B-Chat-GPTQ

Press the Download button. This will take some time.
![](/uploads/HrKSLo7ao3bEGM1KW_IXvUQGnK9dNkDZGGdlcJnc4NRvzpVj7G64eJKKA2GSgEaaNWbtknyamaA9_10sTS0bo6G-qwvG53hEe6LT5Y-Mp4aBT7cdYZCcqh3wZYAR.png)

### Step 4: Load the model.

Slide all of the GPU memory sliders all the way to the right, so that you’re allocating all available memory to the model.

Once the download is complete the model will be listed in the drop down to the upper left. Select it and click load.

### Step 5: Use the model.

Click chat on the top menu. This will take you to a chat interface - you can now interact with the model in the familiar way.

## Use SSH to run the model!

An alternative way to run the model is to use SSH.

### Step 1: Set up your SSH access.

Ultimately this boils down to two things:

- Generating your key pair.
- Putting your public key on your account.

Some of this is platform specific - but detailed assistance can be found here: https://vast.ai/docs/instance-setup/ssh

### Step 2: Select the Oobabooga template.

Follow the same steps that as in Step 1 from the Web UI instructions

### Step 3: Find an appropriate machine - and click the Rent button to start it.

Follow steps 2 & 3 above to select an appropriate GPU server and match the total GPU VRAM with the size of the quantized model you want to run. Approximately 100GB of total VRAM should work.

### Step 4: SSH to the instance.

Once the instance has started - click the “connect to” button. This will bring up a screen that has SSH details - copy those to your terminal and execute them.

### Step 5: Install packages & Execute example Python code.

pip3 install transformers>=4.33.0 optimum>=1.12.0
pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/ # Use cu117 if on CUDA 11.7

At the command line for your instance (which you’ve SSH’d to) execute the following:

```
Python

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_name_or_path = "TheBloke/Falcon-180B-Chat-GPTQ"

model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                             device_map="auto",
                                             revision="gptq-3bit--1g-actorder_True")

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)

prompt = "Tell me about dogs"
prompt_template=f'''User: {prompt}
Assistant: '''

print("\n\n*** Generate:")

input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, do_sample=True, temperature=0.7, max_new_tokens=512)
print(tokenizer.decode(output[0]))
```

This sample code is from “TheBloke’s” Hugging Face card, at https://huggingface.co/TheBloke/Falcon-180B-Chat-GPTQ. It shows the system operating - though I’m sure you have many ideas on more interesting things to do with it, once you get it running.

Wrapping Up

So there you have it - quickly, painlessly spinning up the powerful Vast.ai platform and executing a cutting edge large language model. So whatever your AI use-case is, you can now be confident that you can make it a reality with minimal effort.


Running Falcon 180B on Vast


Here at Vast, we offer both **on-demand** and **interruptible** rental options. For most users, the on-demand option provides stable access to the GPU rented for the duration of the rental period. However for advanced users who want to save money and can have their job interrupted, interruptibles are an interesting option.

## On-Demand

**On-demand instances** have a fixed price set by the host. The advantage of renting on demand is that you have high priority and gain exclusive control over the GPU(s) for the lifetime of the instance. This duration of time is up to the client, but a maximum duration determined by the host appears on the offer card when searching the marketplace.

Once the designated time limit expires, the instance is automatically stopped. You can then copy data off of the stopped instance to a new instance or to your local machine. It is best to assume that your instance will be lost once its lifetime expires, but the host does have the option to renew the contract, adding more lifetime to your instance. Reasons not to do so may include taking the machine offline for scheduled maintenance or upgrades.

If you need an instance to run for many days or weeks at a time, such as for larger artificial intelligence or machine learning tasks, ensure that the maximum duration exceeds the estimated job completion time. Another best practice is to select a host with a high reliability score.

## Interruptible

Our **interruptible instances** use a direct bidding system to determine priority on a continual basis. This sets Vast.ai apart from others in the field. For instance, although AWS spot instances and GCE interruptible instances can similarly be interrupted by on-demand instances, they do not use a direct bidding system and some will terminate interruptible instances after 24 hours.

With Vast.ai, clients set a bid price for their instance, and the current highest bid determines the instance that runs; any others are paused. Higher bids are obviously favored to maximize revenue for hosts, but this system can nonetheless reduce clients’ costs by fifty percent or even more in certain use cases.

For any given interruptible instance, a higher bid means higher priority on the machine. Lower-priority instances are paused until their bid is raised enough to regain the highest priority or until a higher bid finishes up and is no longer running. On-demand instances will always take precedence over interruptible instances, as well.

## Important Considerations

As you get started with your GPU rental, the following are a few crucial points to keep in mind:

- The bidding method cannot be changed after an instance is rented.

- An interruptible instance can be abruptly paused at any time if another user places a higher bid or if an on-demand rental is created for the same resources. This stops all processes that are running, so it is recommended to save your work to disk and consider saving your outputs to cloud storage at regular intervals as well. You can still transfer data off a stopped instance.

- It’s best to limit the use of interruptible instances to fault-tolerant workloads capable of handling pauses in runtime. Some examples include batch jobs, code builds, load tests, background processing, data analysis, and optional tasks.

Lastly, we do offer another rental type where you can deposit credits to get [reserved instance discounts](/docs/instance-rental/prepaid-plans) anytime you like. This involves converting an on-demand rental into a prepaid reserved rental at a discount, ideal for users who have predictable longer-term computing needs and want to get the most out of their budget.

For more information about the above rental options and answers to any further questions you may have, we welcome you to reach out anytime via the chat box below or by emailing us at [support@vast.ai](mailto:support@vast.ai). We look forward to helping you find the GPU rental solution that best fits your unique needs!


Here at Vast, we offer both on-demand and interruptible rental options

On Demand vs Interruptible Rental Types


# Vast.ai Product Update | September 2023

At Vast.ai, we understand how crucial it is to provide our customers with the top-notch performance that they’ve come to expect from our platform. To that end, we are constantly looking for ways to improve and bring you a service that continues to innovate.

When bugs pop up on occasion, we work quickly to address and rectify them, ensuring seamless performance moving forward. The following are some of the latest bug fixes we’ve made since our [previous product update](https://vast.ai/article/august-product-update):

- Hosts can no longer set a price below 0
- Template editor now updates correctly
- Broken template link has been fixed
- Issue with Backblaze data sync is fixed
- Backblaze copy now has an “operation complete” message

## NVIDIA Cloud GPU Updates

In our product updates, we also aim to deliver enhancements that make the user experience as smooth and uncomplicated as possible. Below are some of the latest adjustments we’ve made to boost usability and performance:

- Replaced the modal errors on the search page with snackbars (messages that temporarily show on the screen)
- Added a snackbar that says “copied” when someone copies a shareable link
- The instance extended view now shows actual status and intended status instead of current state and next state
- Template editor version tag will now auto-populate the docker image version
- Machine page client end date is no longer an editable box

You can check out the full change log for our September 2023 product updates below. It is our hope that these latest changes will continue to advance our goal of making Vast.ai as user-friendly as possible for our current and future customers.

If you’re interested in learning more about how we can assist you with your GPU computing needs, or if you need some help with deployment on our cost-effective infrastructure, please feel free to contact us anytime. There is a chatbox at the bottom right for your convenience, and you can also reach us via email at [support@vast.ai](mailto:support@vast.ai), or join our Discord server for instant access to a wealth of information and resources.

## Change Log

- Replaced the modal errors on search page with snackbars
- Added a snackbar that says "copied" when someone copies a shareable link
- Report machine icon is now a wrench and the warning (!) icon is used for scheduled maintenance
- The instance extended view shows intended status and actual status vs. current state and next state
- Template editor version tag will auto-populate the docker image version
- Machine page client end date is now not an editable box
- Bugfix: hosts now cannot set a price below 0
- Bugfix: Backblaze copy now has an "operation complete" message
- Bugfix: template editor now updates correctly
- Bugfix: issue with Backblaze data sync fixed
- Bugfix: search page inline image editor clicks/button pushes only do 1 clear action
- Bugfix: broken template link is fixed

- CLI driver_version field now supports >, >= queries
- Reserve credit balance calculation now based on GPU+storage
- New cluster rental page for cluster orders - https://vast.ai/clusters
- Added more visible warning that Coinbase does not support refunds
- Problem button stays orange after problem is reported on that machine
- Selecting template with no account active now opens login flow
- Cloud copy dialogue closes once operation starts
- Cloud copy remembers last arguments
- Earning dashboard shows payment service type
- GPU price for a stopped instance now displays in gray strikethrough when stopped
- Template editor renamed "Dockerhub location" to "Docker Options"
- Bugfix: fixed GPU flops display
- Bugfix: expired contracts now charging correctly
- Bugfix: template page template links fixed
- Bugfix: cloud sync now reporting errors correctly
- Bugfix: "add card" message in upper right now displays correctly
- Bugfix: bandwidth pricing for hosts now displays correctly in machine interface
- Bugfix: client end date now ignores host jobs
- Bugfix: docker tag selector fixed for the inline template editor
- Bugfix: env variables not highlighted red
- Bugfix: template readme editor working again
- Bugfix: open button logic fixed


September Product Update


# Getting Started with ComfyUI for Stable Diffusion: A Beginner's Handbook to Running on Vast.ai

Artwork created using artificial intelligence (AI) has made a big splash in recent years. Simple text descriptions using natural language can generate a vast array of images in different styles. From watercolor, oil, and acrylic-style digital paintings to cartoons and photorealism, fashion photography, and nature scenes, the possibilities are limited only by one’s imagination.

But how do you go from text to image? Deep learning AI tools like Stable Diffusion – based on latent diffusion processes – make it possible to create complex images with just a few words typed into a text box. Frameworks like ComfyUI make that tool accessible to just about anyone.

Released in January 2023, [ComfyUI](https://www.futuretools.io/tools/comfyui) is an open-source modular Stable Diffusion workflow engine with a node-based interface. It can create and execute advanced Stable Diffusion pipelines for use cases like text-to-image generation, image-to-image translation, and image interpolation – aka inpainting and outpainting, or filling in / extending the missing areas of an image. But ComfyUI goes even further than other AI art interfaces, providing cutting-edge configurability.

With ComfyUI, the user builds a specific workflow of their entire process. This approach is more technically challenging but also allows for unprecedented flexibility. For instance, you can preview images at any point in the generation process, or compare sampling methods by running multiple generations simultaneously.

The concept of node-based interfaces is most commonly associated with industries like 3D design and visual effects (VFX). Each node typically carries out specific code-based functions.

In the case of ComfyUI, every node is a representation of distinct facets within the Stable Diffusion process. Through the creation and interconnection of these nodes you gain the ability to seamlessly run Stable Diffusion experiments.

Speed is a major advantage, as well. Due to the granularity of its node-based interface, ComfyUI is able to re-execute only the components that have changed between executions, optimizing performance and speed. Many users have reported that ComfyUI image generation is two to three times faster than other interfaces.

Some other useful features of ComfyUI include the following:

- Workflows can be saved as JSON files, allowing you to easily reload an entire layout of nodes for consistent and reproducible results across multiple images – and share and revisit projects later.
- The interface supports loading checkpoints and safetensors models, as well as different upscaling models.
- Full workflows are saved in image metadata and can be loaded from the resulting PNG files.

With these innovations and more, ComfyUI ushers in a flexible, infinitely configurable approach to AI-powered image generation. The easy-to-use interface – with no programming knowledge required – means even beginners can quickly achieve great results. Here’s how you can get started with Vast.ai.

## Running ComfyUI on Vast

You can get ComfyUI up and running in just a few clicks.

1. Head to our [Templates page](https://cloud.vast.ai/templates/) and select ComfyUI. Yep, it’s that simple. (Already signed in? Click [here](https://cloud.vast.ai/?ref_id=43484&template_id=b349721c098bec541f600cd96e50b00f) for our ComfyUI template directly.)
   _Note: A template contains a Linux docker image, related settings and launch mode(s) for connecting to the machine. Whenever you edit a template, a new version is created and stored in your recent folder._

![](/uploads/comfyBlogScreenshot.png)

2. Filter and select the machine (GPU) for your project.
3. Select enough storage for your project.
   _Note: You can't change this after instance creation, so make sure to size correctly at start! Use the slider and filters on the left-hand side of the console to find the best fit for you._
4. Start your Instance by clicking the blue RENT button. The instance will appear in the Instances tab. After loading, your instance will start up, and you can click the Open button to launch ComfyUI.

Now you’re running ComfyUI! **Remember:** destroy your instance once you’re done to stop your GPU and avoid incurring storage charges.

It’s that easy!

For more Vast.ai templates, check out our [console](https://cloud.vast.ai/templates/).
For in-depth guides on all things Vast.ai, [explore our docs](https://vast.ai/docs/overview/introduction).

For curious minds, here’s how you install ComfyUI on your own machine. Fair warning, it’s a bit more complicated than running directly on Vast.ai.

### Installing ComfyUI Locally

For **Windows** users, the easiest method to install ComfyUI is by using the standalone installer available on the releases page:

1. Visit the official releases page: [ComfyUI Releases](https://github.com/comfyanonymous/ComfyUI/releases).
2. Download the latest comfy-windows.zip file from the list of releases.
3. Extract the downloaded zip file using a tool like 7-Zip or WinRAR.
4. After extraction, you'll have a portable ComfyUI folder that's fully configured.
5. Place your Stable Diffusion models in the "models/checkpoints" directory within the ComfyUI folder.
6. To launch ComfyUI, simply run "comfyui.exe" from the extracted folder.

## Installing ComfyUI on Linux

You can easily install ComfyUI on various Linux distributions, including Ubuntu, Debian, and Arch, by following these requirements and steps:

**Requirements:**

- NVIDIA GPU with CUDA support
- Python 3.7 or higher
- Pip package manager

**Installation Steps:**

- Clone the ComfyUI repository :://github.com/comfyanonymous/ComfyUI.git
- Install PyTorch with CUDA support
- Install dependencies
- Download Stable Diffusion models and place them in models/checkpoints
- Download a VAE like Latent Diffusion and place it in models/vae
- Launch ComfyUI


Getting Started with ComfyUI for Stable Diffusion: A Beginner's Handbook to Running on Vast.ai.

Getting Started With ComfyUI


As part of our commitment to outstanding customer service at Vast.ai, we are continually working to provide our users with the best possible product experience. An important aspect of that is making adjustments to functionality as soon as we become aware of opportunities for improvement. This means that Vast.ai customers get to enjoy a service that only grows better with time.

The occasional bug here and there is inevitable, so our first priority is to address any issues that have popped up since our last product update. Some of our latest fixes include the following:

- Delete instance no longer auto-scrolls to the top
- Stripe now works with SEPTA cards
- Earnings page correctly displays transfers
- New instance creation is prevented on occupied GPU
- Reserved instances now show as reserved correctly
- Interruptible instances display proper status
- Host billing charts show correct amounts
- Cloud sync rclone updating to show 100% complete upon success

## NVIDIA Cloud GPU Updates

Another goal we strive toward in our product updates is delivering meaningful improvements to the platform itself. Below are some highlights of the latest capabilities that have been added to improve flexibility and overall performance:

- New section for referral program to docs
- Cancel button added for rclone
- Template share link icon added to templates and confirmation dialog pop-up
- Updated [docs](https://vast.ai/docs/) with API info, search, and better navigation
- Added button to report a machine problem
- Improved ranking system for popular templates
- Refactored the host dashboard and search page, and restyled billing page
- Added filter for Internet bandwidth pricing

The full change log for our August 2023 product updates is available below for your perusal. Please check it out to see what new capabilities are possible, to help you make the most of Vast.ai!

For more information about the many features and services we offer, you can talk with us right here on our site via the chatbox at the bottom right. You can also join our Discord server to learn more and stay informed of developments, or drop us a line at [support@vast.ai](mailto:support@vast.ai).

## Change Log

- Added section for referral program to docs
- Added a CLI and GUI filter for machines with static IP
- Added cancel button for rclone
- Added H100 SMX and PCIE into the GPU filter
- Added RTX 3090 Ti into the GPU filter
- Added template share link icon to templates and confirmation dialog pop-up

- Bugfix: delete instance no longer auto-scrolls to top
- Bugfix: Stripe now works with SEPTA cards
- Bugfix: in the host install script, Ubuntu 22 NVIDIA-docker bug detection now works
- Bugfix: fixed issues with multi-GPU contract ordering
- Bugfix: earnings page displays transfers correctly
- Bugfix: open TCP/UDP ports slider now works with GPU filter
- Bugfix: fixed issue with Fios test speeds
- Bugfix: fixed issue with pricing hover
- Bugfix: fixed min CUDA filter
- Bugfix: fixed OPEN_BUTTON_PORT env

- Updated [docs](https://vast.ai/docs/) with API info, search, and better navigation
- Added button to report a machine problem
- Added better ranking system for popular templates
- Added a smart retry option for the CLI to handle rate limits
- Added warning and confirmation for instance copy from / to /
- Refactored the host dashboard
- Restyled billing page
- Added hover text to SSH info
- Refactored search page
- Added template edit button in search interface
- Minor update to privacy policy for Google Drive integration

- Bugfix: prevent new instance creation on occupied GPU
- Bugfix: force re-approve in Dropbox

- Added filter for Internet bandwidth pricing
- Copy buttons now prevent users from copying / to /
- Added a template link for referrals that always loads the latest template version
- CLI show instances command returns results in human-readable format
- Replaced warning icon with wrench to report instance issues
- Added option to sort by VRAM or VRAM/$
- Added a billing history CSV export
- Added confirmation messages after API-key, SSH key, and email account changes

- Bugfix: reserved instances now show as reserved correctly
- Bugfix: show proper status for interruptible instance
- Bugfix: host billing charts now show correct amounts
- Bugfix: cloud sync rclone updating now to show 100% complete upon success
- Bugfix: open button connecting→open now test correct ports
- Bugfix: fixed issues with host interface setting end date


As part of our commitment to outstanding customer service at Vast.ai, we are continually working to provide our users with the best possible product experience.

Vast Product Update


# Apple GPT: What We Know So Far

Apple has been secretly developing a generative AI tool called "Apple GPT." The tool's foundation is the Ajax framework, which Apple developed last year to streamline its machine learning development.

In simple terms, Apple GPT is a chatbot that can generate text, translate languages, create content, and answer your inquiries. Apple has not yet provided all the information on its intriguing project, but based on what is known so far, it will compete with even the most powerful AI chatbots, like ChatGPT, and might be made available as early as 2024.

## What Should We Expect From Apple GPT?

- According to Mark Gurman of Bloomberg, Apple uses AI technology for internal chatbot access for Apple workers and product prototyping.
- There is no easy way to incorporate the Apple GPT output into customer-facing new product features. Apple GPT's "Ajax" framework, created especially for huge language models like ChatGPT, Microsoft Bing, and Google Bard, is its beating heart. Ajax is based on Google's Jax machine learning framework, and it runs on Google Cloud.
- Historically, Apple has always taken a conservative, privacy-over-function stance when developing artificial intelligence and has frequently come under fire for having inferior skills to rivals.
- Apple CEO Tim Cook stressed the need for a careful and systematic approach to the development of AI while acknowledging that there are still problems in the field.

## Potential Features of Apple GPT

- Apple GPT might be used to enhance the precision and fluidity of Siri as well as other Apple services that rely on NLP (Natural Language Processing).
- Apple may employ Apple GPT to develop new goods and services, such as a chatbot to assist you at work or a writing helper to support your creative endeavors.
- Apple GPT might provide individualized replies to your queries or offer content to improve the user experience on its devices.

## How Effective Will Apple GPT Be?

It isn't easy to compare Apple GPT's performance to its rivals because it is currently under development and not yet offered to the marketplace.

According to a Bloomberg report, the chatbot remains relatively constant from ChatGPT, Bard, or other generative AI tools, according to multiple staff members who have tried it.
But as Apple is known for delaying the launch of new products until they are confident they offer a superior experience, we may infer that when Apple GPT is ultimately released, it will be a powerful competitor to the significant AI chatbots and have a distinct edge.

## Apple finally joins the AI revolution

It isn't the first time this year that Apple's possible AI projects have come to light; in April, it was reported that the company was developing an AI-powered health app with emotional analysis capabilities. Of course, Apple already uses AI in various ways across all of its devices.

For instance, Siri, the company's voice-activated personal assistant in all iPhones, is an example of artificial intelligence.

Apple CEO Tim Cook, who had earlier predicted that AI would be "huge," has also called attention to several privacy issues that, in his view, need to be resolved right now.

## When Will Apple GPT Be Available?

The million-dollar question people are clamoring to find out…

Insiders and rumor sites claim that Apple will make a "significant AI-related announcement" in 2024, which many believe would be regarding Apple GPT and the company's plans to make the platform available to consumers.

## Takeaway

Apple is aggressively investing in AI technology to stay competitive and provide cutting-edge solutions to its user base. However, details about the public release of Apple GPT and its possible capabilities are still unknown. Apple's strategic leap into the AI chatbot market opens the door for new developments in the near future as the realm of AI continues to develop.


Apple GPT: What We Know So Far


# Harness the Power of Cloud GPUs with Vast.AI: Running the 70B LLama2 GPTQ

The rapid advancements in the AI domain have given rise to the need for powerful computational resources. If you're involved in data science or AI research, you're already aware of the immense processing capabilities required to train complex models like the 70B LLama2 GPTQ. Fortunately, with cloud GPU rental services like [Vast.AI](https://vast.ai/), accessing these resources has become easier and more efficient than ever.

## Quick Guide to Launch Oobabooga webUI on Vast.AI

For those interested in leveraging the groundbreaking [70B LLama2 GPTQ](https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ), TheBloke made this possible.

[We’ve created a template](https://cloud.vast.ai/?ref_id=43484&template_id=47c1a7f2cd646ecbb295e4491cd48134) to auto-launch the Oobabooga webUI. Notably, this template grants users the versatility of direct SSH and Jupyter (notebook) integration.

Users should note this particular model demands a staggering 40 GB of VRAM. To accommodate this, Vast.AI's interface has a **VRAM slider** to guide you in selecting suitable machines like the A6000, A40, and A100.

For the 70B Model, we recommend the A6000 (_currently $0.50/hr_) or A40 (_currently $0.40/hr_). To run the 13B model, we recommend either a 3090 (_currently $0.20/hr_) or 4090 (_currently $0.48/hr_). To see all current pricing, refer to our dynamic [pricing page](https://vast.ai/pricing), or head to our [search console](https://cloud.vast.ai/).

## Steps to Download and Execute the Model in text-generation-webui:

1. Navigate to the Model tab.
2. In the section labeled Download custom model or LoRA, input TheBloke/Llama-2-70B-chat-GPTQ.
3. For downloading from specific branches, for instance, TheBloke/Llama-2-70B-chat-GPTQ:gptq-4bit-32g-actorder_True, refer to the Provided Files section for a comprehensive list of branches for each choice.
4. Click Download. Wait for the model to download, and once it's completed, a "Done" message will appear.
5. Set the Loader to ExLlama if you're planning to utilize a 4-bit file. Alternatively, opt for AutoGPTQ or GPTQ-for-LLaMA.
6. Using AutoGPTQ? Ensure that the "No inject fused attention" option is selected.
7. On the top left, click the refresh icon adjacent to the Model label.
8. From the Model dropdown, select the freshly downloaded model: **TheBloke/Llama-2-70B-chat-GPTQ**.
9. Once the model automatically loads, it's set and ready for deployment!
10. Secure your settings by clicking Save settings for this model. Then, click Reload the Model in the top right to ensure your configurations are retained.
11. Now, head over to the Text Generation tab, input your desired prompt, and watch the magic unfold!

For those considering running LLama2 on GPUs like the 4090s and 3090s, **TheBloke/Llama-2-13B-GPTQ** is the model you'd want. Vast.AI's platform is diverse, offering a plethora of options tailored to meet your project's requirements.

For additional details and to delve deeper, please visit the official [github page for ooba](https://github.com/oobabooga/text-generation-webui).

In today's competitive digital environment, it's crucial to have scalable and reliable computational resources at your fingertips. Vast.AI is revolutionizing the way researchers and developers access and utilize GPU power, making AI model training seamless and efficient. Whether you're an AI novice or an established researcher, Vast.AI is your go-to solution for all your cloud GPU rental needs.


Harness the Power of Cloud GPUs with Vast.AI: Running the 70B LLama2 GPTQ.

Harness the Power of Cloud GPUs with Vast.AI: Running the 70B LLama2 GPTQ


## Introduction

Meta has recently launched Llama 2, the latest edition of the widely recognized Llama models, which has been trained with a 40% increase in data. The announcement of this new model is quite thrilling, considering the meteoric rise in popularity of open-source large language models (LLMs) like Llama. The Llama LLM bares some similarities to other large language models, but it’s commercially available for free, which levels the playing field. Llama 2 was pre-trained on an enormous dataset of publicly available online text and code. The fine-tuned model, Llama-2-chat, was trained on this dataset as well as over 1 million human annotations. These models have given rise to popular offspring such as Vicuna and Falcon.

Concurrently, the open-source community has launched an abundance of utilities designed to fine-tune and deploy these language models. Tools such as Peft, Bitsandbytes, and TRL allow for fine-tuning of LLMs on machines that don't have the capacity to hold the full precision model in their GPU RAM.

The aim of this blog post is to guide you on how to fine-tune Llama 2 models on the Vast platform. We have benchmarked this on an RTX 3090, RTX 4090, and A100 SMX4 80GB. See the [latest pricing](https://vast.ai/pricing) on Vast for up to the minute on-demand rental prices.

## Benchmarks

The RTX 4090 demonstrates an impressive 1.5 8-bit samples/sec with a batch size of 8. This is almost twice as fast as running this on an A100 when accounting for batch size! Considering that the RTX 4090 is $0.50/hr, the price for performance is about 6X when compared to an A100 for $1.50/hr.

<table className="llamaTable mx-auto table-fixed" style={{color: "white"}}>
<tr className="llamaTableRow" style={{ padding: '10px', textAlign:'left',     borderBottom: "1px solid lightGray"}}>
<th style={{  width: '4.5rem' }}>Model</th>
<th style={{  width: '10rem' }}>GPU</th>
<th style={{  width: '7rem' }}>Batch Size</th>
<th style={{ width: '12rem'}}>4bit samples/sec[1]</th>
<th style={{ width: '11rem'}}>8bit samples/sec[1]</th></tr>

<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>RTX 4090</th>
  <th>8</th>
  <th>1</th>
  <th>1.5</th>
</tr>
<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>A100 SMX4 80GB</th>
  <th>32</th>
  <th>0.5</th>
  <th>0.8</th>
</tr>
<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>A100 RTX 3090</th>
  <th>8</th>
  <th>0.5</th>
  <th>0.9</th>
</tr>

</table>

[1] Here samples/sec is calculated by multiplying batch size by the inverse of s/iter that the sft_trainer script reports. All training runs had gradient accumulation steps equal to 1.

## How to run this yourself on Vast

Follow the rest of this post for a guide on how to fine-tune LLaMA 2 on Vast using TRL.

## Sign up with Meta and Hugging Face for access:

1- Request access from Meta here: [https://ai.meta.com/resources/models-and-libraries/llama-downloads/](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)

2- Request access from Hugging Face on any of the model pages: [https://huggingface.co/meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b)

3 - Set up an auth token with Hugging Face here: [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)

You will need to do both 1 and 2 in order to get access to LLaMA 2.

## Rent a powerful GPU on Vast.ai

Vast has RTX 3090s, RTX 4090s and A100s for on-demand rentals. Our pricing is typically the best you can find online.

To run LLaMA 2 fine tuning, you will want to use a Pytorch image on the machine you pick. To do that, click on [this Vast console link](https://cloud.vast.ai/?ref_id=43484&template_id=2d307d4c1ba166b8e4bafcd40d3033e2), which will select our recommended Pytorch template with SSH enabled and other settings enabled. If you don’t have an account set up, no problem! You will need to first quickly set up a Vast account by registering your email, verifying your email, and then purchasing credits.

To rent a machine, pick an RTX 3090 or RTX 4090 by selecting that filter in the upper right. Move the storage slider over on the left of the interface to ~30GB so your instance will have enough storage to download the model weights.

Hit the rent button to start the instance. Once it is done loading, hit the blue <\_ button to get the SSH details and SSH into the instance by copying and pasting that SSH command to your command line. You will now be logged into your Vast instance via SSH. If you have trouble setting up SSH, [read our SSH docs](https://vast.ai/docs/instance-setup/ssh).

## Set up environment

Setup the packages you will need:

`pip install transformers peft trl bitsandbytes scipy`

Clone the TRL repo for the training script

`git clone https://github.com/lvwerra/trl`

Log into HuggingFace on CLI

`huggingface-cli login`

Copy the auth token you created earlier (from https://huggingface.co/settings/tokens) and paste it into the prompt when asked. You can decline adding the token to your git credentials.

Fine tune!

`python trl/examples/scripts/sft_trainer.py --model_name meta-llama/Llama-2-7b-hf --dataset_name timdettmers/openassistant-guanaco --load_in_8bit --use_peft --batch_size 8 --gradient_accumulation_steps 1`

This will download the model weights automatically, so the first time you run, it will take a bit to actually start training. The dataset is specified there and it is using the openassistant-guanaco set. If you want to try loading in the 4bit version or change the batch size, modify that command.

You should end up seeing output like this:

```
{'loss': 1.6493, 'learning_rate': 1.4096181965881397e-05, 'epoch': 0.0}
{'loss': 1.3571, 'learning_rate': 1.4092363931762796e-05, 'epoch': 0.0}
{'loss': 1.5853, 'learning_rate': 1.4088545897644193e-05, 'epoch': 0.0}
{'loss': 1.4237, 'learning_rate': 1.408472786352559e-05, 'epoch': 0.0}
{'loss': 1.7098, 'learning_rate': 1.4080909829406987e-05, 'epoch': 0.0}
{'loss': 1.4348, 'learning_rate': 1.4077091795288384e-05, 'epoch': 0.0}
{'loss': 1.6022, 'learning_rate': 1.407327376116978e-05, 'epoch': 0.01}
{'loss': 1.3352, 'learning_rate': 1.4069455727051177e-05, 'epoch': 0.01}
```

## Summary

The price for performance on Vast when using an RTX 4090 is quite impressive when compared to an A100. In this post we have shown to easy it is to spin up a very low cost GPU ($0.20 per hour) and fine-tune the LLaMA 2 models.


LLaMA 2 fine-tuning on a Vast RTX 4090 for $0.50/hr outperforms an A100.

Fine-tuning the LLaMA 2 model on RTX 4090


In the rapidly evolving world of cloud computing, Vast.ai remains steadfast in our commitment to being the leader in cost-effective cloud GPU rental. Today, we are thrilled to announce our latest feature – Reserved Instance Discounts – aimed at helping our users get even better pricing for longer term reservations.

## On-Demand vs Reserved: Striking the Right Balance

In the on-demand world of cloud computing, users only pay for what they use and have the flexibility to end a contract at any time. While this offers unrivaled flexibility, the occupancy rate – or the fraction of time GPUs are rented – often clocks in at less than 100%, sometimes even as low as 50%. The resulting impact on provider revenue is noticeable as their hardware sits idle.

Clients willing to commit upfront to a longer rental term can now leverage our Reserved Instance Discounts feature which solves this idle problem. With this, clients can commit to an extended rental contract by pre-paying in advance. While they give up the on-demand liquidity, they receive a hefty discount in return. The provider controls the discount percentage, so each offer will vary. 

## How Reserved Instance Discounts Work

Here's the breakdown:

1. Users can deposit credits into a specific instance using either the [CLI](https://cloud.vast.ai/cli/) or website GUI (look for the **green tag**). The CLI gives users a detailed control over deposit amounts, while the GUI provides a user-friendly interface. The clickable green tag in the GUI is shown for participating machines in the bottom right of the instance card.

![green tag](/uploads/prepaidicon.png)

2. The discount schedule is a function of the time duration of the balance. Larger deposits unlock greater discounts, albeit with diminishing returns. The default discount schedule offers a 20% discount for a 1-month commitment, 30% for a 3-month commitment, and 40% for a 6-month commitment.

![](/uploads/reservedmodal.png)

3. To give an example, if an instance usually costs $1/hr on-demand, a 1-month prepaid rental with the default discount schedule would cost $576 instead of $720. That's a savings of 20%!

4. When additional funds are deposited into an existing reserved instance, everything is recalculated according to the new total balance, leading to a new discount rate.

The specific discount schedule can vary from provider to provider, with 40% at 6 months being the default setting.

## Link to the Documentation

For the complete description of how these works, read our documentation for [Reserved Instance Discounts](https://vast.ai/docs/instance-rental/prepaid-plans).

## Seize the Savings

Our new Reserved Instance Discounts feature is here to change the way you use cloud computing, providing an avenue to commit to your long-term compute needs while reaping the benefits of considerable discounts. So whether you are a start-up scaling your operations or an established business with predictable computing demands, let Vast.ai help you make the most of your cloud computing budget. Your savings start today with Reserved Instance Discounts. Take control of your cloud computing journey and optimize your budget like never before.





Our on-demand rentals provide incredible value. For hosts that enable this feature, you can get further discounts by reserving an instance for months.

Stretching Your Cloud Computing Budget: Introducing Reserved Instance Discounts


As we delve deeper into the realm of artificial intelligence, one of the most impactful developments is the rise of large language models. These sophisticated tools, designed to understand and generate human-like text, have a myriad of applications spanning diverse fields. This article provides a comprehensive analysis of the key factors impacting their major use cases, including customer service, education, healthcare, legal, and marketing. We will also discuss the trade-offs and challenges associated with these different approaches, highlighting the importance of informed decision-making when choosing a language model.

## An Overview of Large Language Models

Large language models, such as OpenAI's GPT-3 and Google's BERT, are AI models trained on vast amounts of text data. Their capabilities range from understanding context in a conversation, translating languages, to even generating human-like text. The implications of these capabilities are far-reaching, with applications in various sectors.

## Major Use Cases: From Customer Service to Marketing

One of the significant applications of large language models is in customer service. These models can power chatbots and virtual assistants, providing fast, accurate, and personalized responses to customer inquiries. However, the trade-off here is the lack of human touch and potential misunderstandings due to the limitations in understanding complex human emotions and intentions.

In education, large language models can help develop intelligent tutoring systems, providing personalized learning experiences for students. While this promises a more scalable education solution, the risk lies in ensuring the accuracy and appropriateness of the content generated.

In the healthcare sector, large language models can assist in analyzing patient records and medical literature, providing recommendations or aiding in diagnoses. However, the stakes are high, as inaccurate or misinterpreted information can have serious consequences.

Legal professionals can use large language models to analyze legal documents, contracts, and case laws, which can automate some aspects of legal research. But the nuanced and critical nature of legal work means that outputs must be thoroughly reviewed for accuracy.

Lastly, in marketing, language models can generate creative marketing content and analyze consumer sentiment. The challenge here lies in ensuring that the content remains authentic and resonates with the target audience.

## Challenges and Considerations

Despite the impressive capabilities of large language models, their use is accompanied by several challenges. One notable issue is the risk of bias, as these models can inadvertently reproduce biases present in their training data. This risk underscores the importance of carefully curating training data and monitoring model outputs.

The selection of an appropriate language model often depends on the user's specific needs and constraints. While more powerful models can generate higher-quality outputs, they also require more computational resources, thus increasing costs.

## Conclusion

Large language models offer transformative potential across various sectors, from customer service to marketing. However, their implementation is not without challenges, and users must carefully weigh the trade-offs. By considering factors such as performance, cost, and the potential for bias, users can make informed decisions about the best model to use. Despite the complexities, the world of large language models holds immense promise and potential for advancement.

We take a deep dive into the pros and cons of large language models like OpenAI's GPT-4 and Google's BERT systems.

The Versatile Power of Large Language Models: Exploring Major Use Cases and Their Trade-offs


The Nvidia RTX 4090 is a highly reliable and powerful GPU released to the PC gaming market. However it is also suitable for machine learning and deep learning jobs. Whether you're a data scientist, AI researcher, or developer looking for a GPU with high deep learning performance to help take your projects to the next level, the RTX 4090 is an excellent choice.

The RTX 4090 is a high-end GPU powered by the Ada Lovelace architecture. Equipped with 16,384 CUDA cores and 512 Turing Tensor Cores, this GPU is a monster of computing power and can easily tackle the most complex deep learning tasks. This makes it the ideal choice for tackling difficult projects such as facial recognition, natural language processing, and computer vision.

Vast.ai has on-demand [cloud RTX 4090](https://cloud.vast.ai/?gpu_option=RTX%204090) rentals available now.

## NVLink Phased Out

Nvidia is opting for PCIe 4.0 instead of NVLink as a bridge for those who desire a multi-GPU setup with the latest RTX 4000 series, catering to gamers, creators, and workstation users. It will use PCIe 4.0 even when inserted in a PCIe 5.0 slot, due to backward compatibility.

Before launch, Nvidia CEO Jensen Huang announced to reporters that the company will be discontinuing its NVLink bridge and introducing PCIe Gen 5 as its replacement. Huang and Nvidia stated that this new connection will be fast enough for linking multiple GPUs and that the space previously used by the NVLink connector will now be dedicated to additional AI computing.

"The reason why we took [NVLink] out was because we needed the we needed the I/Os for something else, and so, so we use the I/O area to cram in as much as much AI processing as we could," Huang confirmed and explained of NVLink's absence.

## CUDA Library Support on 4090

The RTX 4090 also supports the latest version of Nvidia's CUDA-X AI Library. This library provides data scientists and developers with a rich set of optimized algorithms for deep learning applications. With its help, they can quickly and easily build complex projects which require high-performance computing, and add GPU acceleration to their projects build complex deep-learning models in a shorter time.

To put it simply, the GeForce RTX 4090 is an excellent choice for those interested in affordable deep learning. This is especially true for students, researchers, and creators on a budget. While it's faster the previous flagship GPU, the GeForce RTX 3090, it's also more cost-effective in terms of training throughput.

## Additional Benefits for Deep Learning

The 4090 is also equipped with 24 GB of GDDR6 VRAM, allowing it to store and retrieve large amounts of data quickly and effectively. It's well suited for developers and data scientists who need to quickly perform significantly higher computation tasks in the world of machine and deep learning.

The GPU also contains DLSS AI upscaling, which can improve the performance of your deep learning models by 200%. This makes the 4090 a great choice for both training and serving models.

## Final Thoughts on the RTX 4090 for Deep Learning

Overall, the RTX 4090 is an impressive piece of technology for deep learning. The 4090 is an ideal choice for those looking to take their deep learning and machine learning projects to the next level. With its powerful computing capabilities and easy integration with Nvidia's CUDA libraries, it's designed to help you tackle any task quickly and effectively.

With its powerful architecture and generous CUDA core count, it has the capability to tackle difficult projects with ease. If you're looking for a reliable GPU that can power your deep learning projects, the RTX 4090 is a great choice.


We dig deeper into the Nvidia RTX 4090 and its uses in deep learning.

Nvidia RTX 4090 as a Deep Learning GPU


The rapid rise in the demand for high-performance computing has led to an increasing need for efficient, cost-effective solutions. One such solution is GPU rental. These services offer organizations the flexibility to access powerful processing units without the hefty upfront investment. Let's look at what exactly renting a GPU entails, their application in Deep Learning (and cryptocurrency), and why Vast.AI stands out as the best rental GPU source.

## Understanding GPU Rentals

A Graphics Processing Unit (GPU) is a powerful electronic circuit designed to quickly manipulate and render images, 3D graphics, and other complex calculations. The parallel processing capabilities of GPUs make them an ideal choice for tasks that demand high computational power.

In recent years, GPU rentals have been on the rise. This service enables users to rent powerful GPUs on a pay-as-you-go basis, significantly reducing the cost associated with purchasing and maintaining high-end GPUs. One such service provider that has gained prominence in the GPU rental market is Vast.AI.

## How Does Vast.AI Work?

Vast.AI is a global GPU marketplace that aims to reduce the cost of cloud GPU rentals for AI/ML and GPU-related compute needs. By leveraging a sharing economy approach, Vast.AI offers GPU rentals at prices significantly lower than its competitors.

Hosts with professional-grade machines can list their devices on Vast.AI and set pricing for various rental components, such as storage and bandwidth. Users looking to rent GPUs can search through the available offers, filtering by GPU type, GPU RAM, CPU, and system RAM. Pricing is transparently displayed for the GPU rental, storage, and any bandwidth charges.

Only at Vast.AI, payments can be made either by credit card or with crypto through either Coinbase or Crypto.com.

Vast.AI uses Linux docker images, and users can select any private or public repo with a Linux docker image to load onto a rented machine. A powerful CLI tool allows for a programmatic interface to automate GPU server rentals.

## Unleashing the Power of Cloud GPUs in Deep Learning

Deep learning, a subset of machine learning, relies on artificial neural networks to mimic the human brain's functioning. It enables the system to learn from vast amounts of data, making it a critical component of artificial intelligence.

GPUs are designed to perform parallel computations, making them an excellent choice for deep learning models. They can expedite large computational tasks due to their many cores and high memory bandwidth. Whether it's training AI models, rendering CAD models, or playing graphics-rich video games, GPUs have become an indispensable tool in the world of technology.

When it comes to deep learning, GPUs can offer performance speeds up to 250 times faster than traditional CPUs. This incredible speed makes them an ideal choice for training neural networks associated with deep learning.

## The Benefits of Cloud GPUs

The advent of cloud computing has taken the capabilities of GPUs to new heights. Cloud GPUs offer several advantages over traditional physical GPUs.

### Scalability

One of the significant benefits of cloud GPUs is their scalability. As an organization grows, its computational needs also increase. Cloud GPUs can easily scale up or down to meet these changing needs, offering flexibility that physical GPUs can't match.

### Cost-Effective

Cloud GPUs also offer a more cost-effective solution. Instead of making a substantial upfront investment in high-powered GPUs, organizations can rent cloud GPUs on an hourly basis. This pay-as-you-go model can significantly reduce costs, especially for organizations that don't need constant access to high-powered GPUs.

### Space and Resource Efficiency

Cloud GPUs don't consume local resources, unlike physical GPUs that take up significant space on your computer. This means your computer can be used for other tasks while the heavy computational work is offloaded to the cloud.

### Time-Saving

Cloud GPUs can save a significant amount of time. Tasks that once took hours or even days to complete can now be finished in minutes, thanks to the power of cloud GPUs. This increased efficiency can lead to higher productivity and more time for innovation.

## Why Vast.AI is the Best Rental GPU Source

Vast.AI stands out in the GPU rental market due to its unique approach. Transparently, Vast.AI offers GPU rentals at significantly lower prices than its competitors.

But cost isn't the only factor that sets Vast.AI apart. The platform also provides a user-friendly interface with detailed information about each available machine, making it easy for users to find the perfect GPU for their needs.

Furthermore, Vast.AI's flexibility of payment options makes it easier for users to pay for their rentals in the way that works best for them.

## Conclusion

In conclusion, GPU rental has emerged as a game-changing solution in the world of high-performance computing. Whether it's deep learning, crypto mining, or any other task that requires significant computational power, GPU rental can provide a cost-effective, flexible solution.

Vast.AI, with its unique approach and user-friendly platform, stands out as the best rental GPU source. Whether you're an AI researcher looking for a cost-effective way to train your models or a crypto miner looking to maximize your efficiency, Vast.AI can provide the powerful, affordable GPU rental solution you need.


We take a look at what renting a GPU entails, their application in Deep Learning, and why Vast.AI stands out as the best rental GPU source.

Revolutionizing Cloud GPU Rentals for Deep Learning with Vast.ai


In the rapidly evolving world of machine learning, the ability to leverage powerful computing resources on-demand has become crucial. The need to run large language models such as LLaMA, GPT-J, Pythia, and Falcon has grown more pronounced, as these models are increasingly important in various AI applications. However, these models require significant computational power, typically in the form of high-performance graphics processing units (GPUs). Many of the models require 40+ GB of GPU RAM to load the models.

This is where Vast and Oobabooga come in.

Vast offers on-demand GPU rentals, providing users with the raw power needed to run these models. It's known for being the most cost-effective offering on the market, making it an ideal choice for those who are budget-conscious or just starting out with AI and machine learning.

[Oobabooga](https://github.com/oobabooga/text-generation-webui) is a web user interface (UI) designed specifically for running large language models. It's simple and intuitive to use, requiring no advanced technical skills or extensive prior knowledge of AI and machine learning.

In this post, we will guide you through the process of deploying Oobabooga on Vast, showing you just how easy it is to run large language models on a budget.

## Getting Started with Vast

The first step is to sign up for an account on Vast. You will need to add credits before you can rent. Go to the billing page and choose a payment method. Larger language models will require more expensive GPUs, so be sure to add enough credits.

## Selecting the Oobabooga template

Vast rents out Linux Docker instances on-demand. We have a pre-built Oobabooga template that you will need to load. Load the template by clicking on [this link](https://cloud.vast.ai/?template_id=d8339b706edae2f312c29fb3940a4b17). 

## Add enough storage

Underneath the Edit image and config options there is a slider which will allocate storage for your instance. Large language models are typically 70+ GB, so make sure to move the slider to 80 GB so your instance will have enough storage to download the model you want to run.

## Pick a GPU

Now comes the fun part! Vast is a marketplace of on-demand cloud GPUs. You can easily select a powerful GPU that supports your selected configuration. Many large language models require the absolute best GPU right now. Read about how much GPU RAM your model needs to run. The top of the line GPU is the A100 SMX4 80GB or A100 PCIE 80GB. To ensure your instance will have enough GPU RAM, use the GPU RAM slider in the interface.

Once you find a suitable GPU, click RENT. 

## Running Your Models

The instance will need to load the Docker image and complete setup. Once this is complete, a blue OPEN button will appear. Click on that to open the Oobabooga web UI interface. 

Read the Oobabooga documentation on their project website [here](https://github.com/oobabooga/text-generation-webui). You will need to download an LLM to the instance which can take some time depending on the internet connection speed of the machine you picked and the place you are downloading the model from. HuggingFace is typically pretty fast. 

Once you have the model loaded, you can start interacting with it. You'll be able to monitor the progress in real-time through the Oobabooga interface, and once the computations are complete, the results will be displayed directly on the screen. The whole process is seamless, and thanks to Vast and Oobabooga, you can run large language models without breaking the bank or needing advanced technical knowledge.

## Conclusion
With the power of Vast's affordable GPU rentals and the simplicity of Oobabooga's user interface, running large language models has never been easier or more accessible. Whether you're a student, a researcher, or an AI enthusiast, this combination of tools provides an excellent way to delve into the world of AI and machine learning without the usual barriers of cost and complexity.

An easy, cost-effective method for deploying LLaMa, GPT-J, Pythia and Falcon

Deploying Oobabooga for Running Large Language Models


Ongoing product updates are an important part of what we offer at Vast.ai. Our customers get excellent performance and reliability in large part because of the consistency with which we work to improve our service. As a Vast.ai customer, you can not only expect to get outstanding NVIDIA GPU cloud performance on day one, but you can expect that experience to continue to improve over time.

The starting point for every product update is to address any bugs that have popped up along the way. Our latest update is no different, including the fixes listed below --

- Ensuring all instances show on instance page
- Correct port allocation that caused instances to fail to start
- Improve disk speed test rates
- Solve error on 4090s causing NVLINK to show erroneously
- Fix billing page error that showed ratelimit error
- Disable extra filters with the show incompatible filter

If you are an existing user, you will notice that these fixes streamline your experience and make the product more enjoyable to use day after day. For potential new customers, you can rest assured that these fixes were made promptly and the save level of service will be delivered on future corrections that need to be made.

## NVIDIA Cloud GPU Fixes

Fixes are one thing, but we also utilize updates to deliver added capabilities and flexibility to the platform. Here are some of the highlights --

- The Juypter open button will run a test to confirm it is running before displaying
- Updated the mobile UX to improve info bar issues
- GUI filter settings now persist
- Hosts receive email notifications of expired machines
- Add ability to delete templates from recent tab
- Added 4080 and 4070 search filter
- Added function for hosts to disconnect Stripe connect for payouts

To confirm the many features and capabilities offered by Vast.ai, you are welcome to view our full documentation, or you can simply reach out directly to our team for more information. You can chat with us directly here on our site, where we will respond quickly and clear up any confusion you may have about what we offer. We also offer a Discord server that you can join to learn more, or you can drop us a message at support@vast.ai. Either way, we would be happy to assist you with your GPU rental needs.

## Change Log

-bugfix with the instance page to ensure all instances show

-bugfix with port allocation causing instances failing to start

-bugfix on the slow rate of disk speed tests

-bugfix on 4090s causing NVLINK to erroneously show

-bugfix on the billing page which sometimes showed a ratelimit error

-bugfix with the show incompatible filter to disable extra filters

-total instances cost per hour/day meter added in the billing page

-Juypter open button now tests to ensure Juypter is running before displaying

-Filters can be passed into the create page via URL

-Updated the mobile UX to fix login/user infobar issues

-Added the ability for hosts to disconnect Stripe connect for payouts

-Added a new recommended template for Bittensor 3.7.0 with readme (edited)

-Relisting offers will only extend the end date of existing contracts when the new price is the same or less

-CLI command to cancel rsync added

-Added ability to delete templates from recent tab

-Added the ability to create a readme file for templates

-Docs updated to reflect template system

-GUI filter settings now persist

-Hosts now receive email notification of expired machines

-Added 4080 and 4070 to search filter (edited)


Ongoing product updates are an important part of what we offer at Vast.ai...

Vast Product Update | April 2023


Cloud GPU rental services are becoming increasingly popular in the past few years for those who want to train deep learning models. Deep learning is a subset of machine learning that utilizes neural networks with multiple layers to analyze and process data. It has been a rapidly growing field in recent years, with many breakthroughs and advancements in the areas of image recognition, natural language processing, and self-driving cars.

Renting GPUs for training can be more cost-effective than purchasing them outright. It also gives users the ability to access the latest and greatest hardware. With cloud-based solutions, organizations can easily access these new GPUs without having to go through the process of purchasing and installing new hardware.

## GPUs for Deep Learning

Deep learning models, such as neural networks, are composed of many layers of interconnected nodes, or neurons. Each neuron performs a simple mathematical operation, such as a dot product or an activation function, on its input data. However, when these operations are applied to millions of neurons across many layers, the computational requirements become extremely large.

This is where GPUs come in. They are specifically designed to perform massive amounts of parallel computation, making them well suited for deep learning tasks. In contrast, traditional CPUs (Central Processing Units) are not optimized for parallel computation and are not able to keep up with the computational demands of deep learning models.

Another reason why GPUs are used for deep learning is that they can be used to perform both the training and the inference of deep learning models. The training process of a deep learning model requires a large amount of data and computational power. This is because the model needs to adjust the values of its parameters, or weights, to minimize the difference between the predicted outputs and the actual outputs. Inference, on the other hand, is the process of applying the trained model to new data. This also requires a significant amount of computational power.

As deep learning models continue to become more complex and data-intensive, the use of GPUs will likely become even more prevalent in the field.

## Choosing a Deep Learning GPU provider

There are a number of cloud GPU rental services available. Some of the most popular cloud GPU rental providers include Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. Each cloud provider's GPU cloud instances have different performance and cost characteristics, so it's important to do your research to find the best instance for your deep learning application.

Vast.ai is a decentralized compute marketplace that allows users to rent cloud GPUs at much lower prices when compared to major providers. The company utilizes a peer-to-peer network of individual and corporate GPU owners to provide users with access to powerful GPU resources at a fraction of the cost of traditional cloud providers. Hosts are both semi-professional and professional datacenters.

No matter which cloud provider you choose, cloud GPUs are ideal for running large-scale deep learning applications such as computer vision, natural language processing (NLP), and machine learning algorithms. You can also use cloud GPUs to build and test deep learning models in less time than with traditional CPUs.

## Choosing the Right GPUs for Deep Learning

When choosing a cloud GPU rental service, it is important to consider the needs of your deep learning applications when you pick a particular GPU, such as an A100, A6000 or RTX 4090. Factors to consider include the size of your training dataset, the number of epochs you need to train your model, the training time for each epoch, and the computational and memory resources required.

Based on these factors, you can compare the prices of different cloud GPU providers and choose the one that best suits your needs.

## Reduce Expenses for Large Datasets and Faster Training

The expense of cloud GPU rental services can differ depending on the company and the particular instance type being utilized. In general, though, renting GPUs from the cloud is much more reasonably priced than purchasing them outright - especially when you take into account all of the other benefits that come with using a cloud service.

If you're training deep learning models and are looking for a cost-effective way to do so, cloud GPU rental services are definitely worth considering. The cost of consumer-grade GPUs makes purchasing them less than stellar.

Cloud providers offer reliable and powerful cloud GPU instances that can help you reduce expenses for large datasets and faster training. With cloud GPU rental services, you can focus on building better models without breaking the bank, and access the best GPUs right away.

So cloud GPUs are the way to go if you have large datasets and need faster training times. Cloud GPU rental services can provide the necessary computing resources without breaking the bank.


GPU prices continue to increase, but there exists a happy and unexpected middle ground for AI applications.

High-Performance Deep Learning with Cloud GPUs


Renting cloud GPUs and buying GPUs are two different options for individuals and organizations that need access to powerful graphics processing units (GPUs) for tasks such as machine learning and data analysis. Both options have their own advantages and disadvantages and the choice between them will depend on the specific needs and budget of the user.

Companies, universities, researchers, and home users alike have started to rent a GPU or entire GPU servers for deep learning and neural networks. Renting GPUs has quickly become the preferred option by many people who need access to powerful GPU cards without the long-term commitment or expense that comes with buying them outright.

## Cost advantages to renting a GPU for machine learning

Let's face it: the price of GPUs is high. A single GPU can range in price from a few hundred dollars to upwards of $10,000.

But, they're incredibly necessary for complex tasks that require high performance and reliability, like image rendering, data science, and blockchain processing on a large scale. Renting a GPU takes that anxiety off the table and prevents people from making costly mistakes (and experiencing buyer's remorse). To make matters worse, a GPU's value doesn't increase in value over time. As a rule of thumb, people can expect a [GPU to lose about 15% of its value relative to the MSRP on the used market each year](https://www.techspot.com/article/2107-your-graphics-card-worth/).

This also offers you a great deal of flexibility. You can start and stop at any time, and when a new GPU comes out, you don't have to replace all your old hardware - just rent the next GPU in the technology tree. You can scale your configuration up and down as needed, add memory and performance to your technology stack, and

## Flexible, short-term infrastructure needs

First and foremost, renting a GPU allows you to rent high-performance GPU hardware for short periods of time, allowing you to rent and test different GPUs without the long-term commitment of a purchase. This is ideal for those who may only need the GPU for a small project or experiment. Users can choose the GPU that best suits their needs and easily scale up or down as needed. Additionally, cloud GPU providers often include additional features such as easy data transfer and remote access.

By choosing to rent, it becomes possible for someone to find the GPU that is best-suited for their particular project. Think of renting as a matchmaking service between the project you're working on or dreaming about and the tech you need to bring it to life.

## Get powerful GPU servers for much less

Renting cloud GPUs and buying GPUs are two different options for individuals and organizations that need access to powerful graphics processing units (GPUs) for tasks such as machine learning and data analysis. Both options have their own advantages and disadvantages and the choice between them will depend on the specific needs and budget of the user.

Companies, universities, researchers, and home users alike have started to rent a GPU or entire GPU servers for deep learning and neural networks. Renting GPUs has quickly become the preferred option by those who need access to powerful GPU cards without the long-term commitment or expense that comes with buying them outright.

## Cost advantages to renting a GPU for machine learning

Let's face it: the price of GPUs is high. A single GPU can range in price from a few hundred dollars to upwards of $10,000.

But, they're incredibly necessary for complex tasks that require high performance and reliability, like image rendering, data science, and blockchain processing on a large scale. Renting a GPU takes that anxiety off the table and prevents people from making costly mistakes (and experiencing buyer's remorse). To make matters worse, a GPU's value doesn't increase in value over time. As a rule of thumb, people can expect a [GPU to lose about 15% of its value relative to the MSRP on the used market each year](https://www.techspot.com/article/2107-your-graphics-card-worth/).

This also offers you a great deal of flexibility. You can start and stop at any time, and when a new GPU comes out, you don't have to replace all your old hardware - just rent the next GPU in the technology tree. You can scale your configuration up and down as needed, add memory and performance to your technology stack, and

## Flexible, short-term infrastructure needs

First and foremost, renting a GPU allows you to rent high-performance GPU hardware for short periods of time, allowing you to rent and test different GPUs without the long-term commitment of a purchase. This is ideal for those who may only need the GPU for a small project or experiment. Users can choose the GPU that suits their needs and easily scale up or down as needed. Additionally, cloud GPU providers often include additional features such as easy data transfer and remote access.

By choosing to rent, it becomes possible for someone to find the GPU that is best suited for their particular project. Think of renting as a matchmaking service between the project you're working on or dreaming about and the tech you need to bring it to life.

## Get powerful GPU servers for much less

Additionally, this allows you to rent more powerful GPUs than you can afford to buy. This also gives users the ability to rent GPUs with unique characteristics that may not be available for purchase, such as special processor configurations or multiple cards in one system. Any future updates can be added, enabling you to keep up with the latest trends and new technology, without having to re-sell the old hardware to buy anew.

## Minimize upfront costs

Finally, renting GPUs is much more cost-effective than buying a GPU outright. Not only do you avoid the upfront expense of purchasing GPU hardware, but you also avoid the ongoing costs of maintenance, depreciation, and energy consumption that come with owning a GPU.

GPUs enable computers to process many pieces of data at once, making them invaluable for people who are into deep learning, edit movies, mine cryptocurrencies, frequently use gaming applications, and more. For anyone who has wondered if renting or buying a GPU is best for their needs, read on to find out why renting makes the most sense.

## A high-performance computing cloud that stays relevant

Talk to anyone with an interest in electronics, whether they always want the latest phone, music equipment, or power tool, and they will all say the same thing: "something bigger and better is always right around the corner."

Not to worry when renting, with technology changing quickly, topics like maintenance, upgrades, and new product launches aren't even in your vocabulary. Leave it to someone else--you have access to the latest and greatest at your fingertips!

## Future-proof your systems

Just like with any major purchase, trying to buy something complicated with limited knowledge can turn a purchase into a risky gamble. Renting a GPU will save you tons of time on research. Rather than spending hours reading product reviews and bouncing around to different sites, renting makes your life easy. The company you rent from has done all of the work and condensed hundreds of articles into easy-to-digest information, allowing you to make your selections with a few clicks of a mouse. Now, that's an easy button!

From houses and cars to parking spaces, RVs and tools, the option to rent has become more and more popular. And why wouldn't it -- it takes away the stress and financial implications of owning. Renting a GPU is the smartest way to get access to the resources you need at a lower cost.

## About Vast.ai

Vast.ai is a decentralized compute marketplace that allows users to rent cloud GPUs at much lower prices than major cloud providers like Amazon Web Services (AWS), Google Cloud, and Microsoft Azure. The company utilizes a peer-to-peer network of individual GPU owners to provide users with access to powerful GPU resources at a fraction of the cost of traditional cloud providers. Additionally, this allows you to rent more powerful GPUs than you can afford to buy. This also gives users the ability to rent GPUs with unique characteristics that may not be available for purchase, such as special processor configurations or multiple cards in one system. Any future updates can be added, enabling you to keep up with the latest trends and new technology, without having to re-sell the old hardware to buy anew.

## Minimize upfront costs

Finally, renting GPUs is much more cost-effective than buying a GPU outright. Not only do you avoid the upfront expense of purchasing GPU hardware, but you also avoid the ongoing costs of maintenance, depreciation, and energy consumption that come with owning a GPU.

GPUs enable computers to process many pieces of data at once, making them invaluable for people who are into deep learning, edit movies, mine cryptocurrencies, frequently use gaming applications, and more. For anyone who has wondered if renting or buying a GPU is best for their needs, read on to find out why renting makes the most sense.

## A high-performance computing cloud that stays relevant

Talk to anyone with an interest in electronics, whether they always want the latest phone, music equipment, or power tool, and they will all say the same thing: "something bigger and better is always right around the corner."

Not to worry when renting -- with technology changing quickly, topics like maintenance, upgrades, and new product launches aren't even in your vocabulary. Leave it to someone else--you have access to the latest and greatest at your fingertips!

## Future-proof your systems

Just like with any major purchase, trying to buy something complicated with limited knowledge can turn a purchase into a risky gamble. Renting a GPU will save you tons of time on research. Rather than spending hours reading product reviews and bouncing around to different sites, renting makes your life easy. The company you rent from has done all of the work and condensed hundreds of articles into easy-to-digest information, allowing you to make your selections with a few clicks of a mouse. Now, that's an easy button!

From houses and cars to parking spaces, RVs and tools, the option to rent has become more and more popular. And why wouldn't it -- it takes away the stress and financial implications of owning. Renting a GPU is the smartest way to get access to the resources you need at a lower cost.

## About Vast.ai

Vast.ai is a decentralized compute marketplace that allows users to rent cloud GPUs at much lower prices than major cloud providers like Amazon Web Services (AWS), Google Cloud, and Microsoft Azure. The company utilizes a peer-to-peer network of individual GPU owners to provide users with access to powerful GPU resources at a fraction of the cost of traditional cloud providers.


Blog