
Vast.ai provides a platform where developers can access a wide selection of GPUs from various hosts and data centers, offering competitive pricing to fit any budget. The cost and availability of GPUs are critical to the success and speed of generative AI projects. We’re excited to announce that [dstack](https://dstack.ai/) now includes integration with our [GPU marketplace](https://vast.ai/). dstack is an open-source cloud platform for training and deploying gen AI models. This integration enables our users to access an extensive selection of affordable GPUs, breaking down the barriers to entry for developers and ensuring that high costs do not stifle innovation.

# Diverse GPU Options at Your Fingertips

Vast.ai’s marketplace stands out by offering a wide selection of GPUs, ensuring that developers can select the best fit for their projects without cost constraints. The platform recommends GPU instances based on a comprehensive quality score but also gives you the flexibility to set a maximum price, ensuring that you stay within your budget.

# Empowering Developers with dstack and Vast.ai

With dstack 0.12.3, accessing Vast.ai's marketplace becomes a seamless part of the development workflow. Now, dstack’s intuitive CLI and API are equipped to tap into Vast.ai's resources, allowing you to train and deploy generative AI models with greater ease and less financial strain. This integration streamlines your development process, enabling you to utilize the cost benefits of Vast.ai without the hassle of complex cloud infrastructure management.

![](/uploads/dstack-top.png)

# Effortless Setup and Deployment

Using Vast.ai with dstack is easy and straightforward:

1. Log into your Vast.ai account and navigate to the 'Account' section to find your API Key.
2. With your API Key, configure the dstack backend according to the provided steps.

![](/uploads/dstack-bottom.png)

3. Once configured, begin using the robust features of dstack’s CLI and API.

This integration not only facilitates [fine-tuning](https://dstack.ai/docs/guides/fine-tuning/) and [deployment](https://dstack.ai/docs/guides/text-generation/) of models but also the creation of [development environments](https://dstack.ai/docs/guides/dev-environments/) that support interactive coding, making it a perfect fit for developers who prefer to work within their favorite IDEs.

# Seamless Switching Between Cloud Providers

The flexibility that dstack offers is unparalleled. You can now effortlessly switch to Vast.ai when it suits your project needs while retaining the option to use other cloud providers as required. This ensures that your workflow is as efficient and cost-effective as possible.

# Streamline Your AI Projects with dstack and Vast.ai

If you're looking for a simple and cost-effective way to train and deploy generative AI models, the combination of dstack and Vast.ai is ready for you. It's designed to save you time and money. With dstack’s new preview versions of the [fine-tuning](https://dstack.ai/docs/guides/fine-tuning/) and [text generation](https://dstack.ai/docs/guides/text-generation/) APIs, your development process just got even better.

Need a hand? Our [community](https://discord.gg/hSuEbSQ4X8) as well as dstack’s [community on Discord](https://discord.gg/u8SmfwPpMd) is available to offer support and answer your questions. We're committed to helping you make the most of what Vast.ai and dstack have to offer.


We’re excited to announce that dstack now includes integration with our GPU marketplace.

Accessing the GPU marketplace with Vast.ai and dstack


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support! 

With this latest update, we're welcoming both the Radeon and Instinct product lines from AMD to the existing array of NVIDIA machines available for rent on our marketplace. As we widen the range of supported hardware, we can help connect even more customers with the specific GPU compute requirements they're looking for. 

Here are the [AMD GPUs we have available now](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76).

## Now Supporting AMD Instinct and Radeon GPUs

The Instinct line is AMD's brand of data center GPUs, intended to accelerate deep learning, artificial neural network, and high-performance computing tasks, while the Radeon line is aimed at gaming and consumer-grade applications. With both now supported on Vast, this expansion means more versatility and options for all of our users.

Travis Cannell, COO at Vast.ai, highlights the strategic advantage of the update: "We are interested in partnering with managed service providers who have the Instinct line and who want to increase the utilization of their clusters, as well as the Radeon line for as our users have needs for remote gaming, rendering and more." 

Ultimately, our platform will now be better equipped to meet the diverse needs and preferences of our user base – from data centers to gamers and everyone in between. Whether you're scaling up operations or just starting out, we're here to support your computing journey.

## How to Get Started

To browse the active AMD listings on cloud.vast.ai, you will need to select an AMD template which will then filter results for AMD GPUs. Renters looking to leverage these new GPU options on Vast should use the ROCm template. The two templates supported now include an Ubuntu 22.04 image and a PyTorch image.

To see the active results, select the desired template:

- [AMD Ubuntu 22.04 ROCm 6.02](https://cloud.vast.ai/?ref_id=43484&template_id=9e190d88664e20818ee75eb33dce943c)
- [AMD ROCm PyTorch](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76)

You can also find the latest AMD templates by looking for AMD on the [template page](https://cloud.vast.ai/templates/).

![Vast.ai AMD Templates](/uploads/amd-templates.PNG)


## How to Host AMD Equipment

For those interested in hosting their GPUs, Vast.ai provides comprehensive instructions in our [Machine Hosting Setup guide](https://cloud.vast.ai/host/setup/). 

Whether you're setting up an AMD or NVIDIA machine, the step-by-step guide gives you all the information you need to get started. From system requirements and networking configuration to troubleshooting and how to list your machine, learn to optimize your hosting setup and get up and running as soon as possible.

## Leading the Way in GPU Rental Innovation

The addition of support for AMD GPUs comes at a time when Vast.ai has announced an incredible 265% year-over-year growth since 2019. This growth illustrates the heavy demand in the GPU rental market. With more than 350 independent hosts supplying 16,000+ GPUs – and growing – Vast.ai efficiently matches GPU supply and demand, ensuring that every user finds the best deal for their specific compute needs.

As Vast.ai leads the charge in GPU rental innovation, the addition of AMD support is more than just an upgrade – it's also a reflection of our commitment to the democratization of AI. Join us as we continue to make powerful computing more accessible to everyone.


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support!

Announcing AMD Support


## Background
vLLM is an open source framework for Large Language model inference. It specifically focuses on throughput for serving and batch workloads. This is important for building apps for multiple users and at scale.

vLLM provides an OpenAI compatible server, which means that you can integrate it into chatbots, and other applications

As companies build out their AI products, they often hit roadblocks like rate limits and cost for using these models. With vLLM on Vast, you can run your own models in the form factor you need, but with much more affordable compute. As inference grows in demand with agents and complicated workflows, vLLM on Vast shines for performance and affordability where you need it the most.


This guide will show you how to setup vLLM to serve an LLM on Vast. We reference a notebook that you can use [here](/uploads/serving_vllm_on_vast.ipynb)

## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account and add credits, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving an LLM, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU with decent RAM since we're going to serve a single small model. `vLLM` also requires Cuda version 12.1 or higher, so we will filter for that as well. We will query the vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.1' 
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 


```bash 
vastai create instance <instance-id> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 40 --args --model stabilityai/stablelm-2-zephyr-1_6b
```

## Connecting and Testing:
To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view.png)

We will copy over the IP address and the port into the cell below.
```bash
# This request assumes you haven't changed the model. If you did, fill it in the "model" value in the payload json below
curl -X POST http://<IP-Address>:<Port>/v1/completions -H "Content-Type: application/json"  -d '{"model" : "stabilityai/stablelm-2-zephyr-1_6b", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

## Advanced Usage: Serving a Quantized Llama-3-70b Model:
Now that we've spun up a model on vLLM, we can get into more complicated deployments. We'll work on serving this specific quantized Llama-3 70B [model](https://huggingface.co/casperhansen/llama-3-70b-instruct-awq).

With this quantized model, we can easilly serve this model on on 4 4090 GPU's.

### What is different this time around:
1. The model string - we need to use the new model id.
2. We're going to use 4 GPU's instead of just 1.
3. We need to provision much more space on our system to be able to download the full set of weights. 100 GB in this case should be fine.
4. We need to set up tensor parallelism inside vLLM to split up the model across these 4 gpus. 
5. We need to let vLLM know that this is a quantized model

First, we will search for instances that match our needs
``` bash
vastai search offers 'compute_cap >= 800 gpu_ram >= 24 num_gpus = 4 static_ip=true direct_port_count > 1 cuda_vers >= 12.1' 
```

In our instance creation, we will increase our disk usage to 100GB.

Then, we will tell vllm to: 1. use the specific model, 2. split across 4 GPU's, and 3. Let it know that it is in fact a quantized model.

``` bash
vastai create instance <Instance-ID> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 100 --args --model casperhansen/llama-3-70b-instruct-awq --tensor-parallel-size 4  --quantization awq 
```

### Other things to look out for with other configurations:

If you are downloading a model that needs authentication from the huggingface hub, passing `-e HF_token=<Your-Read-Only-Token>` within vast's `--env` variable string should help.

Sometimes the full context of a model can't be used given the space allocated for vLLM on the GPU + the models size. In those cases, you might want to increase `--gpu-memory-utilization`, or decrease the `max-model-len`. Increasing `--gpu-memory-utilization` does come with CUDA OutOfMemory Issues that can be hard to predict ahead of time.

We won't need either of these for this specific model and GPU configuration.

### Testing:

Copy the IP address from your instance once it is ready, and then we can use the following code to call it. Note that while your server might have ports ready, the model might not have downloaded yet as it is much larger this time. You can check the status of this via the logs to see when it has started serving.

```bash
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'casperhansen/llama-3-70b-instruct-awq',
    'prompt': 'Hello, how are you?',
    'max_tokens': 50,
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/completions', headers=headers, json=json_data)
print(response.content)
```


## Conclusions
Model inference is expensive, and leveraging more affordable compute/models makes a huge difference for engineering teams in terms of margins, and shipping velocity.

Using vLLM on Vast is perfect for this, pairing Vast's access to affordable compute with the simplicity and State of the Art throughput of the vLLM backend.

vLLM is a great beginning to building Generative AI Apps. We will continue to explore using this tool more with Vast in future posts.

Llama-3 is already ready to go on Vast to start experimenting and building!


Serving Online Inference with vLLM API on Vast.ai


NVIDIA's highly anticipated next-generation GH200 Grace Hopper Superchip platform is finally available. Perfectly suited for the era of accelerated computing and generative AI, it's essentially the world's most powerful GPU paired with the most versatile CPU. 

  

With its scalable design, the NVIDIA GH200 can handle the most complex generative AI workloads, from large language models (LLMs) and recommender systems to vector databases, graph neural networks (GNNs), and beyond. 

  

"To meet surging demand for generative AI, data centers require accelerated computing platforms with specialized needs," [said](https://nvidianews.nvidia.com/news/gh200-grace-hopper-superchip-with-hbm3e-memory) NVIDIA founder and CEO Jensen Huang. This is precisely the gap that the GH200 aims to fill. 

  

Huang added, "[It] delivers this with exceptional memory technology and bandwidth to improve throughput, the ability to connect GPUs to aggregate performance without compromise, and a server design that can be easily deployed across the entire data center."

  

### Groundbreaking Memory and Bandwidth

  

The GH200's architecture redefines high-performance computing (HPC). It integrates the cutting-edge Hopper GPU and the versatile Grace CPU into a single superchip – bridged by the lightning-fast, memory-coherent NVIDIA NVLink Chip-2-Chip (C2C) interconnect. 

  

The NVLink-C2C interconnect is the heart of the GH200 Grace Hopper Superchip. The power of its coherent memory is pretty incredible; the interconnect delivers 900GB/s of bidirectional CPU-GPU bandwidth. That's seven times the performance compared to the PCIe Gen5 connections found in conventional accelerated systems! Plus, interconnect power consumption is slashed by more than 5X.

  

With NVLink-C2C, applications can oversubscribe the GPU's memory, harnessing the high-bandwidth memory of the Grace CPU directly. The GH200 offers up to 480GB of LPDDR5X CPU memory, meaning that – combined with 96GB HBM3 or 144GB HBM3e – the Hopper GPU can directly access up to 624GB of fast memory. 




![](/uploads/wxoZpDKUOFRqBvNJkCLco6gy3-gBofvjPwirVSYuHDdHVBbFVT-JDjSOQHIYWTSV6VTnVj4U5kFJ9lhfUeHalOA5N0QRdDAV0TxByVawrhTyJID0yActvuTghOYzziRHm9xDCI0xIwY3ubuD5nRSEas.png)


(Image credit: NVIDIA)

  

Key features of the NVIDIA Grace CPU include:

  

- 2X the performance per watt of conventional x86-64 platforms
    
- 72 Neoverse V2 Armv9 cores with up to 480GB of server-class LPDDR5X memory with error-correction code (ECC)
    
- Up to 53% more bandwidth at one-eighth the power per GB per second, compared to an eight-channel DDR5 design
    

  

Then there's the H100 Tensor Core GPU, based on the new Hopper GPU architecture, which showcases several innovative highlights:

  

- Unprecedentedly fast matrix computations by the new fourth-generation Tensor Cores, enabling a wider range of AI and HPC tasks
    
- Up to 9X faster AI training and up to 30X faster AI inference with its new Transformer Engine compared to the prior-generation NVIDIA A100
    
- Maximized quality of service for smaller workloads thanks to secure Multi-Instance GPU (MIG) partitioning the GPU into isolated, right-size instances
    

  

In a nutshell, if you need raw power, you've got it with the GH200. Unfortunately, it's so new that there hasn't really been much comprehensive benchmarking. But let's take a look anyway. 

### Benchmarks: Pitting the GH200 Against its Main Rivals

  

On the Linux benchmark site Phoronix, author Michael Larabel [published](https://www.phoronix.com/review/nvidia-gh200-gptshop-benchmark) some initial HPC benchmarks run on a GH200 workstation sourced from GPTshop.ai. Testing was done on the Grace CPU performance only.

  

The GH200 system in question was tested with 72 cores, a Quanta S74G motherboard, 480GB of RAM, and 960GB + 1920GB SAMSUNG SSD drives. (A complete description of specs and environment can be found on the Phoronix site at the above link.) Again, this benchmarking was merely preliminary, as it focused on CPU performance, and no power consumption numbers are available – but there were some interesting results nonetheless. 

  

According to the standard [HPCG](https://www.hpcg-benchmark.org/) memory bandwidth benchmark, the GH200 Grace CPU came in at a solid 41.7 GFLOPS. 

  

  ![](/uploads/OatdrZqcQK_Bp41PV_DBcUFPeSRnvYSJpF4O1uc1Q6D_dLyr4ESSPbeS5HzrYaK92ybZSAVi3ma0rKQ0FmlpXcgDBrhGdY06dP5C980mThA-hIlAexWMgn0yiXt3GXod1eAklyI8p1HdjCGyRpoEw-U.png)

Results for NVIDIA GH200 running HPCG benchmark. (Image credit: Phoronix)

5vFBmCRUcbfgb9PR7NpFoXpSeRSFqr47lr9V1YtOG-k2Aus1jPY1T217XD_EnBKKZKnn8MTFUs-Jx2lMYVR_8Ek4Q9eQcM-kml5l0C2B4wfzwdytunAxtEcGZNfi_pp-GQKy8jTPCrEpqRUtlrk8OXw

Another notable result came from the [NWChem](https://www.nwchem-sw.org/) benchmark, with the GH200 coming in second place at 1403.5 seconds.

  

![](/uploads/5vFBmCRUcbfgb9PR7NpFoXpSeRSFqr47lr9V1YtOG-k2Aus1jPY1T217XD_EnBKKZKnn8MTFUs-Jx2lMYVR_8Ek4Q9eQcM-kml5l0C2B4wfzwdytunAxtEcGZNfi_pp-GQKy8jTPCrEpqRUtlrk8OXw.png)

Results for NVIDIA GH200 running NWChem benchmark. (Image credit: Phoronix)

  

On a geometric mean basis across all the benchmarks, the GH200 Grace CPU performance was perfectly respectable overall. 

  
![](/uploads/O_VgErmTsszfyrnSV20nbY-BdZb4QvnVJe7LOp55l33JKKI3JFoQSRpWU2lia3g_QlqKTwi4_1VXjUQ2gE1iHKNYy_A4JkgtMsPnn5uzc7TBf9JjVzH1rifiNFA_n17d4TI_t7MB6bsjIMCMYGMcrXo.png)

(Image credit: Phoronix)

  

A set of GPU benchmarks run by researcher Simon Butcher [compared](https://blog.hpc.qmul.ac.uk/benchmarking-grace-hopper-nodes/) performance of the PyTorch ResNet50 [training recipes](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnet50v1.5) published by NVIDIA. The ImageNet 2012 dataset (150GB) was used, and training was run for 90 epochs, or about an hour. The GH200 showed excellent performance here.

  

![](/uploads/qlwGIbeeRNXXquUTSoF6EDJmoss443fwqWUEQg9PtgAVPwIHTNTbi0U5BsEFx3Wx2dX10b0gfTdlgMfQxU_kHulwTU5QaOGlz7Oy9U8GBpdkvj4pCE7wjW7PJzfU_lTK-mdd3VUlbSjUaXEx2RVtloA.png)

(Image credit: Simon Butcher, Queen Mary University of London)

  

NVIDIA also [published](https://resources.nvidia.com/en-us-dgx-gh200/nvidia-grace-hopper-superchip-datasheet) some performance comparisons that may be of interest.

  

![](/uploads/dXjLeW3nCbOSpjMxABHSZ6LP84CLuzszoNLvi2m9CD1Dxf9GgTYk17wr2YlxWxqNM5xF3U6GXV-E6JFILwTqmj4zEmdd9Rv5MNAUNK9xaa6YeTfM9krmcLeMRaUQ4eF8qt_G0-xW81JoRiaEz4157xQ.png)

(Image credit: NVIDIA)

  

### Final Thoughts

  

The NVIDIA GH200 Grace Hopper Superchip delivers the performance required for truly giant-scale AI and HPC applications running terabytes of data. Whether you're a scientist, an engineer, or a massive data center, you'll find it up to the task. 

As for what the future holds? Nvidia [recently announced](https://www.tomshardware.com/pc-components/gpus/nvidias-next-gen-ai-gpu-revealed-blackwell-b200-gpu-delivers-up-to-20-petaflops-of-compute-and-massive-improvements-over-hopper-h100) the GH200's successor: Grace Blackwell B200, the next-generation data center and AI GPU.

  

Here at Vast.ai, we're proud to say that we recently added ARM64 and GH200 support to our lineup. On our GPU rental platform, just about anyone can get access to top-tier GPUs at affordable rates for a variety of computational needs. You can skip the cost-prohibitive investment in hardware and get right to work today – we'd love to help you discover the possibilities!

NVIDIA's highly anticipated next-generation GH200 Grace Hopper Superchip platform is finally available. Perfectly suited for the era of accelerated computing and generative AI, it's essentially the world's most powerful GPU paired with the most versatile CPU.

Blog

Accessing the GPU marketplace with Vast.ai and dstack

Diverse GPU Options at Your Fingertips

Empowering Developers with dstack and Vast.ai

Effortless Setup and Deployment

Seamless Switching Between Cloud Providers

Streamline Your AI Projects with dstack and Vast.ai

Continue Reading: