
We're back with another monthly update for our customers old and new! As we continue to redefine norms in cloud GPU rental, we're committed to continuously refining our platform.

This month, we've revamped the billing page! We also fixed an issue with earnings payouts showing incorrectly and added some clarity and warnings about auto-billing to the new billing page.

Plus, we've expanded our list of GPU types to include the H100 NVL. Designed exclusively for large language model (LLM) deployment, the NVIDIA H100 NVL is a powerhouse for high-performance computing. It's our hope that hosts and renters alike can benefit from this new addition that broadens the options on Vast.

No service is immune to hiccups, but we prioritize swift action to tackle any bugs that arise – so you can depend on a seamless customer experience. Here are some of the latest bug fixes we've implemented since our last update:

- Fixed PDF export issues
- Recommended templates now show properly
- Improved error notifications
- Fixed SSH validation issues
- Fixed machine earnings discrepancy for hosts
- Hosts now receive correct portion of early termination penalty for reserved instances

## Cloud GPU Rental Platform Updates

Here at Vast.ai, our platform updates go beyond just routine maintenance. As mentioned, we have a new billing page and added the H100 NVL as a new GPU type – and the following are a few more updates from this month:

- New auth flow and consistent visuals
- New CLI launch_instance combines search/create command
- Autoscaler/serverless docs have been updated
- Slight fixes to threshold and newsletter

For the full list of all improvements from this month, check out the change log at the bottom of this page.

If you have any questions or need assistance with navigating our platform, don't hesitate to reach out. Our team is always ready to provide support and ensure you get the most out of your GPU rental experience. Feel free to contact us through the chatbox at the bottom right, email us at support@vast.ai, or join our dynamic Discord community for additional insights and resources. We're here to help you every step of the way!

## Change Log

### Features:
- New billing page
- New auth flow and consistent visuals
- New CLI launch_instance combines search/create command
- New machine page and bulk edit
- Added automatic soft blacklisting to search based on user's personal report machine history
- Search backend now supports continent options like front end
- CLI/API logs command now supports a grep option

### Bug Fixes:
- Refunded payments now display properly
- Fixed PDF export issues
- Recommended templates now show properly
- Improved error notifications
- Fixed SSH validation issues
- Fixed machine earnings discrepancy for hosts
- Hosts now receive correct portion of early termination penalty for reserved instances
- Fixed issue with earnings payouts showing incorrectly on billing page
- Added auto-billing clarity & warnings to new billing page
- Warning when copying a public template
- Transfers shown properly on billing page
- Autobilling threshold fixed for team accounts
- GPU occupancy now shows for unlisted machines
- Gpu option search query filter works
- Open button for auth instances
- Reserved pricing coherent display
- Billing page properly shows negative balance
- Billing page properly shows failed payments
- Cloud rsync creates dest drives instead of failing
- Fixed env parsing bug in create_instance using template_hash
- Added autoscaler tag option for templates
- Misc UX/UI Improvements

### Miscellaneous Tasks:
- Added H100 NVL as a new GPU type
- Updated Autoscaler/serverless docs
- Updated billing text & warnings
- Slight fixes to threshold and newsletter


This month's update includes a revamped billing page, the addition of the H100 NVL GPU, and various bug fixes and improvements to enhance your experience on Vast.ai.

July 2024 Monthly Update: Billing Page Revamp, H100 NVL GPU Addition, and More!


# Serving Online Inference with vLLM on Vast.ai

## Background

vLLM is an open source framework for Large Language model inference. It specifically focuses on throughput for serving and batch workloads. This is important for building apps for multiple users and at scale.

vLLM provides an OpenAI compatible server, which means that you can integrate it into chatbots, and other applications

As companies build out their AI products, they often hit roadblocks like rate limits and cost for using these models. With vLLM on Vast, you can run your own models in the form factor you need, but with much more affordable compute. As inference grows in demand with agents and complicated workflows, vLLM on Vast shines for performance and affordability where you need it the most.

This guide will show you how to setup vLLM to serve an LLM on Vast. We reference a notebook that you can use [here](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/XEyMo8/72acf45925f8230b75195b7da9fff1884fa4052d/files/serving_vllm_on_vast.json)

## Setup and Querying

First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving an LLM, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU with decent RAM since we're going to serve a single small model. `vLLM` also requires Cuda version 12.4 or higher, so we will filter for that as well. We will query the vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.4'
```

## Deploying the Image:

The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below.

```bash
vastai create instance <instance-id> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 40 --args --model stabilityai/stablelm-2-zephyr-1_6b
```

## Connecting and Testing:

To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view_vllm.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_vllm.png)

We will copy over the IP address and the port into the cell below.

```bash
# This request assumes you haven't changed the model. If you did, fill it in the "model" value in the payload json below
curl -X POST http://<IP-Address>:<Port>/v1/completions -H "Content-Type: application/json"  -d '{"model" : "stabilityai/stablelm-2-zephyr-1_6b", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

In the [notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/XEyMo8/72acf45925f8230b75195b7da9fff1884fa4052d/files/serving_vllm_on_vast.json), we include ways to call this model with requests or OpenAI

## Advanced Usage: Serving a Quantized Llama-3-70b Model:

Now that we've spun up a model on vLLM, we can get into more complicated deployments. We'll work on serving this specific quantized Llama-3 70B [model](https://huggingface.co/casperhansen/llama-3-70b-instruct-awq).

With this quantized model, we can easilly serve this model on on 4 4090 GPU's.

### What is different this time around:

1. The model string - we need to use the new model id.
2. We're going to use 4 GPU's instead of just 1.
3. We need to provision much more space on our system to be able to download the full set of weights. 100 GB in this case should be fine.
4. We need to set up tensor parallelism inside vLLM to split up the model across these 4 gpus.
5. We need to let vLLM know that this is a quantized model

First, we will search for instances that match our needs

```bash
vastai search offers 'compute_cap >= 800 gpu_ram >= 24 num_gpus = 4 static_ip=true direct_port_count > 1 cuda_vers >= 12.4'
```

In our instance creation, we will increase our disk usage to 100GB.

Then, we will tell vllm to: 1. use the specific model, 2. split across 4 GPU's, and 3. Let it know that it is in fact a quantized model.

```bash
vastai create instance <Instance-ID> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 100 --args --model casperhansen/llama-3-70b-instruct-awq --tensor-parallel-size 4  --quantization awq
```

### Other things to look out for with other configurations:

If you are downloading a model that needs authentication from the huggingface hub, passing `-e HF_token=<Your-Read-Only-Token>` within vast's `--env` variable string should help.

Sometimes the full context of a model can't be used given the space allocated for vLLM on the GPU + the models size. In those cases, you might want to increase `--gpu-memory-utilization`, or decrease the `max-model-len`. Increasing `--gpu-memory-utilization` does come with CUDA OutOfMemory Issues that can be hard to predict ahead of time.

We won't need either of these for this specific model and GPU configuration.

### Testing:

Copy the IP address from your instance once it is ready, and then we can use the following code to call it. Note that while your server might have ports ready, the model might not have downloaded yet as it is much larger this time. You can check the status of this via the logs to see when it has started serving.

```bash
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'casperhansen/llama-3-70b-instruct-awq',
    'prompt': 'Hello, how are you?',
    'max_tokens': 50,
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/completions', headers=headers, json=json_data)
print(response.content)
```

Or use Open AI:

```bash
pip install openai
```

```python
from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
completion = client.completions.create(model="casperhansen/llama-3-70b-instruct-awq",
                                      prompt="Hello, how are you?",
                                      max_tokens=50)
print("Completion result:", completion)
```

## Conclusions

Model inference is expensive, and leveraging more affordable compute/models makes a huge difference for engineering teams in terms of margins, and shipping velocity.

Using vLLM on Vast is perfect for this, pairing Vast's access to affordable compute with the simplicity and State of the Art throughput of the vLLM backend.

vLLM is a great beginning to building Generative AI Apps. We will continue to explore using this tool more with Vast in future posts.

Llama-3 is already ready to go on Vast to start experimenting and building!


Leverage the power of vLLM for efficient and scalable language model inference on Vast.ai's affordable compute infrastructure.

Cloud Computing

July 2024 Monthly Update: Billing Page Revamp, H100 NVL GPU Addition, and More!

Serving Online Inference with vLLM API on Vast.ai