
# Serving Online Inference with TGI on Vast.ai

## Background
TGI is an open source framework for Large Language model inference. It specifically focuses on throughput for serving, automatic batching, and ease of use with the Huggingface Ecosystem. 

TGI provides an OpenAI compatible server, which means that you can integrate it into chatbots, and other applications

As companies build out their AI products, they often hit roadblocks like rate limits and cost for using these models. With TGI on Vast, you can run your own models in the form factor you need, but with much more affordable compute. As inference grows in demand with agents and complicated workflows, using Vast is great for performance and affordability where you need it the most.


This guide will show you how to setup TGI to serve an LLM on Vast. We reference a notebook that you can use [here](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/EdjnXy/7992b62a4e4472388e7414ea80484b7a8b57427d/files/serving_tgi_on_vast.json). It is based upon our guide and notebook for Serving `vLLM`, so that users can see the small differences they need to handle if they care to switch.


## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving an LLM, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU with decent RAM since we're going to serve a single small model. `TGI` also requires Cuda version 12.1 or higher, so we will filter for that as well. We'll query the vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.1' 
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 

We also told TGI to serve on port 8000 as that is the default port for the OpenAI SDK, but TGI would normally serve on port 8080 instead.

```bash 
vastai create instance <instance-id> --image ghcr.io/huggingface/text-generation-inference:1.4 --env '-p 8000:8000' --disk 40 --args --port 8000 --model-id stabilityai/stablelm-2-zephyr-1_6b
```


## Connecting and Testing:
To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view_tgi.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_tgi.png)

We will copy over the IP address and the port into the cell below.
```bash
# This request assumes you haven't changed the model. If you did, fill it in the "model" value in the payload json below
curl -X POST http://<IP-Address>:<Port>/v1/completions -H "Content-Type: application/json"  -d '{"model" : "stabilityai/stablelm-2-zephyr-1_6b", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

TGI is an open source framework for Large Language model inference....

Serving Online Inference with TGI on Vast.ai | June 2024


# Security and Compliance at Vast.ai

At Vast.ai, your security is our priority. We're proud of our track record of excellence over the past six years serving clients worldwide while keeping up the highest standards of regulatory compliance.

In fact, here at Vast.ai, we're currently in the process of completing our SOC 2 Type 1 certification – further solidifying our commitment to data security and regulatory compliance.

Our [Compliance Policy](https://vast.ai/compliance) is designed to protect your data every step of the way. Here's how:

## Strengthening Security with Datacenter Partners

On our GPU cloud platform, our top-tier datacenter providers set the standard. These aren't your average hosts. We partner exclusively with vetted datacenter providers who maintain rigorous third-party compliance certifications. They hold a minimum of ISO 27001 certification – and many go further, maintaining compliance with industry-specific regulations in areas like healthcare, finance, and government.

You can trust that any datacenter provider you work with through Vast has undergone extensive due diligence. We audit their compliance regularly, ensuring that everything from physical access controls to continuous monitoring systems are in place. If you need documentation, we can provide the relevant certifications directly to your compliance team.

> To filter for these providers only on Vast, be sure to select the **"Secure Cloud (Only Trusted Datacenters)"** checkbox.

## Key Security Features of Our Datacenter Partners

To ensure the highest level of security for your data, our trusted datacenter partners implement a comprehensive set of protective measures. Here are the key features that set them apart:

### Third-Party Certifications

All of our datacenter partners are ISO 27001-certified, ensuring they meet globally recognized information security standards. They may also hold additional certifications, such as HIPAA, NIST, PCI, and HITRUST, as well as GDPR compliance.

### Physical and Environmental Security

Our partners manage their equipment in secure facilities with restricted physical access, regular access reviews and recertifications, continuous video monitoring with 90+ day retention, and stringent asset tracking and removal approval processes. Environmental controls, including fire suppression systems and power backups, are tested regularly to prevent disruptions.

### Enhanced Auditing

We conduct thorough audits of our datacenter partners, verifying their ownership structure, source of funds, and identity, while ensuring they adhere to best practices and maintain equipment in line with their certifications.

### Extended Legal Agreements

All datacenter hosts sign expanded hosting agreements that include a separate Data Processing Agreement (DPA) for added protection.

Other compliance measures include robust incident response protocols and regular security and compliance training for all employees.

## Securing Your Peace of Mind

Compliance can be complex and daunting – but we're here to help!

Whether you're working in healthcare, finance, or other sectors with stringent data protection requirements, we offer a level of security that you can trust for your most sensitive workloads. With a six-year track record of no major incidents, Vast.ai remains a trusted choice for Fortune 500 companies and individuals alike.

If you have questions about how Vast.ai's system fits within your compliance policy framework, or if you'd like more information about our vetted datacenter providers, we’d love to talk. Feel free to schedule a consultation using [this Calendly link](https://calendly.com/travis-_da/compliance?month=2024-10), or reach out to us at [compliance@vast.ai](mailto:compliance@vast.ai). Our team is ready to help you navigate your security and compliance needs with confidence.


At Vast.ai, your security is our priority. We're proud of our track record of excellence over the past six years serving clients worldwide while keeping up the highest standards of...

Security and Compliance at Vast AI


# Serving Online Inference with TGI and Medusa on Vast.ai

## Background

Medusa is a method of specular decoding. Speculative decoding speeds up inference of large language models by having a smaller model multiple tokens and lets the larger model just verify. If the Verification for the large model is cheaper than generating the tokens themselves. If the smaller model is accurate enough, then the cost to generate tokens goes down overall.

Medusa is slightly different than other types of speculative decoding in that it adds a piece of the original model to do the speculation.

TGI is the first major serving framework for large language models that enables Medusa-style speculative decoding.

Speculative decoding comes with a trade-off between adding extra memory used and generation speed. Speculative decoding, including Medusa, requires more overall memory. But if your "draft" model is good enough, then it can provide an excellent speed up opportunity.

Compared to other forms of speculative decoding, Medusa requires a fine-tuned adapter on top of the original model. There exist some of these already that we can use. Be on the lookout for more posts on how to create these adapters from us.

To set up TGI with Medusa on Vast, you can follow the steps outlined in the provided notebook. The notebook will guide you through the process of configuring TGI to serve Medusa and optimizing the setup for your specific use case.

## Setup and Querying

First, set up your environment and Vast API key:

```bash
pip install --upgrade vastai
```

Once you create your account, go [here](https://cloud.vast.ai/cli/) to find your API key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving a language model, we're looking for a machine with a static IP address, available ports to host on, and a single modern GPU with decent RAM, as we'll serve a single small model. TGI also requires CUDA version 12.1.1 or higher, so we'll filter for that as well. We will query the Vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.1.1'
```

## Deploying the Image

The easiest way to deploy this instance is to use the command line. Copy and paste a specific instance ID you choose from the list above into `instance-id` below. This medusa model also needs to download gemma-7b-it, which is a gated model. You'll need to accept the terms on gemma-7b-it and pass in your Huggingface API Token into `HF_TOKEN=<Your_Huggingface_Token>`
.

```bash
vastai create instance <instance-id> --image ghcr.io/huggingface/text-generation-inference:latest --env '-p 8000:8000 -e HF_TOKEN={"<Huggingface-Token>"}' --disk 40 --args --port 8000 --model-id text-generation-inference/gemma-7b-it-medusa --speculate 2
```

## Connecting and Testing

To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view_medusa_tgi.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_medusa_tgi.png)

We will copy over the IP address and the port into the cell below.

```bash
# This request assumes you haven't changed the model. If you did, fill it in the "model" value in the payload json below
curl -X POST http://<IP-Address>:<Port>/v1/completions -H "Content-Type: application/json"  -d '{"model" : "text-generation-inference/gemma-7b-it-medusa", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

In the [notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/q7gnzg/30783bfb0b148eaf67a02f083b9cf579be87c5fc/files/serving_medusa_on_vast.json), we include ways to call this model with requests or OpenAI

### Other things to look out for with other configurations

If you are downloading a model that needs authentication from the Hugging Face Hub, passing `-e HF_token=<Your-Read-Only-Token>` within Vast's `--env` variable string should help.

Sometimes the full context of a model can't be used given the space allocated for TGI on the GPU + the model's size. In those cases, you might want to increase `--gpu-memory-utilization`, or decrease the `max-model-len`. Increasing `--gpu-memory-utilization` does come with CUDA OutOfMemory issues that can be hard to predict ahead of time.

We won't need either of these for this specific model and GPU configuration.

### Testing

Copy the IP address from your instance once it is ready, and then we can use the following code to call it. Note that while your server might have ports ready, the model might not have downloaded yet as it is much larger this time. You can check the status of this via the logs to see when it has started serving.

```bash
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'text-generation-inference/gemma-7b-it-medusa',
    'prompt': 'Hello, how are you?',
    'max_tokens': 50,
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/completions', headers=headers, json=json_data)
print(response.content)
```

Or use OpenAI:

```bash
pip install openai
```

```python
from openai import OpenAIH

# Modify OpenAI's API key and API base to use TGI's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
completion = client.completions.create(model="text-generation-inference/gemma-7b-it-medusa",
                                      prompt="Hello, how are you?",
                                      max_tokens=50)
print("Completion result:", completion)
```

## Conclusions

Medusa, when paired with TGI on Vast, offers a compelling solution for engineering teams looking to optimize their AI inference costs and improve shipping velocity. The combination of Medusa's faster inference capabilities and TGI's state-of-the-art throughput enables better user experiences and reduced GPU time for serving users and processing data.

By leveraging Medusa's speed advantages, you can achieve higher throughput on GPUs, allowing you to handle more requests simultaneously and deliver faster responses to users. This is particularly valuable in scenarios where low latency and real-time interactions are crucial, such as in chatbots or virtual assistants.

Moreover, the increased throughput provided by Medusa and TGI translates to more efficient utilization of GPU resources. With faster inference times You spend less money for batch workloads and need less GPUs to handle traffic surges. Overall, you can scale your AI applications more effectively.

Vast's affordable compute options further enhance the benefits of using Medusa with TGI. By running Medusa on Vast's infrastructure, you can access cost-effective GPU resources that align with your budget and performance requirements. This allows you to maximize your margins while still delivering high-quality AI experiences to your users.

By leveraging the strengths of Medusa Decoding and the affordability of Vast, you can build cutting-edge AI applications that scale seamlessly and deliver exceptional results.


Medusa is a method of specular decoding. Speculative decoding speeds up inference of large language models...

Serving Online Inference with TGI and Medusa on Vast.ai


# Serving Online Inference with Text-Embeddings-Inference on Vast.ai

## Background
Text-Embeddings-Inference is an open source framework for embedding and reranker inference. It is made by Hugging Face and is focused on low latency, high throughput serving with short boot times due to a small docker image.

Embeddings and Re-rankers are often very important for many use cases like RAG, Information Retrieval, Search, and dataset cleaning + exploration. 

This guide will show you how to setup `Text-Embeddings-Inference` to serve an embedding model for online inference on Vast. We reference a notebook that you can use [here](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/Bzkbbr/3cb18e2bca4e49129125a322bafb38346abf6abd/files/serving_text_embeddings_inference_on_vast.json).

## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU since we're going to serve a single small model. `Text-Embeddings-Inference` also requires Cuda version 12.2 or higher, so we will filter for that as well. We'll query the Vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap >= 800 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.2' 
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 

We also told `Text-Embeddings-Inference` to serve on port 8000 as that is the default port for the OpenAI SDK, but it would normally serve on port 80 instead. Feel free to change this as you like

```bash 
vastai create instance <instance-id> --image ghcr.io/huggingface/text-embeddings-inference --env '-p 8000:80' --disk 16 --args --model-id jinaai/jina-embeddings-v2-base-en
```

### Compute Architecture and Server Image Selection:
`Text-Embeddings-Inference` does offer support for multiple different GPU Types from Nvidia. However it is often best to match up the hardware you're using with the specific image to increase performance and reliability. They give you the recommended image for each GPU [here](https://github.com/huggingface/text-embeddings-inference?tab=readme-ov-file#docker-images)

## Connecting and Testing:
To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view_tei.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_tei.png)

We will copy over the IP address and the port into the cell below.
```bash
curl <Instance-IP-Address>:<Port>/embed -X POST -d '{"inputs":"What is Deep Learning?"}' -H 'Content-Type: application/json'
```

You will see a response from your model in the output, which will be the embedding of your query. Your model is up and running on Vast!

## Connecting with the OpenAI SDK
The OpenAI SDK also allows for generating embeddings. We can use it as well with the following code.
```bash
pip install openai
```
```
from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
model = "jinaai/jina-embeddings-v2-base-en"
embeddings = client.embeddings.create(model=model, input="What is Deep Learning?")
print(embeddings)
```

This is great, as it helps us integrate our embeddings into applications that use the OpenAI SDK.

## Next Steps:

Embeddings and Re-Ranking are fundamental building blocks to many different ways of building AI applications. Stay Tuned as we cover these topics more in depth!

Text-Embeddings-Inference is an open source framework for embedding and reranker inference...

Blog

Serving Online Inference with TGI on Vast.ai | June 2024

Serving Online Inference with TGI on Vast.ai

Background

Setup and Querying

Deploying the Image:

Connecting and Testing:

Continue Reading: