
# Security and Compliance at Vast.ai

At Vast.ai, your security is our priority. We're proud of our track record of excellence over the past six years serving clients worldwide while keeping up the highest standards of regulatory compliance.

In fact, here at Vast.ai, we're currently in the process of completing our SOC 2 Type 1 certification – further solidifying our commitment to data security and regulatory compliance.

Our [Compliance Policy](https://vast.ai/compliance) is designed to protect your data every step of the way. Here's how:

## Strengthening Security with Datacenter Partners

On our GPU cloud platform, our top-tier datacenter providers set the standard. These aren't your average hosts. We partner exclusively with vetted datacenter providers who maintain rigorous third-party compliance certifications. They hold a minimum of ISO 27001 certification – and many go further, maintaining compliance with industry-specific regulations in areas like healthcare, finance, and government.

You can trust that any datacenter provider you work with through Vast has undergone extensive due diligence. We audit their compliance regularly, ensuring that everything from physical access controls to continuous monitoring systems are in place. If you need documentation, we can provide the relevant certifications directly to your compliance team.

> To filter for these providers only on Vast, be sure to select the **"Secure Cloud (Only Trusted Datacenters)"** checkbox.

## Key Security Features of Our Datacenter Partners

To ensure the highest level of security for your data, our trusted datacenter partners implement a comprehensive set of protective measures. Here are the key features that set them apart:

### Third-Party Certifications

All of our datacenter partners are ISO 27001-certified, ensuring they meet globally recognized information security standards. They may also hold additional certifications, such as HIPAA, NIST, PCI, and HITRUST, as well as GDPR compliance.

### Physical and Environmental Security

Our partners manage their equipment in secure facilities with restricted physical access, regular access reviews and recertifications, continuous video monitoring with 90+ day retention, and stringent asset tracking and removal approval processes. Environmental controls, including fire suppression systems and power backups, are tested regularly to prevent disruptions.

### Enhanced Auditing

We conduct thorough audits of our datacenter partners, verifying their ownership structure, source of funds, and identity, while ensuring they adhere to best practices and maintain equipment in line with their certifications.

### Extended Legal Agreements

All datacenter hosts sign expanded hosting agreements that include a separate Data Processing Agreement (DPA) for added protection.

Other compliance measures include robust incident response protocols and regular security and compliance training for all employees.

## Securing Your Peace of Mind

Compliance can be complex and daunting – but we're here to help!

Whether you're working in healthcare, finance, or other sectors with stringent data protection requirements, we offer a level of security that you can trust for your most sensitive workloads. With a six-year track record of no major incidents, Vast.ai remains a trusted choice for Fortune 500 companies and individuals alike.

If you have questions about how Vast.ai's system fits within your compliance policy framework, or if you'd like more information about our vetted datacenter providers, we’d love to talk. Feel free to schedule a consultation using [this Calendly link](https://calendly.com/travis-_da/compliance?month=2024-10), or reach out to us at [compliance@vast.ai](mailto:compliance@vast.ai). Our team is ready to help you navigate your security and compliance needs with confidence.


At Vast.ai, your security is our priority. We're proud of our track record of excellence over the past six years serving clients worldwide while keeping up the highest standards of...

Security and Compliance at Vast AI


# Serving Online Inference with TGI and Medusa on Vast.ai

## Background

Medusa is a method of specular decoding. Speculative decoding speeds up inference of large language models by having a smaller model multiple tokens and lets the larger model just verify. If the Verification for the large model is cheaper than generating the tokens themselves. If the smaller model is accurate enough, then the cost to generate tokens goes down overall.

Medusa is slightly different than other types of speculative decoding in that it adds a piece of the original model to do the speculation.

TGI is the first major serving framework for large language models that enables Medusa-style speculative decoding.

Speculative decoding comes with a trade-off between adding extra memory used and generation speed. Speculative decoding, including Medusa, requires more overall memory. But if your "draft" model is good enough, then it can provide an excellent speed up opportunity.

Compared to other forms of speculative decoding, Medusa requires a fine-tuned adapter on top of the original model. There exist some of these already that we can use. Be on the lookout for more posts on how to create these adapters from us.

To set up TGI with Medusa on Vast, you can follow the steps outlined in the provided notebook. The notebook will guide you through the process of configuring TGI to serve Medusa and optimizing the setup for your specific use case.

## Setup and Querying

First, set up your environment and Vast API key:

```bash
pip install --upgrade vastai
```

Once you create your account, go [here](https://cloud.vast.ai/cli/) to find your API key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving a language model, we're looking for a machine with a static IP address, available ports to host on, and a single modern GPU with decent RAM, as we'll serve a single small model. TGI also requires CUDA version 12.1.1 or higher, so we'll filter for that as well. We will query the Vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.1.1'
```

## Deploying the Image

The easiest way to deploy this instance is to use the command line. Copy and paste a specific instance ID you choose from the list above into `instance-id` below. This medusa model also needs to download gemma-7b-it, which is a gated model. You'll need to accept the terms on gemma-7b-it and pass in your Huggingface API Token into `HF_TOKEN=<Your_Huggingface_Token>`
.

```bash
vastai create instance <instance-id> --image ghcr.io/huggingface/text-generation-inference:latest --env '-p 8000:8000 -e HF_TOKEN={"<Huggingface-Token>"}' --disk 40 --args --port 8000 --model-id text-generation-inference/gemma-7b-it-medusa --speculate 2
```

## Connecting and Testing

To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view_medusa_tgi.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_medusa_tgi.png)

We will copy over the IP address and the port into the cell below.

```bash
# This request assumes you haven't changed the model. If you did, fill it in the "model" value in the payload json below
curl -X POST http://<IP-Address>:<Port>/v1/completions -H "Content-Type: application/json"  -d '{"model" : "text-generation-inference/gemma-7b-it-medusa", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

In the [notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/q7gnzg/30783bfb0b148eaf67a02f083b9cf579be87c5fc/files/serving_medusa_on_vast.json), we include ways to call this model with requests or OpenAI

### Other things to look out for with other configurations

If you are downloading a model that needs authentication from the Hugging Face Hub, passing `-e HF_token=<Your-Read-Only-Token>` within Vast's `--env` variable string should help.

Sometimes the full context of a model can't be used given the space allocated for TGI on the GPU + the model's size. In those cases, you might want to increase `--gpu-memory-utilization`, or decrease the `max-model-len`. Increasing `--gpu-memory-utilization` does come with CUDA OutOfMemory issues that can be hard to predict ahead of time.

We won't need either of these for this specific model and GPU configuration.

### Testing

Copy the IP address from your instance once it is ready, and then we can use the following code to call it. Note that while your server might have ports ready, the model might not have downloaded yet as it is much larger this time. You can check the status of this via the logs to see when it has started serving.

```bash
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'text-generation-inference/gemma-7b-it-medusa',
    'prompt': 'Hello, how are you?',
    'max_tokens': 50,
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/completions', headers=headers, json=json_data)
print(response.content)
```

Or use OpenAI:

```bash
pip install openai
```

```python
from openai import OpenAIH

# Modify OpenAI's API key and API base to use TGI's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
completion = client.completions.create(model="text-generation-inference/gemma-7b-it-medusa",
                                      prompt="Hello, how are you?",
                                      max_tokens=50)
print("Completion result:", completion)
```

## Conclusions

Medusa, when paired with TGI on Vast, offers a compelling solution for engineering teams looking to optimize their AI inference costs and improve shipping velocity. The combination of Medusa's faster inference capabilities and TGI's state-of-the-art throughput enables better user experiences and reduced GPU time for serving users and processing data.

By leveraging Medusa's speed advantages, you can achieve higher throughput on GPUs, allowing you to handle more requests simultaneously and deliver faster responses to users. This is particularly valuable in scenarios where low latency and real-time interactions are crucial, such as in chatbots or virtual assistants.

Moreover, the increased throughput provided by Medusa and TGI translates to more efficient utilization of GPU resources. With faster inference times You spend less money for batch workloads and need less GPUs to handle traffic surges. Overall, you can scale your AI applications more effectively.

Vast's affordable compute options further enhance the benefits of using Medusa with TGI. By running Medusa on Vast's infrastructure, you can access cost-effective GPU resources that align with your budget and performance requirements. This allows you to maximize your margins while still delivering high-quality AI experiences to your users.

By leveraging the strengths of Medusa Decoding and the affordability of Vast, you can build cutting-edge AI applications that scale seamlessly and deliver exceptional results.


Medusa is a method of specular decoding. Speculative decoding speeds up inference of large language models...

Serving Online Inference with TGI and Medusa on Vast.ai


# Serving Online Inference with Text-Embeddings-Inference on Vast.ai

## Background
Text-Embeddings-Inference is an open source framework for embedding and reranker inference. It is made by Hugging Face and is focused on low latency, high throughput serving with short boot times due to a small docker image.

Embeddings and Re-rankers are often very important for many use cases like RAG, Information Retrieval, Search, and dataset cleaning + exploration. 

This guide will show you how to setup `Text-Embeddings-Inference` to serve an embedding model for online inference on Vast. We reference a notebook that you can use [here](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/Bzkbbr/3cb18e2bca4e49129125a322bafb38346abf6abd/files/serving_text_embeddings_inference_on_vast.json).

## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU since we're going to serve a single small model. `Text-Embeddings-Inference` also requires Cuda version 12.2 or higher, so we will filter for that as well. We'll query the Vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap >= 800 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.2' 
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 

We also told `Text-Embeddings-Inference` to serve on port 8000 as that is the default port for the OpenAI SDK, but it would normally serve on port 80 instead. Feel free to change this as you like

```bash 
vastai create instance <instance-id> --image ghcr.io/huggingface/text-embeddings-inference --env '-p 8000:80' --disk 16 --args --model-id jinaai/jina-embeddings-v2-base-en
```

### Compute Architecture and Server Image Selection:
`Text-Embeddings-Inference` does offer support for multiple different GPU Types from Nvidia. However it is often best to match up the hardware you're using with the specific image to increase performance and reliability. They give you the recommended image for each GPU [here](https://github.com/huggingface/text-embeddings-inference?tab=readme-ov-file#docker-images)

## Connecting and Testing:
To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view_tei.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_tei.png)

We will copy over the IP address and the port into the cell below.
```bash
curl <Instance-IP-Address>:<Port>/embed -X POST -d '{"inputs":"What is Deep Learning?"}' -H 'Content-Type: application/json'
```

You will see a response from your model in the output, which will be the embedding of your query. Your model is up and running on Vast!

## Connecting with the OpenAI SDK
The OpenAI SDK also allows for generating embeddings. We can use it as well with the following code.
```bash
pip install openai
```
```
from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
model = "jinaai/jina-embeddings-v2-base-en"
embeddings = client.embeddings.create(model=model, input="What is Deep Learning?")
print(embeddings)
```

This is great, as it helps us integrate our embeddings into applications that use the OpenAI SDK.

## Next Steps:

Embeddings and Re-Ranking are fundamental building blocks to many different ways of building AI applications. Stay Tuned as we cover these topics more in depth!

Text-Embeddings-Inference is an open source framework for embedding and reranker inference...

Serving Online Inference with Text-Embeddings-Inference on Vast.ai | June 2024


# Serving Online Inference with TGI on Vast.ai

## Background
TGI is an open source framework for Large Language model inference. It specifically focuses on throughput for serving, automatic batching, and ease of use with the Huggingface Ecosystem. 

TGI provides an OpenAI compatible server, which means that you can integrate it into chatbots, and other applications

As companies build out their AI products, they often hit roadblocks like rate limits and cost for using these models. With TGI on Vast, you can run your own models in the form factor you need, but with much more affordable compute. As inference grows in demand with agents and complicated workflows, using Vast is great for performance and affordability where you need it the most.


This guide will show you how to setup TGI to serve an LLM on Vast. We reference a notebook that you can use [here](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/EdjnXy/7992b62a4e4472388e7414ea80484b7a8b57427d/files/serving_tgi_on_vast.json). It is based upon our guide and notebook for Serving `vLLM`, so that users can see the small differences they need to handle if they care to switch.


## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving an LLM, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU with decent RAM since we're going to serve a single small model. `TGI` also requires Cuda version 12.1 or higher, so we will filter for that as well. We'll query the vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.1' 
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 

We also told TGI to serve on port 8000 as that is the default port for the OpenAI SDK, but TGI would normally serve on port 8080 instead.

```bash 
vastai create instance <instance-id> --image ghcr.io/huggingface/text-generation-inference:1.4 --env '-p 8000:8000' --disk 40 --args --port 8000 --model-id stabilityai/stablelm-2-zephyr-1_6b
```


## Connecting and Testing:
To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view_tgi.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_tgi.png)

We will copy over the IP address and the port into the cell below.
```bash
# This request assumes you haven't changed the model. If you did, fill it in the "model" value in the payload json below
curl -X POST http://<IP-Address>:<Port>/v1/completions -H "Content-Type: application/json"  -d '{"model" : "stabilityai/stablelm-2-zephyr-1_6b", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

TGI is an open source framework for Large Language model inference....

Serving Online Inference with TGI on Vast.ai | June 2024


# Maximizing Value with NVIDIA A40 & RTX A6000

For many people and organizations, the cost of high-end hardware is prohibitive when it comes to tasks like fine-tuning large language models (LLMs) and other AI workloads. It may not make sense to purchase a super-powered machine like the NVIDIA A100 or H100 when a more affordable option exists and can get the job done just about the same.

For instance, the NVIDIA A40 and RTX A6000 GPUs are incredibly attractive options for the more budget-conscious user – at least when compared to such expensive higher-end machines! Not only do they balance performance and cost, but they're also far more readily available than the A100 and H100 and can scale AI projects quickly.

### NVIDIA A40 & RTX A6000: Similarities and Differences

The A40 and A6000 are both professional-grade GPUs, well suited for high-performance computing. The A40 is intended for server environments and data centers while the A6000 is designed for desktop workstations, but otherwise they're quite similar with just a few minor differences.

Both GPUs are Ampere architecture-based with a PCIe Gen 4.0 interface and have 48GB of GDDR6 RAM including error-correction code (ECC). However, the A40 offers 696 GB/s of peak memory bandwidth while the A6000 provides a touch more at 768 GB/s of bandwidth – as well as a slightly higher clock speed.

The two GPUs are tailor-made to handle demanding, large-scale AI workloads, with each featuring 10,752 CUDA cores (shading units), 84 second-gen RT cores, and 336 third-gen Tensor cores. Both include hardware support for a fine-grained structured sparsity feature, which can be used to accelerate inference and other deep learning workloads.

The A40 is passively cooled and employs bidirectional airflow, allowing air to move in either direction through the heatsink. This makes it better suited for use in servers. The A6000, on the other hand, has active cooling. Both GPUs use up quite a bit of power with a maximum power consumption of 300 watts.

The A40 has three display outputs, although they are disabled by default since the GPU is configured out of the box to support virtual graphics and compute workloads in virtualized environments. This makes it highly suitable for cloud-based applications and services, as it can easily deliver high-performance graphics and compute capabilities to remote users. The A6000 has four display ports that are enabled by default but are not active when using virtual GPU software.

Unlike the NVIDIA A100 and H100, the A40 and A6000 do not support Multi-Instance GPU (MIG), so it's not possible to run separate and fault-isolated workloads in parallel on the same physical GPU. However, their memory can be expanded by integrating a second GPU using NVLink technology, which allows the two to pool resources and operate as one unit with reduced latency and a combined memory of 96GB – plenty for most uses!

### Benefits and Uses

Because the A40 and A6000 GPUs are so well suited to cloud environments and are more readily available than higher-end hardware, they allow organizations to scale their AI initiatives in a cost-effective and operationally efficient manner. And now, with the introduction of 10x GPU servers, the A40 and A6000 can be deployed in extremely powerful configurations to undertake AI projects that are even more ambitious and require significant computational resources.

According to some benchmarks, the A6000 can run about 10% faster than the A40 overall, due to its higher clock speed and memory bandwidth. But this advantage must be evaluated against the features of the A40, such as its secure and measured boot with hardware root of trust; NEBS Level 3 compliance (making it ideal for use in a wide array of network and telecom applications where stability and reliability are critical); and superior suitability to server environments.

Based on specs and performance, some examples of appropriate uses for the A40 and A6000 GPUs include:

- **AI and Deep Learning Workflows** – Training sophisticated neural networks, fine-tuning LLMs, running AI inference at scale, and deploying AI applications across various sectors like healthcare and finance.
- **Scientific Research and Engineering Simulations** – Running detailed simulations, modeling, data analysis, and computer-aided engineering (CAE) tasks in areas like climate study, bioinformatics, and the automotive, aerospace, and manufacturing industries.
- **Advanced Visualization** – Performing tasks where rapid rendering and visual fidelity are paramount, such as professional content creation and graphic design, virtual production, broadcast-grade streaming, real-time visual effects, and animation for film and game studios.

### The Bottom Line

Ultimately, the NVIDIA A40 and RTX A6000 are an excellent choice for organizations and professionals who may not want to pay a supremely high price for the NVIDIA A100 or H100 – or who are happy to make the trade-off between faster processing times and lower cost plus availability – while still enjoying the ability to tackle the largest workloads in AI, visual computing, and data science.

All of that being said, the A40 and A6000 do still cost a pretty penny! Fortunately, purchasing your own hardware isn't a requirement to get started with these powerful machines. Here at Vast.ai, with low-cost cloud GPU rental through our platform, you can access a wide range of compute power across our network of hosts around the world, anytime, anywhere.

We offer the best prices for GPU rental, with on-demand pricing as low as $0.12/hr for the A40 and $0.50/hr for the RTX A6000 – and interruptible instances available via spot auction-based pricing for even more savings. We're proud of our mission to help democratize AI and ensure that its benefits are available to all!

---

**[The table below compares various features and specs between the NVIDIA A40 and RTX A6000, with differences highlighted.]**

| Feature                                              | A40                                                        | RTX A6000                                                  |
| ---------------------------------------------------- | ---------------------------------------------------------- | ---------------------------------------------------------- |
| Architecture                                         | Ampere                                                     | Ampere                                                     |
| Memory Size                                          | 48 GB                                                      | 48 GB                                                      |
| Memory Type                                          | GDDR6                                                      | GDDR6                                                      |
| Error-Correcting Code (ECC)                          | Yes                                                        | Yes                                                        |
| CUDA Cores                                           | 10,752                                                     | 10,752                                                     |
| 3rd-gen Tensor Cores                                 | 336                                                        | 336                                                        |
| 2nd-gen RT Cores                                     | 84                                                         | 84                                                         |
| Render Output Units (ROPs)                           | 112                                                        | 112                                                        |
| FP16 (Half) Performance                              | 37.42 TFLOPS                                               | **38.71 TFLOPS**                                           |
| FP32 (Float) Performance                             | 37.42 TFLOPS                                               | **38.71 TFLOPS**                                           |
| FP64 (Double) Performance                            | 584.6 GFLOPS                                               | **604.8 GFLOPS**                                           |
| Pixel Rate                                           | 194.9 GPixel/s                                             | **201.6 GPixel/s**                                         |
| Texture Rate                                         | 584.6 GTexel/s                                             | **604.8 GTexel/s**                                         |
| Memory Interface                                     | 384-bit                                                    | 384-bit                                                    |
| Memory Bandwidth                                     | 696 GB/s                                                   | **768 GB/s**                                               |
| Clock Speed                                          | 1305 MHz                                                   | **1410 MHz**                                               |
| Boost Speed                                          | 1740 MHz                                                   | **1800 MHz**                                               |
| Memory Clock                                         | 1812 MHz (14.5 Gbps effective)                             | **2000 MHz (16 Gbps effective)**                           |
| Slot Width                                           | Dual-slot                                                  | Dual-slot                                                  |
| Thermal Solution                                     | Passive                                                    | Active                                                     |
| Power Consumption (Total Board Power)                | 300 W                                                      | 300 W                                                      |
| Suggested Power Supply                               | 700 W                                                      | 700 W                                                      |
| System Interface                                     | PCIe 4.0 x16                                               | PCIe 4.0 x16                                               |
| Display Ports                                        | 3x DisplayPort 1.4                                         | **4x DisplayPort 1.4**                                     |
| vGPU Support                                         | Yes (default)                                              | Yes                                                        |
| NVLink                                               | Yes                                                        | Yes                                                        |
| NEBS Ready                                           | **Yes (Level 3)**                                          | No                                                         |
| Secure and Measured Boot with Hardware Root of Trust | **Yes (optional)**                                         | No                                                         |
| Graphics APIs                                        | DirectX 12.07, Shader Model 5.17, OpenGL 4.68, Vulkan 1.18 | DirectX 12.07, Shader Model 5.17, OpenGL 4.68, Vulkan 1.18 |
| Compute APIs                                         | CUDA, DirectCompute, OpenCL, **OpenACC**                   | CUDA, DirectCompute, OpenCL                                |
| Release Date                                         | Oct. 5th, 2020                                             | Oct. 5th, 2020                                             |


Maximizing Value with NVIDIA A40 & RTX A6000...

Vast

Security and Compliance at Vast AI

Serving Online Inference with TGI and Medusa on Vast.ai

Serving Online Inference with Text-Embeddings-Inference on Vast.ai | June 2024

Serving Online Inference with TGI on Vast.ai | June 2024

Maximizing Value with NVIDIA A40 & RTX A6000 | June 2024