
The H100 and H200 GPUs represent the top end of NVIDIA's current lineup for AI and high-performance computing (HPC). Both are designed for serious, large-scale workloads – from training foundation models to running complex scientific simulations. 

At a glance, the two GPUs may appear similar, but for those working across enterprise environments or pushing the boundaries of exascale computing, the differences between them can have a major impact on performance and scalability.

Today, we're taking a look at how the H100 and H200 stack up against each other to help you decide which of these two heavyweights may be the better fit for your needs.

### NVIDIA H100: The Proven Performer

The H100 GPU made waves when it launched in March 2023\. Quickly adopted by big names like OpenAI, Meta, and Stability AI, it became the engine behind some of the most advanced generative AI systems in production today. 

Built on NVIDIA's innovative Hopper architecture, the H100 was engineered for scale, handling everything from trillion-parameter language models to high-throughput inference. It introduced **fourth-gen Tensor Cores** and a dedicated **Transformer Engine**, designed to accelerate deep learning workloads with speed and FP8 precision.

The H100 boasts up to a **30X boost in performance** for large language models (LLMs) over its predecessor, the A100 GPU, according to NVIDIA. This massive improvement helped cement the H100 as the new standard for foundational model training, and to this day, it's still one of the very best GPUs on the market.

#### Some of the H100's more notable features include:

* **80 GB of HBM3 memory**, with some models offering up to 94 GB.  
    
* **3.35 TB/s of memory bandwidth** to handle data-intensive workloads.  
    
* **Fourth-gen Tensor Cores** with superior throughput performance, along with an advanced Transformer Engine and FP64 and FP8 capabilities to reduce memory usage.  
    
* **PCIe Gen5 and fourth-gen NVLink** (with 900 GB/s of GPU-to-GPU interconnect), as well as NDR Quantum-2 InfiniBand networking for efficient scalability.  
    
* **Multi-Instance GPU (MIG) support** for secure partitioning into up to 7 isolated right-size instances that maximize quality of service for multi-tenant environments and parallel workloads.

If your projects involve massive datasets, large-scale simulations, or advanced AI training for foundational models, the H100 has proven itself as a reliable, high-performance solution. But with the arrival of the H200 in late 2024, NVIDIA has taken things a step further, introducing upgrades that push the envelope even more for memory-intensive and high-throughput workloads.

We ran a side-by-side comparison of inference performance on the **DeepSeek R1 Distill Llama 8B** model using `vLLM` with both NVIDIA H100 and H200 GPUs on Vast.ai.

#### **Benchmark Configuration**

* **Model:** `deepseek-ai/DeepSeek-R1-Distill-Llama-8B`  
* **Requests:** 5,000  
* **Batch:** 512  
* **Input Tokens:** 16  
* **Output Tokens:** 128  
* **Backend:** `vLLM`

---

### **Results**

| Metric | H100 | H200 |
| ----- | ----- | ----- |
| **Benchmark Duration** | 45.28 s | 40.02 s |
| **Request Throughput** | 110.42 req/s | 124.93 req/s |
| **Output Throughput** | 13,650 tok/s | 15,435 tok/s |
| **Total Throughput** | 15,416 tok/s | 17,434 tok/s |

### NVIDIA H200: The Supercharged Successor

The H200 builds directly on the foundation laid by the H100, bringing meaningful improvements that target one of the biggest bottlenecks in AI and HPC workloads: memory. 

While it shares the same Hopper architecture as the H100 and has similar core compute specs, the H200 significantly increases both memory capacity and bandwidth, two factors that become critical when operating at enterprise scale or pushing into exascale territory.

Equipped with **141 GB of HBM3e memory** and **4.8 TB/s of bandwidth**, the H200 nearly doubles the memory of the H100 and delivers about **1.4X faster** data access. Early benchmarks reflect that improvement. In MLPerf testing using Llama 2 70B, the H200 achieved over 31,000 tokens per second – about **45% faster than the H100**. 

<img
    src="/uploads/h100vsh200_2.png"
    alt="Image credit: NVIDIA"
    style={{ width: "100%" }}
/>

The H200 retains all of the architectural strengths of the H100, including **fourth-gen NVLink**, **PCIe Gen5**, **MIG support**, and full **Tensor Core and Transformer Engine capabilities**. But for developers and organizations looking to squeeze out more performance without overhauling their infrastructure, the H200 offers a straightforward upgrade path with [measurable gains](https://resources.nvidia.com/en-us-data-center-overview/hpc-datasheet-sc23-h200).

#### To recap, here are a few advantages of the H200:

* **141 GB of HBM3e memory at 4.8 TB/s** – nearly double the capacity of the H100, with 1.4X more memory bandwidth.  
    
* **Up to 1.9X the inference performance** of the H100 when handling LLMs, with more efficient processing per token and per inference cycle.  
    
* **50% reduced energy use and total cost of ownership** (TCO) due to improved thermal management and energy efficiency.  
    
* **Drop-in compatibility with existing Hopper-based systems**, enabling accelerated performance gains in H100-optimized infrastructure with minimal or no additional changes.

In short, the H200 represents a meaningful leap beyond the H100 in memory capacity, bandwidth, and throughput, bringing a new level of performance to AI and HPC workloads. That comes at a premium price, of course. But if you're weighing the tradeoffs between the two GPUs in terms of specs alone, let's take a side-by-side look at how they compare.

### H100 vs. H200: Key Specs Compared

For reference, here's a quick outline of the various features and specs of the NVIDIA H100 and H200:

| Specification | H100 | H200 |
| :---- | :---- | :---- |
| **GPU Architecture** | Hopper | Hopper |
| **GPU Memory** | 80 GB HBM3 | 141 GB HBM3e |
| **GPU Memory Bandwidth** | 3.35 TB/s | 4.8 TB/s |
| **FP64 Tensor Core** | 33.5 TFLOPS | 33.5 TFLOPS |
| **FP32** | 67 TFLOPS | 67 TFLOPS |
| **TF32 Tensor Core** | 989 TFLOPS | 989 TFLOPS |
| **FP16 Tensor Core** | 1,979 TFLOPS | 1,979 TFLOPS |
| **FP8 Tensor Core** | 3,958 TFLOPS | 3,958 TFLOPS |
| **INT8 Tensor Core** | 3,958 TOPS | 3,958 TOPS |
| **Decoders** | 7 NVDEC, 7 NVJPEG | 7 NVDEC, 7 NVJPEG |
| **L2 Cache** | 50 MB | 50 MB |
| **Max. Thermal Design Power** | Up to 700 W | Up to 1000 W |
| **Multi-Instance GPUs** | Up to 7 MIGs @ 10GB each | Up to 7 MIGs @ 18GB each |
| **Form Factor** | SXM | SXM |
| **Interconnect** | PCIe 5.0 x16 | PCIe 5.0 x16 |

Ultimately, choosing the right GPU largely depends on your workload demands and performance priorities.

### Use Cases: Which GPU Should You Choose?

While both the H100 and H200 excel in AI and HPC environments, they serve slightly different needs.

#### You should consider the H100 if you prefer:

* **A proven, high-performance GPU** for training medium to large models and handling intensive AI or HPC workloads.  
* **Use cases involving compute-bound workloads**, where total memory capacity is not as critical.  
* **A time-tested solution** already in widespread production use, balancing performance with lower power and cost.

#### Whereas the H200 may be a better fit if you need:

* **Greater memory capacity and bandwidth** for running *even larger* models, handling memory-bound workloads, or scaling LLM inference on the enterprise level.  
* **Higher throughput**, especially for tasks involving extended context windows, multi-modal AI, or very large batch sizes.  
* **A forward-looking upgrade path** with improved energy efficiency and lower TCO, thanks to higher performance per watt.

\------------------------------

At **Vast.ai**, we understand that choosing a GPU isn't only about specs. Often, the decision comes down to access and affordability, as well – especially when we're talking about some of the most powerful and expensive GPUs in existence. (Speaking of, check out our previous posts comparing the [H100 vs. A100](https://vast.ai/article/H100-vs-A100-Comparing-two-Powerhouse-GPUs), and the [H100 vs. L40S](https://vast.ai/article/nvidia-h100-vs-l40s-power-meets-versatility).)

That's why our cloud GPU rental platform is built to offer high-performance compute at a fraction of the typical cost. Vast gives you flexible, cost-effective access to H100s, H200s, and more.

With options for on-demand pricing and budget-friendly spot instances, you can scale up as needed without overspending. On average, **Vast users save 5- 6X on GPU compute** compared to traditional cloud providers.

Explore our platform today and get the power you need, when you need it – with no upfront investment required.

The H100 and H200 GPUs represent the top end of NVIDIA's current lineup for AI and high-performance computing (HPC). Both are designed for serious, large-scale workloads – from training foundation models to running complex scientific simulations.

NVIDIA H100 vs. H200: Two Hopper-based Heavyweights


## Introduction

Rerankers are powerful tools that excel at determining the relevance between pairs of text - whether you're matching search queries to documents, evaluating LLM outputs against prompts, or finding similar content in a database. Unlike simple keyword matching or embedding similarity, these specialized models perform a detailed comparison between inputs, capturing nuanced relationships that simpler methods might miss. They're particularly valuable in RAG (Retrieval Augmented Generation) systems, recommendation engines, and content filtering pipelines, where they can significantly improve quality while requiring minimal computational overhead.

vLLM has recently expanded its capabilities to include reranker model serving, offering compatibility with both OpenAI and Cohere APIs. In this guide, we'll focus on deploying the `BAAI/bge-reranker-base` model - a powerful yet efficient reranker designed for semantic similarity scoring.

Vast.ai provides a marketplace for renting GPU compute power, offering a cost-effective alternative to major cloud providers. Vast has GPU SKU's that you cannot normally find on clouds, including 4000 series GPU's which are cheaper, with a tradeoff of less RAM. Re-ranker models are often quite small so we can take advantage of these extra affordable prices.

In this guide, we will:

1. Set up a Vast.ai instance with the right GPU specifications for serving rerankers
2. Deploy the model using vLLM's optimized inference server
3. Demonstrate two ways to interact with the reranker:
   - Using the Cohere-compatible API for batch reranking
   - Using the OpenAI-compatible API for cross-encoder scoring

This setup provides a production-ready environment for serving reranker models, with the ability to handle both batch reranking requests and individual similarity scoring tasks.

## Setting Up Vast.ai

First, we'll install the Vast.ai API:

```bash
pip install --upgrade vastai
```

Next, we'll set our API key (found on the [Account Page](https://cloud.vast.ai/account/)):

```bash
export VAST_API_KEY="your-key-here"
vastai set api-key $VAST_API_KEY
```

## Choosing Hardware

The `BAAI/bge-reranker-base` model has modest requirements compared to larger language models. Here's what we need:

1. 16GB GPU RAM for:
   - Model weights (~278M parameters)
   - Batch processing overhead
   - System operations
2. Single GPU with Turing architecture or newer
3. Static IP for stable API endpoint
4. At least one direct port for the API server

Note that you could potentially go down to 8GB of GPU RAM for this model for more affordable usages.

We'll use the `vastai search offers` command to find instances that meet our requirements:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 16 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
rentable = true"
```

## Deploying the Server

Next, we'll copy the `ID` from our chosen instance into the `instance-id` field below along with our `HUGGINGFACE_TOKEN` to rent an instance and deploy `BAAI/bge-reranker-base` using vLLM's OpenAI-compatible server:

```bash
export INSTANCE_ID=<instance-id>
vastai create instance $INSTANCE_ID \
    --image vllm/vllm-openai:latest \
    --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGINGFACE_TOKEN' \
    --disk 40 \
    --args --model BAAI/bge-reranker-base
```

This setup:

- Uses vLLM's optimized inference server
- Exposes port 8000 inside the container (Vast.ai will forward this to a different external port)
- Downloads and serves the `BAAI/bge-reranker-base` model

## Verify the Setup

Before proceeding, verify that your instance is running correctly:

1. Go to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console
2. Wait for the instance to download the image and model (this may take a few minutes)
3. Find your instance's IP address and port from the "Open Ports" panel - you'll see something like `XX.XX.XXX.XX:YYYY -> 8000/tcp` where YYYY is the external port that forwards to the container's port 8000
4. Use these values in `VAST_IP_ADDRESS` and `VAST_PORT` for the test request below

```bash
export VAST_IP_ADDRESS="your-ip-here"
export VAST_PORT="your-port-here"

curl -X POST http://$VAST_IP_ADDRESS:$VAST_PORT/rerank \
    -H "Content-Type: application/json" \
    -d '{
    "model": "BAAI/bge-reranker-base",
    "query": "What is deep learning?",
    "documents": [
        "Deep learning is a type of machine learning"
    ]
    }'
```

## Using the Reranker

vLLM provides two ways to rerank documents using the same underlying model:

1. **OpenAI API Compatible** (`/score`)

   - Raw scoring between query and documents
   - Returns similarity scores for manual sorting
   - Best for custom ranking logic

2. **Cohere API Compatible** (`/rerank`)
   - Direct reranking of documents against a query
   - Returns pre-sorted results
   - Best for quick integration

### OpenAI-Compatible Endpoint

First, we'll create a function to call the `/score` endpoint. We'll need to enter the `IP_ADDRESS` and `PORT` from above:

```python
import requests

IP_ADDRESS = ""
PORT = ""

def openai_score(query,documents):
    base_url = f"http://{IP_ADDRESS}:{PORT}"

    # Format request for score endpoint
    test_request = {
        "model": "BAAI/bge-reranker-base",
        "text_1": query,  # Query is text_1
        "text_2": documents  # Documents are text_2
    }

    # Make request and print raw response
    response = requests.post(f"{base_url}/score", json=test_request)
    print("Status code:", response.status_code)
    print("Raw response:", response.text)

    # If successful, print formatted results
    if response.status_code == 200:
        data = response.json()
        scores = [(doc, item["score"]) for doc, item in zip(documents, data["data"])]
        scores.sort(key=lambda x: x[1], reverse=True)

        print("\nRanked results:")
        for text, score in scores:
            print("\nScore:", score)
            print("Text:", text)
```

We'll start with a simple example to show what happens when we have completely irrelevant documents:

```python
query = "What is Deep Learning?"
documents = [
    "Deep learning is a subset of machine learning that uses neural networks with many layers",
    "The weather is nice today",
    "Deep learning enables computers to learn from large amounts of data",
    "I like pizza"
]
openai_score(query,documents)
```

In this example, we can see how effectively the reranker distinguishes between relevant and irrelevant content:

- The most relevant document (score ~1.0) provides a direct definition of deep learning
- The second document (score ~0.18) mentions deep learning but provides less specific information
- The irrelevant documents about weather and pizza receive nearly zero scores (~0.00004)

```
Ranked results:

Score: 0.99951171875
Text: Deep learning is a subset of machine learning that uses neural networks with many layers

Score: 0.17626953125
Text: Deep learning enables computers to learn from large amounts of data

Score: 3.737211227416992e-05
Text: The weather is nice today

Score: 3.737211227416992e-05
Text: I like pizza
```

Next, we'll run it with a slightly more realistic set of documents to see how the reranker handles more nuanced differences in relevance:

```python
query = "What is Deep Learning?"

documents = [
    "Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input",
    "Machine learning algorithms enable computers to learn from data without being explicitly programmed",
    "The latest smartphone features advanced AI capabilities for photo enhancement",
    "Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection",
    "A neural network is inspired by the biological neural networks that constitute animal brains",
    "Cloud computing provides scalable infrastructure for training deep learning models",
    "The history of artificial intelligence dates back to the 1950s",
    "Deep learning models require significant computational resources and large datasets for training"
]
openai_score(query,documents)
```

The reranker shows sophisticated understanding in this more nuanced example:

1. The comprehensive definition gets a perfect score (1.0)
2. Application examples (computer vision) score moderately well (~0.43)
3. Related concepts (machine learning, neural networks) get low but non-zero scores (~0.03)
4. Infrastructure and historical context receive nearly zero scores (<0.001)

This demonstrates how the reranker can capture subtle differences in relevance, not just obvious distinctions between relevant and irrelevant content.

```
Ranked results:

Score: 1.0
Text: Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input

Score: 0.425537109375
Text: Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection

Score: 0.034942626953125
Text: Machine learning algorithms enable computers to learn from data without being explicitly programmed

Score: 0.001674652099609375
Text: A neural network is inspired by the biological neural networks that constitute animal brains

Score: 0.0004711151123046875
Text: The history of artificial intelligence dates back to the 1950s

Score: 0.00026535987854003906
Text: Deep learning models require significant computational resources and large datasets for training

Score: 5.7816505432128906e-05
Text: Cloud computing provides scalable infrastructure for training deep learning models

Score: 3.737211227416992e-05
Text: The latest smartphone features advanced AI capabilities for photo enhancement
```

### Cohere-Compatible Endpoint

The `/rerank` endpoint provides a higher-level interface that:

- Directly reranks documents against a query
- Returns pre-sorted results
- Handles all scoring logic internally
- Simplifies integration into existing pipelines

First, we'll install Cohere.

```bash
pip install --upgrade cohere
```

We'll then create a function to call the cohere endpoint.

```python
import cohere

IP_ADDRESS = ""
PORT = ""

def cohere_reranker(query,documents):
    base_url = f"http://{IP_ADDRESS}:{PORT}"

    # Initialize v2 client with our endpoint
    co = cohere.ClientV2("sk-fake-key", base_url=base_url)

    result = co.rerank(
        model="BAAI/bge-reranker-base",
        query=query,
        documents=documents
    )

    print("\nRanked results:")
    for doc in result.results:
        print(f"\nScore: {doc.relevance_score}")
        print(f"Text: {doc.document.text}")  # Access text through document.text
```

First, we'll test with our simple example:

```python
query = "What is Deep Learning?"
documents = [
    "Deep learning is a subset of machine learning that uses neural networks with many layers",
    "The weather is nice today",
    "Deep learning enables computers to learn from large amounts of data",
    "I like pizza"
]
cohere_reranker(query,documents)
```

Output:

```
Ranked results:

Score: 0.99951171875
Text: Deep learning is a subset of machine learning that uses neural networks with many layers

Score: 0.17626953125
Text: Deep learning enables computers to learn from large amounts of data

Score: 3.737211227416992e-05
Text: The weather is nice today

Score: 3.737211227416992e-05
Text: I like pizza
```

Notice that the Cohere endpoint produces identical scores to the OpenAI endpoint - this is because both are using the same underlying model, just with different APIs. The key difference is that the Cohere endpoint automatically handles the sorting and formatting of results.

Next, we'll try our more complex example:

```python
query = "What is Deep Learning?"

documents = [
    "Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input",
    "Machine learning algorithms enable computers to learn from data without being explicitly programmed",
    "The latest smartphone features advanced AI capabilities for photo enhancement",
    "Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection",
    "A neural network is inspired by the biological neural networks that constitute animal brains",
    "Cloud computing provides scalable infrastructure for training deep learning models",
    "The history of artificial intelligence dates back to the 1950s",
    "Deep learning models require significant computational resources and large datasets for training"
]
cohere_reranker(query,documents)
```

Output:

```
Ranked results:

Score: 1.0
Text: Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input

Score: 0.425537109375
Text: Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection

Score: 0.034942626953125
Text: Machine learning algorithms enable computers to learn from data without being explicitly programmed

Score: 0.001674652099609375
Text: A neural network is inspired by the biological neural networks that constitute animal brains

Score: 0.0004711151123046875
Text: The history of artificial intelligence dates back to the 1950s

Score: 0.00026535987854003906
Text: Deep learning models require significant computational resources and large datasets for training

Score: 5.7816505432128906e-05
Text: Cloud computing provides scalable infrastructure for training deep learning models

Score: 3.737211227416992e-05
Text: The latest smartphone features advanced AI capabilities for photo enhancement
```

Again we see identical scores to the OpenAI endpoint, demonstrating that:

1. Both APIs provide consistent access to the same model
2. The Cohere endpoint offers a more streamlined interface
3. You can choose whichever API better fits your application's needs

The main advantages of the Cohere endpoint are:

- Pre-sorted results (no need to sort scores manually)
- Simpler response format
- Built-in batching support
- Familiar interface for existing Cohere users

## Key Features

- **Optimized Inference**: vLLM's server provides efficient batch processing and automatic memory management, making it easy to serve rerankers without dealing with low-level optimizations.
- **Dual API Support**: Access the model through both OpenAI and Cohere-compatible APIs, allowing flexible integration with existing applications.
- **Cost-Effective Deployment**: Vast.ai's marketplace lets you access powerful GPUs at a fraction of traditional cloud costs, with easy scaling as your needs grow.

## What's Now Possible

With this setup, you can build:

- More accurate semantic search systems
- Better RAG applications with filtered context
- Content recommendation systems
- Semantic duplicate detection
- Document clustering and organization

The combination of vLLM's efficient serving and Vast.ai's affordable GPUs makes it practical to deploy rerankers in production. You can start small and scale up as needed, while maintaining high performance and cost efficiency. Happy ranking!


Deploy rerankers on Vast.ai using vLLM for cost-effective, high-performance semantic search. Supports OpenAI & Cohere APIs for flexible integration.

Serving Rerankers on Vast.ai using vLLM


# Announcing Virtual Machine Rental on Vast.ai

Have you ever thought about what you could accomplish with more flexibility in your GPU rentals? At Vast.ai, we're always looking for ways to innovate and improve our platform, and today we're excited to announce our latest feature: [virtual machine (VM) rental](https://vast.ai/docs/instance-setup/virtual-machines).

We now offer full VMs alongside Docker-based instance rentals. This new offering is designed to support larger customers and more diverse workflows via GPU remote desktop.

Here's everything you need to know to get started!

## Virtual Machine Rental: Expanding the Possibilities

The new VM rental option will help meet the demand for use cases that Docker can't easily support. **VMs also offer features not available in Docker, such as support for init managers and process tracing.**

### Key Benefits of Virtual Machine Support Include:

- **Enhanced Flexibility:** Users can run low level debugging and performance profiling utilities and are not constrained by container environments.
- **Kernel-space Customizations:** Developers gain granular control over their systems, enabling custom driver installations, kernel tweaks, and integration of specialized frameworks.
- **Faster Iteration:** Rather than pre-building and pushing custom Docker images, users can run Docker inside the instance.
- **Improved Security:** Virtual Machines offers an improved  security posture for sensitive workloads.

Host machines can now tap into these advantages, allowing them to attract more renters and capitalize on the increased need for GPU-powered tasks beyond Docker’s capabilities.

### Important VM Differences to Know:

VMs do come with some trade-offs compared to Docker-based instances. For example, they have slower creation and boot times, higher disk overhead, and users will have to deal with a smaller selection of machines and lesser availability of preconfigured templates. In addition, only SSH launch mode is currently supported.

Notably, the on-start script field allows users to specify a script to run on instance start. Unlike in Docker-based instances, the interpreter must be specified by a shebang.

Another difference is that **the cloud copy utility for VMs only allows migrating the entire VM instead of allowing individual folders to be copied.**

Despite these drawbacks, VMs open up new opportunities for both renters and hosts, offering greater flexibility, broader use cases, and the potential for higher earnings.

### Empowering Renters, Rewarding Hosts

One thing that sets us apart from other providers is our focus on cost-efficiency. We strive to keep rental costs low for users across the board – and in fact, Vast.ai GPU rentals are ~3-5X cheaper than current alternatives.

At the same time, we want to provide hosts with the opportunity to earn more.

As our COO, Travis Cannell, explains, "We've always believed that democratizing GPU access means offering maximum flexibility. Virtual Machine support was the natural next step, providing more control, ease of use, and enabling researchers to build and iterate."

With the opportunity for both renters and hosts to benefit, it's equally important to ensure that setup is seamless and operations remain secure. Here's how we make that happen.

## VMs on Vast: Simple Setup and Strong Security

Getting started with VM rental on Vast.ai is simple. **Our hosting software automatically tests host machines for VM compatibility and enables the feature on any machine where it's supported.** (There is no requirement to be VM compatible, however.)

For new host machines, these tests will be run upon installation. For machines configured before the VM feature release, testing for VM compatibility will happen at a time when the machine is unoccupied.

And that's it! Host machines that don't have VM support enabled will simply be hidden from search results when VM-based templates are selected.

Privacy and security are built into the system, as well. Hosts and users cannot view someone else's VM, and no one has access to see what any other VM is doing. Users can rent VMs with peace of mind knowing that their virtual environment is private and secure.

## Getting Started: Creating a VM

Ready to spin up your first VM? It's as easy as selecting the right template!

1.  Make sure you have an SSH key in your account (guide [here](https://vast.ai/docs/instance-setup/ssh)).
2.  Select the **Ubuntu 22.04 VM template** in the interface or [click here](https://cloud.vast.ai/?ref_id=108031&template_id=e2f271ce754aeef772181fbf9c82a354) to load it.
3.  Adjust the disk storage needed and then rent a GPU!

Note: SSH keys cannot be edited on running VMs, and SSH access is currently the only method preconfigured on VM instances. Also, when you select the correct template, it will automatically filter for machines that support VMs and will launch a VM when you rent with that template.

For more detailed information about VM template customization, see our Docs page [here](https://vast.ai/docs/instance-setup/virtual-machines).

## Power Up with VM Rental on Vast.ai

With our new virtual machine rental feature, we're opening up a world of possibilities for both hosts and renters. Whether you need the flexibility of GPU remote desktops for complex workflows or you're looking to maximize your earning potential as a host, Vast.ai has you covered. We're committed to giving you the tools to succeed – easily, affordably, and securely.

Ready to see what you can accomplish? Experience the [power of VM rental](https://vast.ai/docs/instance-setup/virtual-machines) with Vast.ai today!


At Vast.ai, we're always looking for ways to innovate and improve our platform, and today we're excited to announce our latest feature: virtual machine (VM) rental. 

Announcing Virtual Machine Rental on Vast.ai


# Google Colab vs. Vast: Which Platform Is Best for Your Projects?

If you’ve ever dabbled in machine learning, data science, or coding in general, chances are you’ve heard of Google Colaboratory – better known as **Google Colab**. But what exactly is it, what are its limitations, and how can you make the best use of it?

Let's take a closer look at these questions and more…

## What Is Google Colab?

In a nutshell, **Google Colab** is a free, browser-based platform that lets you run and write code in a Jupyter notebook environment without having to set up anything locally on your machine. It's widely used for Python-based projects, especially in machine learning and data analysis.

Colab offers an easy starting point for experiments, since you can use the Jupyter notebook interface to write and execute code, visualize results, and keep everything neatly organized in one place. This type of interactive programming allows you to experiment with code in real time, making it ideal for data science and research projects that require immediate feedback and iteration.

### Some other benefits include:

- **Accessibility** – The browser-based platform allows you to access your projects from anywhere.
- **Integration with Google Drive** – Save your notebooks and datasets directly to your Google Drive.
- **Collaboration** – Easily share your notebooks with others for real-time collaboration across teams.
- **Wealth of Resources** – Plenty of tutorials, sample notebooks, and community support are available in the Colab ecosystem, making it easier for you to learn and troubleshoot along the way.

These features make Google Colab a convenient tool for a wide range of users. However, it does have some significant limitations that may prompt you to look for other solutions – such as running a Jupyter instance right here on **Vast** and uploading your notebook directly.

## Why Choose Vast Over Google Colab?

While Google Colab is helpful for smaller-scale projects or initial experimentation, **Vast's** cloud GPU platform offers a much more flexible, powerful, and reliable experience through our Jupyter integration. Here’s why you should consider using Vast instead.

### No Resource Limits

With Google Colab, free-tier users face dynamic usage limits with idle timeout periods and resource caps that sometimes fluctuate. The paid plans have their own set of restrictions as well. For example, notebooks can run for up to 12 hours on a free plan, and the highest-tier paid plan supports continuous code execution for at most 24 hours.

But with **Vast**, there are no imposed time limits, so your projects can run uninterrupted for as long as you need. Whether your project takes hours or days, you won't be cut off or forced to restart.

> **Note**: Google Colab does include the option to pay for a dedicated virtual machine (VM) to get around usage limits, but Vast's pricing is extremely competitive as we are the market leader in low-cost cloud GPU rental.

### Customizable GPU Power for Every Project

Google Colab offers different VMs based on the plan you choose. The free tier only allows access to VMs with a standard system memory profile, restricting the compute power you can use. Paid versions of Colab provide access to machines with a higher system memory profile, but you still don't get to select specific GPUs unless you opt to manually pay for a dedicated machine.

**Vast** gives you access right from the start to powerful and customizable hardware options that are also highly affordable. You can select the exact GPU setup you need, whether you're running small tasks or large-scale machine learning models. For high-end performance, Vast supports multi-GPU configurations, making it easy to scale up your projects without hitting resource bottlenecks.

### Simplicity and Reliability

**Vast** simplifies the entire process by running your Jupyter notebook directly in the cloud with no need for SSH or complex setups. Jupyter by itself is much more reliable than Google Colab and doesn't require setting up SSH keys – you can always open a terminal right inside the browser.

And while Google Colab does facilitate sharing and collaboration, **Vast** offers the same, with the added benefit of running on a more stable platform without timeouts or interruptions. Team members can collaborate in real time, sharing notebooks and results without worrying about session limits.

## Making the Switch to Vast

If you've been using Google Colab for simple notebooks and want to step up your workflow, switching to Vast is straightforward. Simply download your notebook from Colab as a `.ipynb` file, run a Vast Jupyter instance with the recommended [PyTorch template](https://cloud.vast.ai/?ref_id=43484&template_id=f5540ef1a1398b8499546edb53dae704), and upload your notebook into Jupyter directly.

For a step-by-step guide on running Jupyter notebooks on Vast, check out our detailed doc [here](https://vast.ai/docs/use-cases/google-colab).

While Google Colab is a useful starting point for basic tasks, it often falls short when you're dealing with more complex workflows. **Vast** provides a more robust and customizable solution through its Jupyter notebook integration, with greater control, higher performance, and fewer interruptions.

With **Vast**, you'll enjoy the freedom to push your projects even further. **Explore the possibilities today!**


Google Colab is a free, browser-based platform that lets you run and write code in a Jupyter notebook environment without having to set up anything locally on your machine.

Google Colab Explained: Simplifying Your Workflow with Cloud Tools


# Serving vLLM Embeddings on Vast.ai

## Background:

[vLLM](https://github.com/vllm-project/vllm) is a developer-friendly framework for serving large language models. It is very fast and focused on high throughput serving multiple simultaneous calls for production use cases. `vLLM` is now more flexible than ever as it also supports embedding models. This brings vLLM's dynamic batching and Paged Attention to embedding models for much faster throughput, all from the docker image that developers are used to.

This guide will show you how to setup vLLM to serve embedding models on Vast.

## Setup and Querying

First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

To serve an embedding model, we'll want a machine with a static IP address + ports to forward. We only will need a single GPU, which can have a small amount of RAM since embedding models are small. We'll prefer a modern GPU so we will add a filter for a compute cap of more than 800.

```bash
vastai search offers 'compute_cap >= 800 gpu_ram >= 24 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.4'
```

## Deploying the Image:

The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below.

### Hosting a Single Embedding Model:

For now, we'll host just one embedding model.
The easiest way to deploy a single model on this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below.

We particularly need `v2` so that we use the correct version of the api, `--port 8000` so it serves on the correct model, and `--model-id intfloat/e5-mistral-7b-instruct` to serve the correct model.

```bash
vastai create instance <instance-id> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 40 --args --model intfloat/e5-mistral-7b-instruct
```

## Connecting and Testing:

To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/serve_vllm_embeddings_instance_view.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/serve_vllm_embeddings_ip_address_view.png)

## Connecting with the python requests library

```
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'intfloat/e5-mistral-7b-instruct',
    "input": ["San francisco is a"],
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/embeddings', headers=headers, json=json_data)
print(response.content)
```

## Connecting with the OpenAI SDK

The OpenAI SDK also allows for generating embeddings. We can use it as well with the following code.

```bash
pip install openai
```

We will copy over the IP address and the port into the cell below.

```python

from openai import OpenAI

# Modify OpenAI's API key and API base to use Infinity's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
model = "intfloat/e5-mistral-7b-instruct"
embeddings = client.embeddings.create(model=model, input="San francisco is a").data[0].embedding
print("Embeddings:")
print(embeddings)
```

Now we can integrate our embedding server into apps that use the OpenAI SDK.

## Next Steps:

Embeddings are an important part of GenAI applications along with running inference of Generative Models. Now with vLLM, you can run them with the same docker image on Vast for ultimate flexibility.


Industry

NVIDIA H100 vs. H200: Two Hopper-based Heavyweights

Serving Rerankers on Vast.ai using vLLM

Announcing Virtual Machine Rental on Vast.ai

Google Colab Explained: Simplifying Your Workflow with Cloud Tools

Serving vLLM Embeddings on Vast.ai