
The H100 and H200 GPUs represent the top end of NVIDIA's current lineup for AI and high-performance computing (HPC). Both are designed for serious, large-scale workloads – from training foundation models to running complex scientific simulations.

At a glance, the two GPUs may appear similar, but for those working across enterprise environments or pushing the boundaries of exascale computing, the differences between them can have a major impact on performance and scalability.

Today, we're taking a look at how the H100 and H200 stack up against each other to help you decide which of these two heavyweights may be the better fit for your needs.

### NVIDIA H100: The Proven Performer

The H100 GPU made waves when it launched in March 2023\. Quickly adopted by big names like OpenAI, Meta, and Stability AI, it became the engine behind some of the most advanced generative AI systems in production today.

Built on NVIDIA's innovative Hopper architecture, the H100 was engineered for scale, handling everything from trillion-parameter language models to high-throughput inference. It introduced **fourth-gen Tensor Cores** and a dedicated **Transformer Engine**, designed to accelerate deep learning workloads with speed and FP8 precision.

The H100 boasts up to a **30X boost in performance** for large language models (LLMs) over its predecessor, the A100 GPU, according to NVIDIA. This massive improvement helped cement the H100 as the new standard for foundational model training, and to this day, it's still one of the very best GPUs on the market.

#### Some of the H100's more notable features include:

-   **80 GB of HBM3 memory**, with some models offering up to 94 GB.

-   **3.35 TB/s of memory bandwidth** to handle data-intensive workloads.

-   **Fourth-gen Tensor Cores** with superior throughput performance, along with an advanced Transformer Engine and FP64 and FP8 capabilities to reduce memory usage.

-   **PCIe Gen5 and fourth-gen NVLink** (with 900 GB/s of GPU-to-GPU interconnect), as well as NDR Quantum-2 InfiniBand networking for efficient scalability.

-   **Multi-Instance GPU (MIG) support** for secure partitioning into up to 7 isolated right-size instances that maximize quality of service for multi-tenant environments and parallel workloads.

If your projects involve massive datasets, large-scale simulations, or advanced AI training for foundational models, the H100 has proven itself as a reliable, high-performance solution. But with the arrival of the H200 in late 2024, NVIDIA has taken things a step further, introducing upgrades that push the envelope even more for memory-intensive and high-throughput workloads.

We ran a side-by-side comparison of inference performance on the **DeepSeek R1 Distill Llama 8B** model using `vLLM` with both NVIDIA H100 and H200 GPUs on Vast.ai.

#### **Benchmark Configuration**

-   **Model:** `deepseek-ai/DeepSeek-R1-Distill-Llama-8B`
-   **Requests:** 5,000
-   **Batch:** 512
-   **Input Tokens:** 16
-   **Output Tokens:** 128
-   **Backend:** `vLLM`

---

### **Results**

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Metric
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H100
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H200
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Benchmark Duration
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                45.28 s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                40.02 s
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Request Throughput
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                110.42 req/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                124.93 req/s
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Output Throughput
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                13,650 tok/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                15,435 tok/s
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Total Throughput
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                15,416 tok/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                17,434 tok/s
            </td>
        </tr>
    </tbody>
</table>

### NVIDIA H200: The Supercharged Successor

The H200 builds directly on the foundation laid by the H100, bringing meaningful improvements that target one of the biggest bottlenecks in AI and HPC workloads: memory.

While it shares the same Hopper architecture as the H100 and has similar core compute specs, the H200 significantly increases both memory capacity and bandwidth, two factors that become critical when operating at enterprise scale or pushing into exascale territory.

Equipped with **141 GB of HBM3e memory** and **4.8 TB/s of bandwidth**, the H200 nearly doubles the memory of the H100 and delivers about **1.4X faster** data access. Early benchmarks reflect that improvement. In MLPerf testing using Llama 2 70B, the H200 achieved over 31,000 tokens per second – about **45% faster than the H100**.

<img
    src="/uploads/h100vsh200_2.png"
    alt="Image credit: NVIDIA"
    style={{ width: "100%" }}
/>

The H200 retains all of the architectural strengths of the H100, including **fourth-gen NVLink**, **PCIe Gen5**, **MIG support**, and full **Tensor Core and Transformer Engine capabilities**. But for developers and organizations looking to squeeze out more performance without overhauling their infrastructure, the H200 offers a straightforward upgrade path with [measurable gains](https://resources.nvidia.com/en-us-data-center-overview/hpc-datasheet-sc23-h200).

#### To recap, here are a few advantages of the H200:

-   **141 GB of HBM3e memory at 4.8 TB/s** – nearly double the capacity of the H100, with 1.4X more memory bandwidth.

-   **Up to 1.9X the inference performance** of the H100 when handling LLMs, with more efficient processing per token and per inference cycle.

-   **50% reduced energy use and total cost of ownership** (TCO) due to improved thermal management and energy efficiency.

-   **Drop-in compatibility with existing Hopper-based systems**, enabling accelerated performance gains in H100-optimized infrastructure with minimal or no additional changes.

In short, the H200 represents a meaningful leap beyond the H100 in memory capacity, bandwidth, and throughput, bringing a new level of performance to AI and HPC workloads. That comes at a premium price, of course. But if you're weighing the tradeoffs between the two GPUs in terms of specs alone, let's take a side-by-side look at how they compare.

### H100 vs. H200: Key Specs Compared

For reference, here's a quick outline of the various features and specs of the NVIDIA H100 and H200:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Specification
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H100
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H200
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                GPU Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">Hopper</td>
            <td class="border border-gray-300 px-4 py-2 text-center">Hopper</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                GPU Memory
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                80 GB HBM3
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                141 GB HBM3e
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                GPU Memory Bandwidth
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3.35 TB/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                4.8 TB/s
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP64 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                33.5 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                33.5 TFLOPS
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">FP32</td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                67 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                67 TFLOPS
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                TF32 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                989 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                989 TFLOPS
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP16 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1,979 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1,979 TFLOPS
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP8 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TFLOPS
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                INT8 Tensor Core
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3,958 TOPS
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Decoders
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                7 NVDEC, 7 NVJPEG
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                7 NVDEC, 7 NVJPEG
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                L2 Cache
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">50 MB</td>
            <td class="border border-gray-300 px-4 py-2 text-center">50 MB</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Max. Thermal Design Power
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 700 W
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 1000 W
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Multi-Instance GPUs
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 7 MIGs @ 10GB each
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                Up to 7 MIGs @ 18GB each
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Form Factor
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">SXM</td>
            <td class="border border-gray-300 px-4 py-2 text-center">SXM</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Interconnect
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                PCIe 5.0 x16
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                PCIe 5.0 x16
            </td>
        </tr>
    </tbody>
</table>

Ultimately, choosing the right GPU largely depends on your workload demands and performance priorities.

### Use Cases: Which GPU Should You Choose?

While both the H100 and H200 excel in AI and HPC environments, they serve slightly different needs.

#### You should consider the H100 if you prefer:

-   **A proven, high-performance GPU** for training medium to large models and handling intensive AI or HPC workloads.
-   **Use cases involving compute-bound workloads**, where total memory capacity is not as critical.
-   **A time-tested solution** already in widespread production use, balancing performance with lower power and cost.

#### Whereas the H200 may be a better fit if you need:

-   **Greater memory capacity and bandwidth** for running _even larger_ models, handling memory-bound workloads, or scaling LLM inference on the enterprise level.
-   **Higher throughput**, especially for tasks involving extended context windows, multi-modal AI, or very large batch sizes.
-   **A forward-looking upgrade path** with improved energy efficiency and lower TCO, thanks to higher performance per watt.

\------------------------------

At **Vast.ai**, we understand that choosing a GPU isn't only about specs. Often, the decision comes down to access and affordability, as well – especially when we're talking about some of the most powerful and expensive GPUs in existence. (Speaking of, check out our previous posts comparing the [H100 vs. A100](https://vast.ai/article/H100-vs-A100-Comparing-two-Powerhouse-GPUs), and the [H100 vs. L40S](https://vast.ai/article/nvidia-h100-vs-l40s-power-meets-versatility).)

That's why our cloud GPU rental platform is built to offer high-performance compute at a fraction of the typical cost. Vast gives you flexible, cost-effective access to H100s, H200s, and more.

With options for on-demand pricing and budget-friendly spot instances, you can scale up as needed without overspending. On average, **Vast users save 5- 6X on GPU compute** compared to traditional cloud providers.

Explore our platform today and get the power you need, when you need it – with no upfront investment required.


The H100 and H200 GPUs represent the top end of NVIDIA's current lineup for AI and high-performance computing (HPC). Both are designed for serious, large-scale workloads – from training foundation models to running complex scientific simulations.

NVIDIA H100 vs. H200: Two Hopper-based Heavyweights


## Introduction

Rerankers are powerful tools that excel at determining the relevance between pairs of text - whether you're matching search queries to documents, evaluating LLM outputs against prompts, or finding similar content in a database. Unlike simple keyword matching or embedding similarity, these specialized models perform a detailed comparison between inputs, capturing nuanced relationships that simpler methods might miss. They're particularly valuable in RAG (Retrieval Augmented Generation) systems, recommendation engines, and content filtering pipelines, where they can significantly improve quality while requiring minimal computational overhead.

vLLM has recently expanded its capabilities to include reranker model serving, offering compatibility with both OpenAI and Cohere APIs. In this guide, we'll focus on deploying the `BAAI/bge-reranker-base` model - a powerful yet efficient reranker designed for semantic similarity scoring.

Vast.ai provides a marketplace for renting GPU compute power, offering a cost-effective alternative to major cloud providers. Vast has GPU SKU's that you cannot normally find on clouds, including 4000 series GPU's which are cheaper, with a tradeoff of less RAM. Re-ranker models are often quite small so we can take advantage of these extra affordable prices.

In this guide, we will:

1. Set up a Vast.ai instance with the right GPU specifications for serving rerankers
2. Deploy the model using vLLM's optimized inference server
3. Demonstrate two ways to interact with the reranker:
    - Using the Cohere-compatible API for batch reranking
    - Using the OpenAI-compatible API for cross-encoder scoring

This setup provides a production-ready environment for serving reranker models, with the ability to handle both batch reranking requests and individual similarity scoring tasks.

## Setting Up Vast.ai

First, we'll install the Vast.ai API:

```bash
pip install --upgrade vastai
```

Next, we'll set our API key (found on the [Account Page](https://cloud.vast.ai/account/)):

```bash
export VAST_API_KEY="your-key-here"
vastai set api-key $VAST_API_KEY
```

## Choosing Hardware

The `BAAI/bge-reranker-base` model has modest requirements compared to larger language models. Here's what we need:

1. 16GB GPU RAM for:
    - Model weights (~278M parameters)
    - Batch processing overhead
    - System operations
2. Single GPU with Turing architecture or newer
3. Static IP for stable API endpoint
4. At least one direct port for the API server

Note that you could potentially go down to 8GB of GPU RAM for this model for more affordable usages.

We'll use the `vastai search offers` command to find instances that meet our requirements:

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 16 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
rentable = true"
```

## Deploying the Server

Next, we'll copy the `ID` from our chosen instance into the `instance-id` field below along with our `HUGGINGFACE_TOKEN` to rent an instance and deploy `BAAI/bge-reranker-base` using vLLM's OpenAI-compatible server:

```bash
export INSTANCE_ID=<instance-id>
vastai create instance $INSTANCE_ID \
    --image vllm/vllm-openai:latest \
    --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGINGFACE_TOKEN' \
    --disk 40 \
    --args --model BAAI/bge-reranker-base
```

This setup:

-   Uses vLLM's optimized inference server
-   Exposes port 8000 inside the container (Vast.ai will forward this to a different external port)
-   Downloads and serves the `BAAI/bge-reranker-base` model

## Verify the Setup

Before proceeding, verify that your instance is running correctly:

1. Go to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console
2. Wait for the instance to download the image and model (this may take a few minutes)
3. Find your instance's IP address and port from the "Open Ports" panel - you'll see something like `XX.XX.XXX.XX:YYYY -> 8000/tcp` where YYYY is the external port that forwards to the container's port 8000
4. Use these values in `VAST_IP_ADDRESS` and `VAST_PORT` for the test request below

```bash
export VAST_IP_ADDRESS="your-ip-here"
export VAST_PORT="your-port-here"

curl -X POST http://$VAST_IP_ADDRESS:$VAST_PORT/rerank \
    -H "Content-Type: application/json" \
    -d '{
    "model": "BAAI/bge-reranker-base",
    "query": "What is deep learning?",
    "documents": [
        "Deep learning is a type of machine learning"
    ]
    }'
```

## Using the Reranker

vLLM provides two ways to rerank documents using the same underlying model:

1. **OpenAI API Compatible** (`/score`)

    - Raw scoring between query and documents
    - Returns similarity scores for manual sorting
    - Best for custom ranking logic

2. **Cohere API Compatible** (`/rerank`)
    - Direct reranking of documents against a query
    - Returns pre-sorted results
    - Best for quick integration

### OpenAI-Compatible Endpoint

First, we'll create a function to call the `/score` endpoint. We'll need to enter the `IP_ADDRESS` and `PORT` from above:

```python
import requests

IP_ADDRESS = ""
PORT = ""

def openai_score(query,documents):
    base_url = f"http://{IP_ADDRESS}:{PORT}"

    # Format request for score endpoint
    test_request = {
        "model": "BAAI/bge-reranker-base",
        "text_1": query,  # Query is text_1
        "text_2": documents  # Documents are text_2
    }

    # Make request and print raw response
    response = requests.post(f"{base_url}/score", json=test_request)
    print("Status code:", response.status_code)
    print("Raw response:", response.text)

    # If successful, print formatted results
    if response.status_code == 200:
        data = response.json()
        scores = [(doc, item["score"]) for doc, item in zip(documents, data["data"])]
        scores.sort(key=lambda x: x[1], reverse=True)

        print("\nRanked results:")
        for text, score in scores:
            print("\nScore:", score)
            print("Text:", text)
```

We'll start with a simple example to show what happens when we have completely irrelevant documents:

```python
query = "What is Deep Learning?"
documents = [
    "Deep learning is a subset of machine learning that uses neural networks with many layers",
    "The weather is nice today",
    "Deep learning enables computers to learn from large amounts of data",
    "I like pizza"
]
openai_score(query,documents)
```

In this example, we can see how effectively the reranker distinguishes between relevant and irrelevant content:

-   The most relevant document (score ~1.0) provides a direct definition of deep learning
-   The second document (score ~0.18) mentions deep learning but provides less specific information
-   The irrelevant documents about weather and pizza receive nearly zero scores (~0.00004)

```
Ranked results:

Score: 0.99951171875
Text: Deep learning is a subset of machine learning that uses neural networks with many layers

Score: 0.17626953125
Text: Deep learning enables computers to learn from large amounts of data

Score: 3.737211227416992e-05
Text: The weather is nice today

Score: 3.737211227416992e-05
Text: I like pizza
```

Next, we'll run it with a slightly more realistic set of documents to see how the reranker handles more nuanced differences in relevance:

```python
query = "What is Deep Learning?"

documents = [
    "Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input",
    "Machine learning algorithms enable computers to learn from data without being explicitly programmed",
    "The latest smartphone features advanced AI capabilities for photo enhancement",
    "Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection",
    "A neural network is inspired by the biological neural networks that constitute animal brains",
    "Cloud computing provides scalable infrastructure for training deep learning models",
    "The history of artificial intelligence dates back to the 1950s",
    "Deep learning models require significant computational resources and large datasets for training"
]
openai_score(query,documents)
```

The reranker shows sophisticated understanding in this more nuanced example:

1. The comprehensive definition gets a perfect score (1.0)
2. Application examples (computer vision) score moderately well (~0.43)
3. Related concepts (machine learning, neural networks) get low but non-zero scores (~0.03)
4. Infrastructure and historical context receive nearly zero scores (`<0.001`)

This demonstrates how the reranker can capture subtle differences in relevance, not just obvious distinctions between relevant and irrelevant content.

```
Ranked results:

Score: 1.0
Text: Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input

Score: 0.425537109375
Text: Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection

Score: 0.034942626953125
Text: Machine learning algorithms enable computers to learn from data without being explicitly programmed

Score: 0.001674652099609375
Text: A neural network is inspired by the biological neural networks that constitute animal brains

Score: 0.0004711151123046875
Text: The history of artificial intelligence dates back to the 1950s

Score: 0.00026535987854003906
Text: Deep learning models require significant computational resources and large datasets for training

Score: 5.7816505432128906e-05
Text: Cloud computing provides scalable infrastructure for training deep learning models

Score: 3.737211227416992e-05
Text: The latest smartphone features advanced AI capabilities for photo enhancement
```

### Cohere-Compatible Endpoint

The `/rerank` endpoint provides a higher-level interface that:

-   Directly reranks documents against a query
-   Returns pre-sorted results
-   Handles all scoring logic internally
-   Simplifies integration into existing pipelines

First, we'll install Cohere.

```bash
pip install --upgrade cohere
```

We'll then create a function to call the cohere endpoint.

```python
import cohere

IP_ADDRESS = ""
PORT = ""

def cohere_reranker(query,documents):
    base_url = f"http://{IP_ADDRESS}:{PORT}"

    # Initialize v2 client with our endpoint
    co = cohere.ClientV2("sk-fake-key", base_url=base_url)

    result = co.rerank(
        model="BAAI/bge-reranker-base",
        query=query,
        documents=documents
    )

    print("\nRanked results:")
    for doc in result.results:
        print(f"\nScore: {doc.relevance_score}")
        print(f"Text: {doc.document.text}")  # Access text through document.text
```

First, we'll test with our simple example:

```python
query = "What is Deep Learning?"
documents = [
    "Deep learning is a subset of machine learning that uses neural networks with many layers",
    "The weather is nice today",
    "Deep learning enables computers to learn from large amounts of data",
    "I like pizza"
]
cohere_reranker(query,documents)
```

Output:

```
Ranked results:

Score: 0.99951171875
Text: Deep learning is a subset of machine learning that uses neural networks with many layers

Score: 0.17626953125
Text: Deep learning enables computers to learn from large amounts of data

Score: 3.737211227416992e-05
Text: The weather is nice today

Score: 3.737211227416992e-05
Text: I like pizza
```

Notice that the Cohere endpoint produces identical scores to the OpenAI endpoint - this is because both are using the same underlying model, just with different APIs. The key difference is that the Cohere endpoint automatically handles the sorting and formatting of results.

Next, we'll try our more complex example:

```python
query = "What is Deep Learning?"

documents = [
    "Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input",
    "Machine learning algorithms enable computers to learn from data without being explicitly programmed",
    "The latest smartphone features advanced AI capabilities for photo enhancement",
    "Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection",
    "A neural network is inspired by the biological neural networks that constitute animal brains",
    "Cloud computing provides scalable infrastructure for training deep learning models",
    "The history of artificial intelligence dates back to the 1950s",
    "Deep learning models require significant computational resources and large datasets for training"
]
cohere_reranker(query,documents)
```

Output:

```
Ranked results:

Score: 1.0
Text: Deep learning is a subset of machine learning that uses neural networks with multiple layers to progressively extract higher-level features from raw input

Score: 0.425537109375
Text: Deep learning has revolutionized computer vision, enabling tasks like facial recognition and object detection

Score: 0.034942626953125
Text: Machine learning algorithms enable computers to learn from data without being explicitly programmed

Score: 0.001674652099609375
Text: A neural network is inspired by the biological neural networks that constitute animal brains

Score: 0.0004711151123046875
Text: The history of artificial intelligence dates back to the 1950s

Score: 0.00026535987854003906
Text: Deep learning models require significant computational resources and large datasets for training

Score: 5.7816505432128906e-05
Text: Cloud computing provides scalable infrastructure for training deep learning models

Score: 3.737211227416992e-05
Text: The latest smartphone features advanced AI capabilities for photo enhancement
```

Again we see identical scores to the OpenAI endpoint, demonstrating that:

1. Both APIs provide consistent access to the same model
2. The Cohere endpoint offers a more streamlined interface
3. You can choose whichever API better fits your application's needs

The main advantages of the Cohere endpoint are:

-   Pre-sorted results (no need to sort scores manually)
-   Simpler response format
-   Built-in batching support
-   Familiar interface for existing Cohere users

## Key Features

-   **Optimized Inference**: vLLM's server provides efficient batch processing and automatic memory management, making it easy to serve rerankers without dealing with low-level optimizations.
-   **Dual API Support**: Access the model through both OpenAI and Cohere-compatible APIs, allowing flexible integration with existing applications.
-   **Cost-Effective Deployment**: Vast.ai's marketplace lets you access powerful GPUs at a fraction of traditional cloud costs, with easy scaling as your needs grow.

## What's Now Possible

With this setup, you can build:

-   More accurate semantic search systems
-   Better RAG applications with filtered context
-   Content recommendation systems
-   Semantic duplicate detection
-   Document clustering and organization

The combination of vLLM's efficient serving and Vast.ai's affordable GPUs makes it practical to deploy rerankers in production. You can start small and scale up as needed, while maintaining high performance and cost efficiency. Happy ranking!


Deploy rerankers on Vast.ai using vLLM for cost-effective, high-performance semantic search. Supports OpenAI & Cohere APIs for flexible integration.

Serving Rerankers on Vast.ai using vLLM


# Announcing Virtual Machine Rental on Vast.ai

Have you ever thought about what you could accomplish with more flexibility in your GPU rentals? At Vast.ai, we're always looking for ways to innovate and improve our platform, and today we're excited to announce our latest feature: [virtual machine (VM) rental](https://vast.ai/docs/instance-setup/virtual-machines).

We now offer full VMs alongside Docker-based instance rentals. This new offering is designed to support larger customers and more diverse workflows via GPU remote desktop.

Here's everything you need to know to get started!

## Virtual Machine Rental: Expanding the Possibilities

The new VM rental option will help meet the demand for use cases that Docker can't easily support. **VMs also offer features not available in Docker, such as support for init managers and process tracing.**

### Key Benefits of Virtual Machine Support Include:

- **Enhanced Flexibility:** Users can run low level debugging and performance profiling utilities and are not constrained by container environments.
- **Kernel-space Customizations:** Developers gain granular control over their systems, enabling custom driver installations, kernel tweaks, and integration of specialized frameworks.
- **Faster Iteration:** Rather than pre-building and pushing custom Docker images, users can run Docker inside the instance.
- **Improved Security:** Virtual Machines offers an improved  security posture for sensitive workloads.

Host machines can now tap into these advantages, allowing them to attract more renters and capitalize on the increased need for GPU-powered tasks beyond Docker’s capabilities.

### Important VM Differences to Know:

VMs do come with some trade-offs compared to Docker-based instances. For example, they have slower creation and boot times, higher disk overhead, and users will have to deal with a smaller selection of machines and lesser availability of preconfigured templates. In addition, only SSH launch mode is currently supported.

Notably, the on-start script field allows users to specify a script to run on instance start. Unlike in Docker-based instances, the interpreter must be specified by a shebang.

Another difference is that **the cloud copy utility for VMs only allows migrating the entire VM instead of allowing individual folders to be copied.**

Despite these drawbacks, VMs open up new opportunities for both renters and hosts, offering greater flexibility, broader use cases, and the potential for higher earnings.

### Empowering Renters, Rewarding Hosts

One thing that sets us apart from other providers is our focus on cost-efficiency. We strive to keep rental costs low for users across the board – and in fact, Vast.ai GPU rentals are ~3-5X cheaper than current alternatives.

At the same time, we want to provide hosts with the opportunity to earn more.

As our COO, Travis Cannell, explains, "We've always believed that democratizing GPU access means offering maximum flexibility. Virtual Machine support was the natural next step, providing more control, ease of use, and enabling researchers to build and iterate."

With the opportunity for both renters and hosts to benefit, it's equally important to ensure that setup is seamless and operations remain secure. Here's how we make that happen.

## VMs on Vast: Simple Setup and Strong Security

Getting started with VM rental on Vast.ai is simple. **Our hosting software automatically tests host machines for VM compatibility and enables the feature on any machine where it's supported.** (There is no requirement to be VM compatible, however.)

For new host machines, these tests will be run upon installation. For machines configured before the VM feature release, testing for VM compatibility will happen at a time when the machine is unoccupied.

And that's it! Host machines that don't have VM support enabled will simply be hidden from search results when VM-based templates are selected.

Privacy and security are built into the system, as well. Hosts and users cannot view someone else's VM, and no one has access to see what any other VM is doing. Users can rent VMs with peace of mind knowing that their virtual environment is private and secure.

## Getting Started: Creating a VM

Ready to spin up your first VM? It's as easy as selecting the right template!

1.  Make sure you have an SSH key in your account (guide [here](https://vast.ai/docs/instance-setup/ssh)).
2.  Select the **Ubuntu 22.04 VM template** in the interface or [click here](https://cloud.vast.ai/?ref_id=108031&template_id=e2f271ce754aeef772181fbf9c82a354) to load it.
3.  Adjust the disk storage needed and then rent a GPU!

Note: SSH keys cannot be edited on running VMs, and SSH access is currently the only method preconfigured on VM instances. Also, when you select the correct template, it will automatically filter for machines that support VMs and will launch a VM when you rent with that template.

For more detailed information about VM template customization, see our Docs page [here](https://vast.ai/docs/instance-setup/virtual-machines).

## Power Up with VM Rental on Vast.ai

With our new virtual machine rental feature, we're opening up a world of possibilities for both hosts and renters. Whether you need the flexibility of GPU remote desktops for complex workflows or you're looking to maximize your earning potential as a host, Vast.ai has you covered. We're committed to giving you the tools to succeed – easily, affordably, and securely.

Ready to see what you can accomplish? Experience the [power of VM rental](https://vast.ai/docs/instance-setup/virtual-machines) with Vast.ai today!


At Vast.ai, we're always looking for ways to innovate and improve our platform, and today we're excited to announce our latest feature: virtual machine (VM) rental. 

Announcing Virtual Machine Rental on Vast.ai


# Serving vLLM Embeddings on Vast.ai

## Background:

[vLLM](https://github.com/vllm-project/vllm) is a developer-friendly framework for serving large language models. It is very fast and focused on high throughput serving multiple simultaneous calls for production use cases. `vLLM` is now more flexible than ever as it also supports embedding models. This brings vLLM's dynamic batching and Paged Attention to embedding models for much faster throughput, all from the docker image that developers are used to.

This guide will show you how to setup vLLM to serve embedding models on Vast.

## Setup and Querying

First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

To serve an embedding model, we'll want a machine with a static IP address + ports to forward. We only will need a single GPU, which can have a small amount of RAM since embedding models are small. We'll prefer a modern GPU so we will add a filter for a compute cap of more than 800.

```bash
vastai search offers 'compute_cap >= 800 gpu_ram >= 24 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.4'
```

## Deploying the Image:

The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below.

### Hosting a Single Embedding Model:

For now, we'll host just one embedding model.
The easiest way to deploy a single model on this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below.

We particularly need `v2` so that we use the correct version of the api, `--port 8000` so it serves on the correct model, and `--model-id intfloat/e5-mistral-7b-instruct` to serve the correct model.

```bash
vastai create instance <instance-id> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 40 --args --model intfloat/e5-mistral-7b-instruct
```

## Connecting and Testing:

To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/serve_vllm_embeddings_instance_view.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/serve_vllm_embeddings_ip_address_view.png)

## Connecting with the python requests library

```
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'intfloat/e5-mistral-7b-instruct',
    "input": ["San francisco is a"],
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/embeddings', headers=headers, json=json_data)
print(response.content)
```

## Connecting with the OpenAI SDK

The OpenAI SDK also allows for generating embeddings. We can use it as well with the following code.

```bash
pip install openai
```

We will copy over the IP address and the port into the cell below.

```python

from openai import OpenAI

# Modify OpenAI's API key and API base to use Infinity's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
model = "intfloat/e5-mistral-7b-instruct"
embeddings = client.embeddings.create(model=model, input="San francisco is a").data[0].embedding
print("Embeddings:")
print(embeddings)
```

Now we can integrate our embedding server into apps that use the OpenAI SDK.

## Next Steps:

Embeddings are an important part of GenAI applications along with running inference of Generative Models. Now with vLLM, you can run them with the same docker image on Vast for ultimate flexibility.


Serving vLLM Embeddings on Vast.ai


When it comes to high-performance computing (HPC), the NVIDIA H100 is among the best of the best GPUs on the market. Its predecessor, the NVIDIA A100, is also a very impressive GPU in its own right. From enterprise to exascale, these GPUs are enabling groundbreaking advancements in everything from AI research to complex scientific simulations.

Today we're taking a look at how the H100 and A100 compare with each other. Depending on your use case, one or the other might be better suited for your needs and preferences. So let's dive in!

## Introduction to the NVIDIA H100 and A100

As mentioned, the H100 GPU is the successor to the A100. It was specifically optimized for machine learning and designed to [deliver](https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/) an "order-of-magnitude performance leap for large-scale AI and HPC" over the A100, with "substantial improvements in architectural efficiency."

Based on the innovative Hopper architecture, the H100 can speed up large language models (LLMs) by 30X over the A100, according to NVIDIA. It features fourth-gen Tensor Cores and – unlike the A100 – a dedicated Transformer Engine to solve trillion-parameter language models.

Whether it's used in smaller enterprise systems or massive, unified GPU clusters, the H100 delivers efficient scalability thanks to its combination of fourth-gen NVLink (with 900 GB/s of GPU-to-GPU interconnect), NDR Quantum-2 InfiniBand networking (speeding up communication by every GPU across nodes), and PCIe Gen5. With superior FP64 and FP8 performance, the H100 is ideal for next-gen AI workloads.

The A100 GPU is known for its versatility, however. Built on the Ampere architecture, it was groundbreaking when released in 2020, providing up to 20X higher performance over the prior NVIDIA Volta generation. It can efficiently scale up or be partitioned using Multi-Instance GPU (MIG) into seven isolated GPU instances, just like the H100, which is ideal for shifting workload demands.

In short, the A100 excels in both training and inference tasks, and while it may not offer the sheer power of the H100, its robust FP64 performance makes it a great fit for scientific simulations and data analytics.

## Key Differences: H100 vs. A100

The H100 has an updated chip design and various other factors that distinguish it from the A100. The following are a few that are important to consider:

### Memory Bandwidth

The A100 has a memory bandwidth of about 2 TB/s, which is plenty for most use cases; however, the H100 offers an impressive 3.35 TB/s, making it ideal for workloads where data transfer bottlenecks may occur.

### CUDA Cores and Tensor Cores

A higher number of CUDA and Tensor Cores means a GPU can perform more parallel computations simultaneously. With 14,592 CUDA Cores and 456 Tensor Cores, the H100 outperforms the A100’s 6,912 CUDA Cores and 432 Tensor Cores, delivering superior performance in applications that can leverage increased parallelism.

### Confidential Computing

The H100's Hopper architecture introduced NVIDIA's new confidential computing environment. It protects data-in-use by performing computation in a trusted execution environment (TEE) that is both hardware-based and attested. This means you essentially have absolute security for use cases where security, privacy, and regulatory compliance are paramount.

### Tensor Memory Accelerator (TMA)

Another feature of the H100's Hopper architecture is the TMA unit, which can transfer large blocks of data efficiently between global memory and shared memory. It supports asynchronous copies between thread blocks in a cluster as well, optimizing data movement and memory management for tensor operations. Ultimately, this enhances the efficiency and speed of large-scale AI and deep learning tasks.

### MIG Capability

The Multi-Instance GPU (MIG) feature is present in both the H100 and A100, providing flexible workload management so you can run multiple LLMs in parallel, for instance. The H100's MIG capability is more robust, however, with better resource allocation and more memory and compute power to split.

## Performance Benchmarks

According to NVIDIA's own benchmarks, the H100 offers up to 30X better inference performance than the A100. In another test where the A100 and H100 were used in clusters to train two large language models (LLMs), the H100 demonstrated remarkable speed improvements. The H100's new NVLink Switch System provided an even bigger boost, enabling the H100 cluster to train the models up to nine times faster than the A100 cluster:

![H100 vs A100 Speed Comparison](/uploads/Speed-up-over-A100.png)

## Choosing the Right GPU

So what does all this mean for you? The best GPU for your needs will depend on numerous factors. Here's a quick outline of the various features and specs of the NVIDIA A100 and H100:

<table class="w-full border-collapse border border-gray-300">
  <thead>
    <tr class="bg-gray-100">
      <th class="border border-gray-300 px-4 py-2 text-left font-semibold">Feature</th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">A100</th>
      <th class="border border-gray-300 px-4 py-2 text-center font-semibold">H100</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">GPU Architecture</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Ampere</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Hopper</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">GPU Memory</td>
      <td class="border border-gray-300 px-4 py-2 text-center">40 or 80 GB HBM2e</td>
      <td class="border border-gray-300 px-4 py-2 text-center">80 MB HBM3</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">GPU Memory Bandwidth</td>
      <td class="border border-gray-300 px-4 py-2 text-center">1.6 to 2 TB/s</td>
      <td class="border border-gray-300 px-4 py-2 text-center">3.35 TB/s</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">CUDA Cores</td>
      <td class="border border-gray-300 px-4 py-2 text-center">6912</td>
      <td class="border border-gray-300 px-4 py-2 text-center">14,592</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">FP64 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">9.7</td>
      <td class="border border-gray-300 px-4 py-2 text-center">33.5</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">FP32 TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">19.6</td>
      <td class="border border-gray-300 px-4 py-2 text-center">67</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">TF32 Tensor Core Flops*</td>
      <td class="border border-gray-300 px-4 py-2 text-center">156 | 312</td>
      <td class="border border-gray-300 px-4 py-2 text-center">378 | 756</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">FP16 Tensor Core Flops*</td>
      <td class="border border-gray-300 px-4 py-2 text-center">312 | 624</td>
      <td class="border border-gray-300 px-4 py-2 text-center">756 | 1513</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">FP8 Tensor Core TFLOPS</td>
      <td class="border border-gray-300 px-4 py-2 text-center">N/A</td>
      <td class="border border-gray-300 px-4 py-2 text-center">3958</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">Peak INT8 TOPS*</td>
      <td class="border border-gray-300 px-4 py-2 text-center">624 | 1248</td>
      <td class="border border-gray-300 px-4 py-2 text-center">3958</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">Media Engine</td>
      <td class="border border-gray-300 px-4 py-2 text-center">0 NVENC<br>5 NVDEC<br>5 NVJPEG</td>
      <td class="border border-gray-300 px-4 py-2 text-center">0 NVENC<br>7 NVDEC<br>7 NVJPEG</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">L2 Cache</td>
      <td class="border border-gray-300 px-4 py-2 text-center">40 MB</td>
      <td class="border border-gray-300 px-4 py-2 text-center">50 MB</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">Power</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Up to 400 W</td>
      <td class="border border-gray-300 px-4 py-2 text-center">Up to 700 W</td>
    </tr>
    <tr class="bg-gray-50">
      <td class="border border-gray-300 px-4 py-2 font-semibold">Form Factor</td>
      <td class="border border-gray-300 px-4 py-2 text-center">SXM4 - 8-way HGX</td>
      <td class="border border-gray-300 px-4 py-2 text-center">SXM5 - 8-way HGX</td>
    </tr>
    <tr>
      <td class="border border-gray-300 px-4 py-2 font-semibold">Interconnect</td>
      <td class="border border-gray-300 px-4 py-2 text-center">PCIe 4.0 x16</td>
      <td class="border border-gray-300 px-4 py-2 text-center">PCIe 5.0 x16</td>
    </tr>
  </tbody>
</table>

\*Without and with structured sparsity.

---

## Use Cases

When it comes to use cases, you may want to consider some general guidelines.

### The A100 is well suited for:

-   Versatile AI workloads including training and inference.
-   Applications needing a balance of performance and cost-efficiency.
-   Scenarios where ultra-high precision isn't the priority.

### The H100 is ideal for:

-   Cutting-edge AI training, large-scale simulations, and other HPC workloads.
-   Environments where computational speed and memory bandwidth are critical.
-   Future-proofing investments in AI infrastructure.

---

When deciding between the A100 and H100, consider your specific workload requirements. If you need top-tier double-precision performance and superior memory bandwidth, or you're dealing with next-gen HPC at datacenter scale and trillion-parameter AI, the H100 is the clear winner. For a more versatile and cost-effective solution that still delivers powerful AI performance, the A100 is a solid choice.

At Vast.ai, we understand the importance of finding the right GPU for your needs as well as your budget. Our cloud GPU rental marketplace offers a variety of machines at the lowest prices possible, saving you 5-6X on GPU compute. Whether you prefer on-demand pricing or interruptible instances through spot auction-based pricing, we've got you covered.

We invite you to discover what Vast can do for you and accelerate your projects with the right GPU today!


Explore the differences between NVIDIA's H100 and A100 GPUs, from architectural advancements to performance benchmarks.

H100 vs A100: Comparing Two Powerhouse GPUs


# Serving Online Inference with LMDeploy on Vast.ai

## Background
LMDeploy is an open-source framework for Large Language Model inference. It is particularly good at high-throughput serving for multi-user or high-load use cases and is one of the most popular serving frameworks today.

LMDeploy provides an OpenAI-compatible server, which means that it can be integrated into applications that use the OpenAI API. This makes it easy to switch from using OpenAI's services to running your own models on more affordable compute.

This guide will show you how to set up LMDeploy to serve an LLM on Vast. We reference a notebook that you can use [here](https://bitbucket.org/jsbcannell/workspace/snippets/EdjpXL#file-serving_lmdeploy_on_vast.json).

### Performance Benefits:
LMDeploy is specifically focused on maintaining throughput while under higher loads. They include [benchmarks](https://github.com/InternLM/lmdeploy?tab=readme-ov-file#performance) where they compare LMDeploy to vLLM on throughput.
![Benchmark](/uploads/lmdeploy_benchmarks.png)
Here we can see LMDeploy's superior performance in the amount of requests per second that it can handle. This is important for reducing the costs of your deployment, and handling traffic spikes more smoothly.

Combining the performance benefits of LMDeploy with accessing affordable compute on Vast enables developers to ship AI applications quickly, affordably, and with low latency.

## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving an LLM, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU with decent RAM since we're going to serve a single small model. We'll also search for cuda drivers 12.0 or higher for stability. We will query the vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.0'
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 
```bash
vastai create instance <instance-id> \
  --image openmmlab/lmdeploy:latest \
  --env '-p 8000:8000' \
  --disk 40 \
  --entrypoint "lmdeploy serve api_server internlm/internlm2_5-7b-chat --model-name internlm/internlm2_5-7b-chat --server-port 8000" \
```

This command sets up LMDeploy to serve the `internlm/internlm2_5-7b-chat` model on internal port 8000.

## Connecting and Testing
To connect to your instance, you'll need to get the IP address and port number. Once your instance is done setting up, you should see something like this in the Vast.ai interface:
![Instance_view](/uploads/instance_view_lmdeploy.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_lmdeploy.png)

We will copy over the IP address and the port into the cell below.

``` bash
curl -X POST http://<Instance-IP-Address>:<Port>/v1/completions \
     -H "Content-Type: application/json" \
     -d '{"model": "internlm/internlm2_5-7b-chat", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

In the [notebook](https://bitbucket.org/jsbcannell/workspace/snippets/EdjpXL#file-serving_lmdeploy_on_vast.json) we include ways to call this model with requests or OpenAI

### Running with OpenAI
We can interact with this model via the OpenAI SDK as well:
``` bash
pip install openai
```

``` python
from openai import OpenAI

# Modify OpenAI's API key and API base to use lmDeploy 's's API server.
openai_api_key = "PASTE_YOUR_OPEN_API_KEY"

openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
completion = client.completions.create(model="internlm/internlm2_5-7b-chat",
                                      prompt="Hello, how are you?",
                                      max_tokens=50)
print("Completion result:", completion)
```

Now you're all set up with super fast inference!

LMDeploy is an open-source framework for Large Language Model inference.

Posts about: Industry

NVIDIA H100 vs. H200: Two Hopper-based Heavyweights

Serving Rerankers on Vast.ai using vLLM

Announcing Virtual Machine Rental on Vast.ai

Serving vLLM Embeddings on Vast.ai

H100 vs A100: Comparing Two Powerhouse GPUs

Serving Online Inference with LMDeploy on Vast.ai