
You wouldn't buy a car without taking it for a test drive, so why deploy a large language model (LLM) without benchmarking it first?

When you're running LLMs at scale, you need more than just theoretical performance data. You're going to want to know how they actually perform under real conditions – throughput, latency, and hardware efficiency all matter. Skipping this step can result in higher costs and wasted resources.

Benchmarking early helps you avoid inefficiencies down the line, and it shouldn't take hours to get started. That's where **vLLM** comes in.

### A Smarter Way to Serve and Benchmark LLMs

vLLM is an open-source library that optimizes the inference and serving of LLMs, and it [simplifies](https://blog.vllm.ai/2023/06/20/vllm.html) the benchmarking process with an efficient architecture built around the attention algorithm PagedAttention. This mechanism reduces memory waste and improves throughput by storing attention data non-contiguously.

When fast and efficient LLM inference is crucial, understanding your model's performance starts with benchmarking. With vLLM, you can spin up a high-performance inference server and start collecting those benchmarks in minutes.

To make things even easier, you don't need to spin up your own infrastructure from scratch. Vast.ai provides an ideal platform to leverage vLLM, offering high-performance GPUs at a fraction of the cost of traditional cloud providers. With Vast, it's easy to find the right hardware for your model, and with simple Docker integration, you can get up and running with vLLM quickly.

This guide demonstrates how to benchmark any LLM using vLLM on Vast.ai, in just a few minutes.

### Getting Started: Benchmarking LLMs with vLLM

This guide walks you through how to **serve a large language model using vLLM** and run a benchmark test to evaluate performance. It’s using a **H100** **GPU server from [Vast.ai](https://vast.ai/)** with the **"PyTorch" template**. In this example, we’ll use the model `meta-llama/Llama-3.1-8B-Instruct`, but you can use **any Hugging Face-supported model** compatible with vLLM.

**1\. Install required Python packages**

First, install the **vLLM** library along with **Pandas** and Hugging Face’s **datasets** library. For example, you can use pip in your Python environment:

```shell
pip install vllm pandas datasets
```

This installs vLLM and the other libraries for data handling.

**2\. Log in to Hugging Face**

To access the Meta Llama 3.1 model, you need a Hugging Face API token. Then run:

```shell
huggingface-cli login
```

This will prompt you to paste your Hugging Face **access token** (from your Hugging Face account settings). After entering the token, the CLI saves it so vLLM and other tools can download the model.

## **3\. Start the vLLM server**

Now start the vLLM server hosting the Llama 3.1 8B Instruct model. vLLM’s OpenAI-compatible server listens on localhost port 8000 by default. Use the `vllm serve` command with **nohup** to run it in the background and redirect output to a log file. For example:

```shell
nohup vllm serve meta-llama/Llama-3.1-8B-Instruct > vllm.log 2>&1 &
```

This command launches the server with the specified model and writes all output (including any Uvicorn logs) to `vllm.log`. The server will run in the background. (The vLLM docs give a similar example: `vllm serve <model-name>`) By default, this serves the model on [`http://localhost:8000`](http://localhost:8000).

## **4\. Test the server with a curl request**

Once the server is running, you can test it by sending a chat completion request to the API. For example, use `curl` to call the chat endpoint (`/v1/chat/completions`) with a simple prompt:

```shell
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
          "model": "meta-llama/Llama-3.1-8B-Instruct",
          "messages": [
              {"role": "system", "content": "You are a helpful assistant."},
              {"role": "user", "content": "What is the capital of France?"}
          ]
        }'
```

This sends a JSON payload to the server. The `model` field names our loaded model, and `messages` contains a chat-style conversation. The server will respond with a JSON completion. (The official vLLM docs show a similar curl example for chat completions)

You should see a JSON response in the terminal with the model’s answer.

## **5\. Clone the vLLM repository**

To access the benchmark script, clone the vLLM source repository:

```shell
git clone https://github.com/vllm-project/vllm.git
cd vllm/benchmarks
```

This pulls the latest vLLM code. Now you have the `benchmark_serving.py` script and related files in the `benchmarks` folder.

## **6\. Run the benchmark script**

Finally, run the built-in benchmark script to measure serving performance. The `benchmark_serving.py` tool sends many requests to the running server and reports throughput/latency. For example, you might run:

```shell
  python3 vllm/benchmarks/benchmark_serving.py \
    --backend vllm \
    --base-url http://127.0.0.1:8000 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --tokenizer meta-llama/Llama-3.1-8B-Instruct \
    --dataset-name random \
    --random-input-len 128 \
    --random-output-len 128 \
    --num-prompts 80 \
    --max-concurrency 4 \
    --temperature 0.7

```

-   `--backend vllm`: Use the vLLM backend to send benchmark requests.

-   `--base-url http://127.0.0.1:8000`: Address of your running vLLM server.

-   `--model meta-llama/Llama-3.1-8B-Instruct`: Name of the model being benchmarked.

-   `--tokenizer meta-llama/Llama-3.1-8B-Instruct`: Tokenizer used for encoding/decoding; usually the same as the model.

-   `--dataset-name random`: Use synthetic/randomly generated prompts instead of a real dataset.

-   `--random-input-len 128`: Length (in tokens) of each generated input prompt.

-   `--random-output-len 128`: Desired length (in tokens) of the model's output.

-   `--num-prompts 80`: Total number of prompts to send during the test.

-   `--max-concurrency 4`: Number of prompts sent in parallel (simulates batch size).

-   `--temperature 0.7`: Controls randomness in generation; higher \= more creative.

This example command (modeled on published vLLM benchmarks) will run 80 chat requests against your server with concurrency 4\. After it finishes, the script prints stats like throughput and latency.

By following these steps and citing vLLM’s documentation and examples, you can serve `Llama-3.1-8B-Instruct` with vLLM and evaluate its performance. Adjust parameters (batch size, number of tokens, etc.) in the commands above as needed for your environment.

### Conclusion

And that's it\! A fast, easy-to-use LLM benchmarking workflow powered by vLLM. In just ten minutes, you can gather meaningful performance data that will help you make smarter decisions about which models (and hardware) are right for your specific project and workload.

Need powerful GPUs to run your tests? Vast.ai offers on-demand access to H100s, RTX 5090s, and other high-performance options at a fraction of the typical price – so you can save 5-6X on GPU compute. Get started today and benchmark on your terms\!


Benchmark LLMs in minutes using vLLM on Vast.ai. Serve models like Llama 3.1-8B, test performance, and optimize cost with fast, efficient, real-world insights.

Posts about: Benchmark

How to Benchmark An LLM with vLLM in 10 Minutes