
# DeepSeek R1: Open-Source Disruptor or Overhyped Upstart?

DeepSeek, a Chinese AI firm, has drawn attention in recent weeks for its release of DeepSeek R1 – a large language model (LLM) that aims to stand shoulder to shoulder with more established players. Its rapid rise has shaken up the AI race, with comparisons to OpenAI's o1 in particular.

## Try it For Yourself on Vast

Our [Ollama + WebUI](https://docs.vast.ai/ollama-webui) guide uses DeepSeek r1:70b as an example. Follow it to try out DeepSeek on our GPU cloud.

## The Disruptive Growth of DeepSeek

DeepSeek's impact isn't limited to industry insiders. It's already gaining massive popularity among everyday users. The DeepSeek mobile app, which provides a chatbot interface for DeepSeek R1, shot to the top of the Apple App Store charts within days of its release on January 20, 2025.

But the biggest reason DeepSeek is [making waves](https://techcrunch.com/2025/01/28/deepseek-everything-you-need-to-know-about-the-ai-chatbot-app/)? **The company has seemingly disrupted the prevailing wisdom that developing cutting-edge LLMs costs a ton of money.** The DeepSeek R1 LLM was developed at a fraction of the cost of comparable models from other companies – reportedly less than $6 million as opposed to hundreds of millions or even billions of dollars.

Another key factor setting it apart: **DeepSeek R1 is free and open source.** Researchers and developers can freely experiment with, modify, and deploy the model without the restrictions of proprietary alternatives.

The meteoric rise of the R1 LLM threw the U.S. financial markets into turmoil, prompting investors to rethink the valuation of companies like NVIDIA and reevaluate the AI prospects of tech giants like Microsoft and Meta. A significant stock market sell-off occurred on January 27, 2025.

Beyond its financial impact, though, what exactly is DeepSeek and how did it develop a model that's turning heads? Let's back up and take a closer look at this new player on the AI scene.

## What Is DeepSeek?

Based in Hangzhou, China, **DeepSeek is an AI development firm that focuses on creating open-source LLMs.** While it operates as an independent lab, DeepSeek is backed by High-Flyer Capital Management, a China-based quantitative hedge fund specializing in AI-driven financial strategies.

From the start, DeepSeek built its own data center clusters to train its generative AI models. U.S. export restrictions on advanced chips had an impact on operations, however. DeepSeek was forced to train its more recent models using NVIDIA H800 chips, a less powerful alternative to the H100 chips available to U.S. companies.

DeepSeek's first model was released in November 2023, and since then the firm has developed several different variations. But DeepSeek only catapulted to global prominence in January 2025, when its R1 model was released – along with a web interface, mobile application, and API access.

## DeepSeek R1: A Challenge to the Status Quo?

The DeepSeek R1 model focuses on logical inference, mathematical reasoning, and real-time problem-solving. Compared to other LLMs, **it uses a more efficient and sparse Mixture-of-Experts (MoE) architecture, where different parts of the model specialize in handling different tasks.** For every input, only a few "expert" neural networks are selected to process each piece of data.

What's really sparked buzz, however, is DeepSeek R1's ability to match performance with OpenAI's latest ChatGPT models across various tasks. So R1 is a more efficient model, developed much more cheaply, with impressive performance. It sounds almost too good to be true...

How was it trained?

According to DeepSeek, the R1 model leveraged the following training innovations:

-   **Reinforcement Learning** – Using a large-scale reinforcement learning approach without supervised fine-tuning, DeepSeek employed techniques like group relative policy optimization (GRPO) to enhance reasoning capabilities.
-   **Reward Engineering** – Instead of relying on traditional neural reward models, DeepSeek created a rule-based reward system that delivers superior results (i.e., by structuring incentives to guide the model's learning during training).
-   **Emergent Behavior Network** – DeepSeek discovered that advanced reasoning patterns can emerge on their own through reinforcement learning, without being explicitly programmed.
-   **Distillation** – DeepSeek used efficient knowledge transfer techniques and distilled knowledge from larger, more complex AI models, allowing R1 to achieve strong performance with reduced computational demands.

Not all of this is new. DeepSeek does present some algorithmic and architectural novelties – such as the sparse MoE design and the use of reinforcement learning to refine synthetic data selection and optimize reasoning pathways during training – but the company isn't necessarily the first to implement these techniques.

Still, it has combined them in a way that makes its R1 model notably efficient and competitive.

## Too Good to Be True?

Unfortunately, there is a catch. **DeepSeek R1 reportedly [relied](https://www.theverge.com/news/601195/openai-evidence-deepseek-distillation-ai-data) on synthetic data from OpenAI's GPT-4o for its training**, pushing some of the computational burden onto the latter, much less efficient model.

So while R1 is touted as being highly efficient, those claims only apply to the model itself, not to the entire process used to train it. The R1 model's lean MoE architecture means it is indeed efficient when running inference, but its training pipeline requires external synthetic data generated by other models like GPT-4o – and that external data generation is computationally expensive.

This means that a significant part of the total AI development cost and compute burden is still being offloaded onto other, more complex models. **It's a cost-saving move for DeepSeek, but it doesn't negate the need for massive specialized compute in the industry overall.**

Time will tell how sustainable DeepSeek's approach is – and what consequences it may have for the company as scrutiny over training methods increases and competition in the AI space continues to grow.

...There may even be national security concerns, as the U.S. government [eyes](https://www.cbsnews.com/news/deepseek-ai-raises-national-security-concerns-trump/) the potential for harmful foreign influence. But that's a topic for another day.

## The Bottom Line on DeepSeek R1

Regardless of how one feels about it, DeepSeek R1 has certainly made a bold entrance into the world. While its efficiency claims come with important caveats, the model remains a fascinating case study in how AI development is evolving.

Plus, its open-source license allows for commercial use, which is a pretty major draw for developers around the world.

## Want to Learn More

DeepSeek R1 is a **massive** model that will not fit on a single 8xH200 instance unless you run a quantized version. The 4bit quantization will run on a 8xH100 or 4xH200 node. The sweet spot might be the 8bit quantized model. If you are interested in trying it out, [reach out to us](https://vast.ai/contact-sales) so we can talk to you about your use case and help get it setup.

As always, we'd be happy to assist you in any way we can! We're here to support your AI experiments, providing the compute power you need to explore, test, and innovate – every day.


DeepSeek R1: Open-source LLM shaking up the AI race. Developed for under $6M, it rivals ChatGPT's performance. Try it on Vast's GPU cloud.

DeepSeek R1: Open-Source Disruptor or Overhyped Upstart?


# Structured Outputs with vLLM and Outlines on Vast.ai

## Introduction

In the world of AI applications, getting consistent, well-formatted responses from language models is crucial. While LLMs are powerful, their free-form outputs can be unpredictable and hard to parse programmatically. This normally prevents LLM's from integrating into existing applications, or giving developers control over their business logic. This is where structured outputs come in - they allow us to enforce specific response formats, making it easier to build reliable AI applications and fit the LLM responses into existing paradigms like Pydantic and JSON schemas.

[vLLM](https://docs.vllm.ai/en/latest/features/structured_outputs.html), combined with the plugin for the Outlines library, provides an elegant solution for generating structured outputs. By enforcing response schemas through Pydantic models or JSON schemas, we can ensure our LLM outputs follow exact specifications. Running this setup on Vast.ai makes it cost-effective and scalable, giving you access to powerful GPUs without the overhead of managing infrastructure.

## Setting Up Your Environment

### Install Vast

Install the Vast SDK

```python
%%bash
pip install --upgrade vastai
```

Set up Vast API Key

```python
%%bash
# Here we will set our api key
export VAST_API_KEY= #Your key here
vastai set api-key $VAST_API_KEY
```

### Choosing the Right Hardware

For optimal performance with vLLM, you'll need:

- GPUs with Turing architecture or newer (compute capability ≥ 7.5)
- At least 32GB GPU RAM for comfortable operation
- A static IP address for stable API access
- At least one direct port that we can forward for the API server

Vast.ai makes it easy to find machines meeting these requirements. Here's how to search for suitable instances:

```python
%%bash
vastai search offers 'compute_cap >= 750 gpu_ram >= 32 num_gpus = 1 static_ip=true direct_port_count >= 1'
```

### Deploying the Server

We'll use vLLM's OpenAI-compatible server, which allows us to use the familiar OpenAI API format and the OpenAI SDK while leveraging vLLM's optimizations.

For this example, the `meta-llama/Meta-Llama-3.1-8B-Instruct` model needs the user to accept the terms of use for the Llama 3.1 model, and use an authentication token. The setup process is otherwise straightforward:

```python
%%bash
vastai create instance <instance-id> \
    --image vllm/vllm-openai:latest \
    --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=<your-token>' \
    --disk 40 \
    --args --model meta-llama/Meta-Llama-3.1-8B-Instruct --guided-decoding-backend outlines
```

## Implementing Structured Outputs

### Install OpenAI and Pydantic

```python
%%bash

pip install --upgrade openai
pip install --upgrade pydantic
```

### Calendar Event Example

Let's look at a practical example - extracting calendar event information from text. First we will set up our connection to the server on our Vast instance. Then, using Pydantic, we can define exactly what fields we expect.

```python
import json

from pydantic import BaseModel
from openai import OpenAI
from typing import List


VAST_IP_ADDRESS=""
VAST_PORT=""


openai_api_key = "EMPTY"
openai_api_base = f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

class CalendarEvent(BaseModel):
    name: str
    date: str
    participants: List[str]
```

The model will now format its responses to match this structure exactly, making it easy to process the output programmatically. Here we use the Pydantic model to define the Calendar Event schema for the response.

```python
completion = client.beta.chat.completions.parse(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "Extract the event information."},
        {"role": "user", "content": "The Science Fair is on Friday. Alice and Bob are going."},
    ],
    extra_body={
        "guided_json": CalendarEvent.model_json_schema()
    }
)

event = json.loads(completion.choices[0].message.content)

print(event)
```

In this example output, the model successfully extracts all required information from the input text:

- The event name ("Science Fair")
- The date mentioned ("Friday")
- Both participants as an array (["Alice", "Bob"])

```python
#output
{'name': 'Science Fair', 'date': 'Friday', 'participants': ['Alice', 'Bob']}
```

### Customer Service Response Schema

For more complex use cases, we can use JSON schemas to define response structures. This is particularly useful for customer service applications where we need consistent response formats:

```python
def generate_structured_response(user_message):
    response_schema = {
        "type": "object",
        "properties": {
            "category": {
                "type": "string",
                "enum": ["Order Issue", "Product Inquiry", "Payment Issue", "General Query"]
            },
            "response": {
                "type": "string"
            },
            "next_steps": {
                "type": "array",
                "items": {
                    "type": "string"
                }
            },
            "follow_up_required": {
                "type": "boolean"
            }
        },
        "required": ["category", "response", "next_steps", "follow_up_required"]
    }

    prompt = f"""
    Given the following user message, generate a structured response in JSON format.

    - Category: Choose from ['Order Issue', 'Product Inquiry', 'Payment Issue', 'General Query']
    - Response: Craft a helpful response
    - Next Steps: Suggest steps the user should take or the support team will take
    - Follow-up Required: Yes or No

    User Message:
    {user_message}

    Respond in JSON format.
    """

    completion = client.chat.completions.create(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": "You are a helpful customer service bot that provides structured JSON responses."},
            {"role": "user", "content": prompt}
        ],
        temperature=0,
        extra_body={
            "guided_json": response_schema
        }
    )

    # Parse the JSON response
    return json.loads(completion.choices[0].message.content)

# User message to be processed
user_message = "I received my order, but a few items were missing. How can I get the missing items?"

# Generate structured response
structured_output = generate_structured_response(user_message)
print(json.dumps(structured_output, indent=2))
```

The response demonstrates the structured output capabilities:

- The issue is correctly categorized as an "Order Issue"
- A professional, detailed response is generated with contact information
- Clear next steps are provided in an array format
- A boolean flag indicates follow-up is required

```python
#output
{
  "category": "Order Issue",
  "response": "We apologize for the inconvenience. To resolve the issue, please contact our customer service team via email at [support@email.com](mailto:support@email.com) or phone at 1-800-SUPPORT. We will assist you in processing a replacement order for the missing items. Please have your order number ready when you reach out to us. We appreciate your patience and apologize again for the missing items.",
  "next_steps": [
    "Contact customer service team via email or phone",
    "Have order number ready for assistance"
  ],
  "follow_up_required": true
}
```

## Key Features

- **Structured Output Control**: vLLM with Outlines enforces strict output formats through JSON schemas and Pydantic models, ensuring consistent and reliable responses from LLMs.
- **OpenAI Compatibility**: Uses the familiar OpenAI API specification, unlocking the ability to integrate this into the OpenAI SDK. This makes it easy to integrate into existing applications or migrate from OpenAI services.

### Future Possibilities

- **Leverage Outputs in Langchain**: Leverage these structured responses in libraries like Langchain or others to build out complex AI systems.
- **Automated Data Processing**: Create pipelines that automatically extract and structure data from unstructured text, emails, or documents using custom schemas. This setup is perfect for batch processing in the background with vLLM's automatic batching.

## Conclusion

Structured outputs with vLLM and Outlines provide a powerful foundation for building reliable AI applications. By running on Vast.ai, you get the benefits of cost-effective GPU access while maintaining full control over your deployment. Whether you're building a customer service bot, data extraction system, or complex AI workflow, this setup gives you the tools you need for consistent, reliable AI outputs.

Start experimenting with structured outputs and see how they can improve your AI applications' reliability and usability.


Ensure structured, reliable AI responses using vLLM and Outlines on Vast.ai. Enforce JSON and Pydantic schemas for predictable outputs, ideal for automation and integration.

Structured Outputs with vLLM and Outlines on Vast.ai


# Generating Videos with Mochi on Vast.ai

## Background

Video generation has become increasingly important in creative and professional workflows. With the advancement of generative AI models, we can now create high-quality videos from text descriptions. Mochi, a new video generation model, allows us to create compelling video content from text prompts. This technology can be used for creating promotional content, educational materials, or artistic expressions without the need for traditional video production equipment.

Mochi, built on a novel diffusion model architecture, offers state-of-the-art capabilities in text-to-video generation. In this guide, we'll show you how to set up and run Mochi for creating videos from text prompts. The notebook demonstrates basic usage including memory optimizations and parameter controls for video length and quality. While this guide focuses on single video generation, the approach can be easily adapted for batch processing or integration into larger content creation pipelines.

With Vast, you can run this computationally intensive model on powerful GPUs at affordable rates, making video generation accessible and cost-effective. You can find the notebook this guide is based on [here](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/k7boE8/6940715c40468472d50053520f45f788465eddbd/files/Video-Gen.ipynb).

## Setting Up the Environment

To deploy on Vast, we recommend using the Vast AI Template for PyTorch: [PyTorch (cuDNN Runtime)](https://cloud.vast.ai?template_id=e6330219a022c9511f290c3c2d44592d). This template includes many required libraries and comes with SSH and JupyterLab pre-configured. While Mochi typically requires around 60GB VRAM for single-GPU operation, we're using the Hugging Face Diffusers implementation which includes optimizations like CPU offloading and VAE tiling that allow it to run on our RTX A6000 (48GB). For production use cases, an H100 or multiple GPUs may be recommended.

First, install the required dependencies:

```bash
pip install -r requirements.txt
```

Then, install the latest development version of diffusers, which includes the MochiPipeline:

```bash
pip install git+https://github.com/huggingface/diffusers.git
```

## Setting up the Pipeline

Let's look at how to set up our video generation pipeline using the Mochi model:

```python
import torch
from diffusers import MochiPipeline
from diffusers.utils import export_to_video
from IPython.display import Video, display

# Load the Mochi pipeline
pipe = MochiPipeline.from_pretrained(
    "genmo/mochi-1-preview",
    variant="bf16",
    torch_dtype=torch.bfloat16
)

# Enable memory optimizations
pipe.enable_model_cpu_offload()
pipe.enable_vae_tiling()
```

## Creating the Video Generation Function

We'll create a function that handles the video generation process:

```python
def generate_video(prompt, output_filename="output.mp4", num_frames=90, fps=30):
    """
    Generates a video given a text prompt using the Mochi Pipeline.
    :param prompt: Text prompt for video generation
    :param output_filename: Name of the output video file
    :param num_frames: Number of frames to generate
    :param fps: Frames per second for the video
    """
    print(f"\nGenerating video for prompt: {prompt}\n")

    with torch.autocast("cuda", torch.bfloat16, cache_enabled=False):
        frames = pipe(prompt, num_frames=num_frames).frames[0]

    # Export to video file
    export_to_video(frames, output_filename, fps=fps)
    print(f"Video saved to: {output_filename}\n")

    # Display the video
    display(Video(output_filename, embed=True))
```

## Example Prompts

Here are some example prompts we can use to generate videos:

```python
# Nature close-up
prompt = "Close-up of a chameleon's eye, with its scaly skin changing color. Ultra high resolution 4k."
generate_video(prompt, "chameleon_eye.mp4")

# Landscape scene
prompt = "A serene waterfall flowing gently into a crystal clear lake surrounded by lush green forest."
generate_video(prompt, "waterfall_forest.mp4")

# Space scene
prompt = "An astronaut floating in space with Earth in the background, ultra-realistic footage, 4k."
generate_video(prompt, "astronaut_space.mp4")

# Nature scene
prompt = "A cute puppy running happily in a field of colorful flowers during sunset."
generate_video(prompt, "puppy_flowers.mp4")
```

## Key Features

- **Memory Optimizations**: The implementation uses CPU offloading and VAE tiling, making it possible to run on GPUs with less VRAM than typically required. This is crucial as the base model needs 60GB VRAM.
- **Handy Convenience Tooling**: Uses the Hugging Face Diffusers library which handles model loading, memory management, and inference, making it easy to get started without dealing with low-level details. The implementation also automatically handles video file writing and display in the notebook.
- **Customizable Generation**: The code supports adjusting key parameters like frame count and FPS, allowing control over video length and quality based on your needs.

## Tips for Best Performance

1. Match your hardware to your needs - the model requires approximately 60GB VRAM for single-GPU operation. For production use cases, Mochi's documentation recommends using H100 GPUs.

2. Be aware of model limitations - the output is 480p resolution and may show warping/distortions in cases with extreme motion.

3. Keep your prompts focused on photorealistic descriptions, as the model is specifically optimized for photorealistic styles rather than animated content.

## Conclusion

This implementation provides a foundation for text-to-video generation using Mochi. You can modify the generation parameters to suit your needs and adapt the code for batch processing of multiple prompts.

With Vast, you can access the GPU power needed for video generation at an affordable price point, making AI video creation accessible and cost-effective. Happy generating!


Generate high-quality videos from text prompts using Mochi on Vast.ai. Leverage powerful GPUs for efficient AI-driven video creation with memory optimizations.

Generating Videos with Mochi on Vast.ai


# Transcribing Audio with Whisper Large V3 on Vast.ai

## Background

With the rise of Generative AI, there have been many advancements in the field of speech recognition. One of the most notable is the development of Whisper, a family of open-source speech recognition models from OpenAI that can transcribe audio in multiple languages. Companies use this technology to transcribe customer service calls, summarize meetings, and create interactive voice assistants. Other applications might be extracting structured data from audio or video about what is being said.

Whisper Large V3, the latest version of the Whisper models, offers improved accuracy and performance over previous versions. In this guide, we'll show you how to set up and run Whisper Large V3 for batch audio transcription. This guide is built for a backend data pipeline, but can be easily modified to interact with users directly.

With Vast, you can run this model on extremely affordable and powerful GPUs to increase the speed of your transcription pipeline. You can see the script that this is based upon [here](https://bitbucket.org/!api/2.0/snippets/jsbcannell/bqz47r/5b696af8939ef28b17138e8162479bb7c03e9705/files/transcriber.py).

## Setting Up the Environment

To deploy on Vast, we will use the Vast AI Template for Pytorch: [PyTorch (cuDNN Runtime)](https://cloud.vast.ai?template_id=e6330219a022c9511f290c3c2d44592d). It has a lot of the libraries that we need, and comes with ssh and jupyterlab out of the box for us. For GPU's the L40s are very cost effective for reasonable batch sizes and the model itself is pretty small. Finally, the dataset is fairly large and gets expanded when creating the dataset splits, so we recommend having 110GB or more of disk space just to be safe.

We will take the requirements.txt file from [here](https://bitbucket.org/!api/2.0/snippets/jsbcannell/k7b4L6/bf17c4b1248ce536e43fe54d85b8efebe59b7b69/files/whisper_vast_requirements.txt) to install the necessary libraries to the template so that we can run the script.

## Running the Script:

### Setting up Transformers

First, let's look at how to set up our transcription pipeline. We'll use Hugging Face's transformers library to load and run the model.

    ```python
    import os
    import torch
    from huggingface_hub import snapshot_download
    from transformers import (
        pipeline,
        AutoModelForSpeechSeq2Seq,
        AutoProcessor,
        logging
    )

    # Configure model details
    MODEL_DIR = "./model"
    MODEL_NAME = "openai/whisper-large-v3-turbo"
    MODEL_REVISION = "41f01f3fe87f28c78e2fbf8b568835947dd65ed9"

    # Create model directory and download model files
    os.makedirs(MODEL_DIR, exist_ok=True)
    snapshot_download(
        MODEL_NAME,
        local_dir=MODEL_DIR,
        ignore_patterns=["*.pt", "*.bin"],
        revision=MODEL_REVISION,
    )
    ```

### Creating the ASR Pipeline

Next, we'll set up our Automatic Speech Recognition (ASR) pipeline. This involves loading the model and processor, and configuring them for optimal performance:

    ```python
    def initialize_asr_pipeline():
        print("Setting up pipeline")

        processor = AutoProcessor.from_pretrained(MODEL_NAME)
        device = "cuda" if torch.cuda.is_available() else "cpu"

        model = AutoModelForSpeechSeq2Seq.from_pretrained(
            MODEL_NAME,
            torch_dtype=torch.float16,
            low_cpu_mem_usage=True,
            use_safetensors=True,
            device_map=device
        )
        model.generation_config.language = "<|en|>"

        # Configure feature extractor
        feature_extractor = processor.feature_extractor
        feature_extractor.sampling_rate = 16000
        feature_extractor.return_tensors = 'pt'

        # Create pipeline
        asr_pipeline = pipeline(
            "automatic-speech-recognition",
            model=model,
            tokenizer=processor.tokenizer,
            feature_extractor=feature_extractor,
            torch_dtype=torch.float16
        )

        return asr_pipeline
    ```

### Loading and Processing Audio Data

We'll use the Hugging Face datasets library to load our audio data. This example uses LibriSpeech, but you can modify it to use your own dataset:

    ```python
    def load_dataset(dataset_name):
        from datasets import load_dataset, Audio

        print("Loading dataset", dataset_name)
        dataset = load_dataset(dataset_name, "clean", split="validation")
        dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
        print("Dataset loaded")
        return dataset["audio"]

    def batch_audio(dataset, batch_size):
        for i in range(0, len(dataset), batch_size):
            print(f"Yielding batch {i}")
            yield dataset[i : i + batch_size]
    ```

### Running the Transcription

Finally, we'll put it all together to transcribe our audio files in batches. Because we defined the functions above, actually running the transcription is very simple.

    ```python
    # Initialize pipeline
    asr_pipeline = initialize_asr_pipeline()

    # Load dataset
    DATASET_NAME = "openslr/librispeech_asr"  # You can change this to your dataset
    dataset = load_dataset(DATASET_NAME)

    # Process batches
    results = []
    for batch in batch_audio(dataset, batch_size=32):
        print("Processing batch")
        with torch.no_grad():
            transcriptions = asr_pipeline(batch)
        print("Batch processed")
        results.extend(transcriptions)

    print(results)
    ```

## Results:

Some of our results are below:

```
'Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.', " Nor is Mr. Quilter's manner less interesting than his matter.", ' he tells us that at this festive season of the year with christmas and roast beef looming before us similes drawn from eating and its results occur most readily to the mind'
```

These can be compared to the original audio dataset [here](https://huggingface.co/datasets/hf-internal-testing/librispeech_asr_dummy). The audio in this dataset is very clear and easy to transcribe.

We can see that the text that is returned from this pipeline is basically perfect for the samples that we show. We would expect that the model's performance would drift if it were run on a private dataset specific to your use case, or if the audio was not as clear as this dataset. Luckily since we have access to the weights, this model can be fine-tuned to your specific use case.

## Key Features

- **Batch Processing**: The code processes audio files in batches of 32 for improved efficiency. This is very important for running this model as it benefits a lot from batching on the GPU.
- **Transformers Pipeline**: This is a very easy way to run the model and handles a lot of boilerplate code to properly prepare the model and the individual samnples for inference. This is extra important as highly performant Whisper serving frameworks aren't released yet.
- **Flexible Dataset Support**: Can work with any dataset hosted on Hugging Face's datasets Hub, with minimal modification for specific dataset schema.

## Tips for Best Performance

1. Adjust the batch size based on your GPU memory to ensure to fully utilize the tensor cores on the GPU.
2. Consider preprocessing your audio files to remove noise or normalize volume
3. Process the audio outputs with an LLM to automatically see if the text is coherent, if it isn't, that might be a good sign that the audio isn't clear.
4. Use these outputs to speed up humans doing the transcription and get a faster transcription with humans in the loop.
5. Leverage the faster and accurately transcribed outputs to fine-tune Whisper to your specific use case, increasing the accuracy and speed of your data flywheel and value to your business.

This implementation provides a solid foundation for batch audio transcription using Whisper Large V3. You can modify the dataset loading code to work with your own audio files, and adjust the batch size based on your available GPU memory.

Now you can see how with Vast, you can save a meaningful amount of money on labor-intensive tasks like transcription with an affordable and powerful GPU. Happy Transcribing!


Transcribe audio efficiently with Whisper Large V3 on Vast.ai. Leverage powerful GPUs for fast, accurate batch processing using Hugging Face’s transformers library.

Transcribing Audio with Whisper Large V3 on Vast.ai


# Vast.ai Highlights: 2024 Round-Up

As we set sail on 2024, we'd like to take a moment to reflect on what's been a landmark year for Vast.ai. From expanding our hardware offerings to improving user experience across the board, we've achieved significant progress in making GPU compute more accessible and efficient for everyone.

Let's take a look back at the highlights of the past year!

## Q1: Setting the Foundation

We kicked off January by introducing our Teams feature, allowing multiple users to manage a single account with customizable roles and permissions. Along with OAuth support for Google and GitHub, this has made account management more flexible and secure than ever.

February brought exciting news for those working with different architectures – we added support for GH200 as well as ARM64 platforms, including the Grace Hopper superchip.

In March, we began beta hosting support for AMD GPUs. We also introduced @Vasti, the Discord bot that is trained on our documentation and can answer questions when it's tagged in a channel or contacted via direct message.

## Q2: Expanding Horizons

May marked a significant expansion in our hardware offerings with the addition of both Radeon and Instinct product lines from AMD. Combined with the existing array of NVIDIA machines available for rent on our marketplace, [this update](https://vast.ai/article/announcing-amd-support) gave users even more options to find exactly the right GPU for their needs.

In June, we eliminated hosting fees entirely. Instead, we transitioned to a more flexible internal surcharge model, making our platform more attractive for hosts while keeping the renter experience unchanged.

## Q3: Powering Innovation

July saw the addition of the H100 NVL to our GPU lineup. This NVIDIA powerhouse, designed specifically for large language model (LLM) deployment, expands the possibilities for both hosts and renters – and its addition represents our commitment to supporting cutting-edge AI development.

In August, we were proud to sponsor [NextEpoch 2024](https://vast.ai/article/NextEpoch-2024-Bringing-ML-to-the-next-generation-of-scientists), contributing both prizes and GPU computing power to support workshop participants as they explored the intersection of machine learning and biological research.

In September, we added to our website resources with the release of a guide on [serving Infinity Embeddings](https://vast.ai/article/serving-vllm-embeddings-on-vast-ai). (It can be found in the [Docs](https://docs.vast.ai/) section of our website, along with many other guides and resources with valuable insights for our users.)

## Q4: Looking to the Future

As we entered the final quarter, October brought the beta release of virtual machine (VM) support running on Kernel Virtual Machine, giving hosts more flexibility in their setups. Later in the year, we rolled out the [official update](https://vast.ai/article/announcing-virtual-machine-rental-on-vast-ai): Vast now offers full VMs alongside Docker-based instance rentals. This offering is designed to support larger customers and more diverse workflows via GPU remote desktop.

Rounding out the year, we added NVIDIA's H200 GPUs to our listings in November, ensuring that our platform remains a go-to destination for users seeking cutting-edge hardware.

Looking back, we're proud of how far Vast.ai has come over the past year. Whether you're a host or a renter, a small startup or an established enterprise, we've worked hard to make our platform better serve your needs.

As we move into 2025, we're excited about what's next! Thank you for being part of our journey this year.


A moment to reflect on what's been a landmark year for Vast.ai

Vast.ai Highlights: 2024 Round-Up


In the high-end GPU market, NVIDIA's H100 accelerator family stands out as some of the most advanced hardware available today. Whether you're training AI models or deploying them, performing scientific simulations, or doing large-scale data analytics, the H100 lineup definitely has the performance you need for enormous workloads.

But which accelerator is right for you? Today we're looking at two options: the NVIDIA H100 NVL and the H100 SXM5. Let's dive into the differences and explore which one might be the better fit for your project needs.

## The H100 NVL: Built for AI Inference at Scale

Announced last year, the NVIDIA H100 NVL is an interesting variant on the H100 PCIe card. At first glance, it's exactly what it looks like: two H100 PCIe cards that come already bridged together.

This massive beast of a card spans four slots – with a TDP of 700W to match its size. Given that it's essentially just two H100s glued together, communication is handled by a pair of PCIe 5.0 x16 slots. It uses three NVLink bridges to connect the pair of H100 GPUs, which deliver 600GB/s of bidirectional bandwidth, or about 4.5x the bandwidth of the dual PCIe interfaces.

NVIDIA states that the H100 NVL is capable of 2.4x to 2.6x the performance of two separate H100s (at least when it comes to FP8 and FP16 workloads) – likely in part because it uses faster HBM3 memory rather than HBM2e. Plus, unlike the standard H100 PCIe GPUs, each card on the H100 NVL has all six stacks of HBM memory enabled instead of just five. The result is that the H100 NVL offers a 17.5% memory increase – and a total of 188GB VRAM (2 x 94GB).

A few other standout features include an HBM3 memory clock at ~5.1Gbps; 6144-bit memory bus width; and memory bandwidth of 2 x 3.9TB/s.

According to NVIDIA, the H100 NVL is ideal for inferencing large language models (LLMs):

"These GPUs work as one to deploy large language models and GPT models from anywhere from five billion parameters to 200 [billion]," [explained](https://www.theregister.com/2023/03/21/nvidia_h100_l4/) Ian Buck, NVIDIA's VP of Accelerated Computing.

So in addition to being able to handle various high-performance computing (HPC) tasks, the H100 NVL is specifically optimized for deploying already-trained neural networks at scale.

## The H100 SXM5: Heavyweight for AI Training

SXM technology is a form factor and interconnect standard primarily used for GPUs in large-scale AI applications and data center environments. Instead of relying on PCIe slots like traditional GPUs, SXM GPUs are directly socketed into the motherboard. This allows for faster, high-bandwidth connections as well as better power delivery and cooling solutions – crucial for high-end GPUs, particularly in dense server environments.

Ultimately, this setup helps ensure that SXM GPUs perform efficiently even under the most intense workloads. For instance, NVIDIA's H100 SXM5 is designed for AI training first and foremost – a powerhouse GPU that is especially well suited to training foundational models.

With 528 tensor cores, 80 GB of HBM3 memory, and a memory bandwidth of 3.35 TB/s, the SXM5 is designed for the most demanding machine learning and deep learning tasks. It also benefits from NVLink interconnects that allow up to 900 GB/s of GPU-to-GPU bandwidth, making it ideal for multi-GPU configurations in large-scale AI model training.

Like the H100 NVL, the SXM5 features HBM3 memory. However, the SXM5’s form factor is a bit more restrictive. With each unit pulling 700W, running these systems requires a lot of power and cooling, which can pose a challenge for some datacenters. Most colocation racks have a power capacity of 6-10KW, meaning systems with four or more SXM5s can push infrastructure to its limits.

(The 700W H100 NVL is comparatively easier to accommodate. A single socket, dual H100 NVL configuration – with four GH100 dies – would likely require around 2.5KW.)

A direct comparison of specs would be helpful in deciding which accelerator best suits your needs, especially when balancing power efficiency, infrastructure requirements, and the performance demands of your workloads.

## Which GPU is Right for You?

Here's an outline of some of the features and specs of the H100 NVL and the SXM5, as well as the H100 PCIe for good measure:

<table class="w-full border-collapse border border-gray-300">
    <thead>
        <tr class="bg-gray-100">
            <th class="border border-gray-300 px-4 py-2 text-left font-semibold">
                Feature
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H100 NVL
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H100 PCIe
            </th>
            <th class="border border-gray-300 px-4 py-2 text-center font-semibold">
                H100 SXM5
            </th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Architecture
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">Hopper</td>
            <td class="border border-gray-300 px-4 py-2 text-center">Hopper</td>
            <td class="border border-gray-300 px-4 py-2 text-center">Hopper</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                ~5.1 Gbps HBM3
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3.2 Gbps HBM2e
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                5.23 Gbps HBM3
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Bus Width
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                6144-bit
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                5120-bit
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                5120-bit
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Memory Bandwidth
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2 x 3.9 TB/s
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">2 TB/s</td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                3.35 TB/s
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP32 CUDA Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2 x 16896
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">14592</td>
            <td class="border border-gray-300 px-4 py-2 text-center">16896</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Tensor Cores
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2 x 528
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">456</td>
            <td class="border border-gray-300 px-4 py-2 text-center">528</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Boost Clock
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1.98 GHz
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1.75 GHz
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1.98 GHz
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">VRAM</td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2 x 94 GB (188 GB)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">80 GB</td>
            <td class="border border-gray-300 px-4 py-2 text-center">80 GB</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                INT8 Tensor
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2 x 1980 TOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1513 TOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                1980 TOPS
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP16 Tensor
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2 x 990 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                756 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                990 TFLOPS
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP32 Tensor
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2 x 495 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                378 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                495 TFLOPS
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                FP64 Tensor
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2 x 67 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                51 TFLOPS
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                67 TFLOPS
            </td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Interconnect
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                NVLink 4 (600 GB/s)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                NVLink 4 (600 GB/s)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                NVLink 4, 18 Links (900 GB/s)
            </td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">GPU</td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2 x GH100
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">GH100</td>
            <td class="border border-gray-300 px-4 py-2 text-center">GH100</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Transistor Count
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2 x 80B
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">80B</td>
            <td class="border border-gray-300 px-4 py-2 text-center">80B</td>
        </tr>
        <tr class="bg-gray-50">
            <td class="border border-gray-300 px-4 py-2 font-semibold">TDP</td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                700-800W
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">350W</td>
            <td class="border border-gray-300 px-4 py-2 text-center">700W</td>
        </tr>
        <tr>
            <td class="border border-gray-300 px-4 py-2 font-semibold">
                Interface
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                2 x PCIe 5.0 (Quad Slot)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">
                PCIe 5.0 (Dual Slot)
            </td>
            <td class="border border-gray-300 px-4 py-2 text-center">SXM5</td>
        </tr>
    </tbody>
</table>

The choice between the NVIDIA H100 NVL and SXM5 ultimately depends on your specific project requirements. The NVL offers a more versatile and accessible option for various applications, while the SXM5 provides exceptional performance and scalability for demanding workloads.

If you prioritize flexibility and ease of integration, the NVL could be the better choice. However, if you need the highest possible performance and are willing to invest in a more specialized setup, the SXM5 is your ideal solution. Carefully consider your project's needs, budget, and infrastructure before making a decision.


A comparison of NVIDIA's high-end GPU options, the H100 NVL and H100 SXM5, for AI inference, AI training, and high-performance computing.

Posts about: Vast.ai

DeepSeek R1: Open-Source Disruptor or Overhyped Upstart?

Structured Outputs with vLLM and Outlines on Vast.ai

Generating Videos with Mochi on Vast.ai

Transcribing Audio with Whisper Large V3 on Vast.ai

Vast.ai Highlights: 2024 Round-Up

H100 NVL vs. SXM5: NVIDIA's Supercomputing GPUs