Pyannote Audio and NVIDIA’s Sortformer enhance Whisper’s speech recognition by adding speaker diarization. We’ll run them on VAST.ai to create speaker-attributed transcripts.

Implementing Speech-to-Text with Speaker Diarization: Comparing Pyannote and Sortformer on VAST.ai


The long-awaited NVIDIA GeForce RTX 5090 is here – but good luck getting your hands on one. From shipping delays to hardware defects, this GPU launch has been far from smooth.

Controversy aside, it's exciting to finally have confirmed performance details for this high-end graphics card. You can [try it now](https://cloud.vast.ai/?gpu_option=RTX%205090) for yourself on Vast.ai. Here's everything you need to know about the RTX 5090...

### RTX 5090: The Performance King?

As NVIDIA's new flagship GPU, the [RTX 5090](https://vast.ai/pricing/gpu/RTX-5090) is undoubtedly an impressive machine. Is it worth the $1999 price tag, though? That depends on more than just raw power – let's take a look.

#### Official Specs

For starters, here are some of the official specs for the RTX 5090:

-   **Architecture:** Powered by NVIDIA's cutting-edge Blackwell architecture.
-   **Gaming Chip:** Featuring the next-gen GB202 gaming chip.
-   **Memory:** 32GB of GDDR7 memory with a blazing-fast 1,792GB/sec bandwidth, supported by a super-wide 512-bit memory interface for even faster data transfer.
-   **CUDA Cores:** 21,760 cores, ideal for high-performance graphics and AI workloads.
-   **L2 Cache:** Expanded to 98MB, up from 73MB on the RTX 4090, improving memory efficiency and reducing latency.
-   **Display Support:** PCIe Gen 5 with DisplayPort 2.1b connectors supporting up to 8K at 165Hz.
-   **Power Requirements:** 575W total graphics power (TGP), with a 1000W power supply recommended for optimal performance.

Another bonus? The RTX 5090 has a redesigned **Founders Edition** with a two-slot design and a dual flow-through fan configuration – so it's compact enough for small form factor PCs, a remarkable shift from its predecessors.

For reference, here's a quick overview of the RTX 5090 specs:

| Spec                           | RTX 5090                       |
| :----------------------------- | :----------------------------- |
| **CUDA Cores**                 | 21,760                         |
| **Ray Tracing Cores**          | 170 (4th-gen)                  |
| **Tensor Cores**               | 680 (5th-gen)                  |
| **Base Clock**                 | 2,017 MHz                      |
| **Boost Clock**                | 2,407 MHz                      |
| **L2 Cache**                   | 98,304KB                       |
| **ROPs**                       | 176                            |
| **VRAM**                       | 32GB 28Gbps GDDR7              |
| **VRAM Interface**             | 512-bit                        |
| **Memory Bandwidth**           | 1,792 GB/s                     |
| **Interface**                  | 16x PCIe 5.0                   |
| **Power Connectors**           | 1 x 16-pin / 4 x 8-pin adapter |
| **Total Graphics Power (TGP)** | 575W                           |

(For more specifications – and to see how the RTX 5090 stacks up against its younger siblings in the RTX 50-series lineup as well as the RTX 4090 – check out the chart at the bottom of this page.)

#### AI and Professional Workloads

Beyond gaming, the RTX 5090 is well suited for high-performance AI and professional workloads. Its **680 Tensor cores** and **32GB of GDDR7** memory efficiently accelerate deep learning, 3D rendering, and other compute-intensive tasks. With **98MB of L2 cache** and a **512-bit memory interface**, the RTX 5090 delivers faster data access and reduced latency – crucial for AI training, scientific computing, and content creation.

The **5th-gen Tensor cores** boost AI inference efficiency in frameworks like TensorFlow and PyTorch, while **4th-gen RT cores** enhance real-time ray tracing for animation, visual effects, CAD work, and more.

For demanding AI and video workloads, the RTX 5090 more than meets the challenge. In Procyon's AI XL (FP16) test, the RTX 5090 was **40% faster** than the RTX 4090, and in PugetBench's DaVinci Resolve video processing test, it was **12% faster**.

However, cutting-edge performance doesn't necessarily mean a flawless launch\! Early adopters are already encountering serious issues in some cases.

#### Next-Gen Gaming Features

The biggest leap in technology for this GPU comes from DLSS 4's new **Multi Frame Generation**, an AI-driven frame interpolation technique exclusive to the RTX 50 series. It uses AI to generate up to three additional frames between traditionally rendered ones, significantly boosting frame rates with minimal visual artifacts.

Further, the RTX 5090 introduces a range of AI-driven enhancements in what's called **neural rendering**, designed to push in-game realism to new levels. **RTX Neural Shaders** improve texture compression and deliver film-quality lighting and shading in real time, while **RTX Neural Faces** uses generative AI to create lifelike skin, hair, and facial details.

Building on the power of DLSS 4, these neural rendering techniques integrate directly into the graphics pipeline, leveraging NVIDIA’s Tensor cores to enhance visual fidelity. Though game support is still in development, early demos – such as _Half-Life 2 RTX_ – [showcase](https://www.pcgamesn.com/nvidia/geforce-rtx-5090-review) the potential for deeper shadows, realistic translucency in materials, and an overall richer environment.

Another breakthrough is **Mega Geometry**, a feature that uses RT cores to significantly boost the number of triangles in ray-traced scenes. This allows game engines to maintain full geometric detail without sacrificing performance, resulting in more realistic object depth, shadows, and fine details.

The above features position the RTX 5090 as a powerhouse for next-generation gaming visuals... but there's a catch.

##### Caveat: Performance Claims vs. Real-World Results

The way that NVIDIA presents its performance numbers can be misleading. The company leans heavily on Multi Frame Generation-boosted FPS (frames per second) to compare these new GPUs against older ones.

Frame Generation and Multi Frame Generation (MFG) are powerful tools, but they aren't universal fixes. Their impact varies by game – some titles see massive FPS gains with little visual trade-off, while others may introduce artifacts or latency.

In addition, **not all games will support these features**, as DLSS adoption depends on individual developers. A GPU's raw performance still matters. MFG-inflated benchmarks can make performance gains seem more significant, but Frame Generation still needs a strong base frame rate to be effective. In CPU-limited games, or titles that don't support DLSS 4, the real-world performance gap may be far smaller.

Ultimately, while the RTX 5090 excels at AI-driven rendering, **real-world results will depend on game support and how well these features are implemented.**

### Early Defects and Controversies

#### Melting Power Connectors

Reports of **melted power connectors** on the RTX 5090 Founders Edition have [surfaced](https://www.theverge.com/news/609207/nvidia-rtx-5090-power-connector-melting-burning-issues). A number of users have experienced burnt plastic at both the GPU and PSU ends – with user error and third-party cables evidently not to blame.

NVIDIA introduced the updated 12V-2x6 power connector for the RTX 50 series following melting and overheating issues with the RTX 40-series GPUs. The new connector features shorter sensing pins and longer conductor terminals to improve connection reliability. Users can still use existing 12VHPWR cables with the RTX 50 series; however, the RTX 5090 can draw up to 575 watts of power – which is dangerously close to the cable's 600-watt rating.

PSU manufacturers have taken precautions, such as adding visual indicators to ensure a secure connection, but it remains unclear whether this fully resolves the risk of overheating. So far, NVIDIA has declined to comment on the reports.

#### Chip-Level Fault

One of the more unexpected issues plaguing the RTX 5090 launch is what's now being called the **"missing ROPs" defect.** NVIDIA has officially [confirmed](https://www.techradar.com/computing/gpu/some-lucky-rtx-5080-buyers-may-not-be-so-lucky-after-all-as-nvidia-confirms-rare-chip-fault-also-applies-to-this-gpu-as-well-as-the-rtx-5090-and-5070-ti) that a small number of RTX 5090, RTX 5090D, and RTX 5070 Ti cards were shipped with fewer ROPs (Render Output Units) than they should have. These ROPs are basically rendering pipelines for 3D graphics. Having fewer of them slows down PC gaming performance.

While this defect reportedly only impacts an estimated 0.5% of units (or around 1 in 200), affected cards have an average 4% drop in graphical performance. Since it's a hardware issue, it can't be fixed via BIOS or driver updates. However, it has no impact on AI and compute tasks, and NVIDIA has corrected the production anomaly while offering free replacements for affected cards.

Tools like GPU-Z can help verify whether your card has the correct number of ROPs; if fewer than 176, it should probably be replaced.

### Shipping Delays and Stock Shortages

Due to a combination of these production issues as well as high demand and supply chain challenges, the RTX 5090 is currently experiencing **significant shipping delays.** Availability is low and wait times are high. UK retailer Overclockers has been particularly transparent about lead times, [revealing](https://www.pcgamesn.com/nvidia/overclockers-stock-announcement) that it could take anywhere from 3-16 weeks for its customers to receive an RTX 5090 that they've already pre-ordered.

Most other retailers are facing similar challenges, with limited stock and long waitlists becoming prevalent. Speculation about imminent price hikes is already growing.

### Price Increase Rumors

It's no surprise that high demand and limited supply often lead to rising prices. With the RTX 5090, though, there may be another factor at play. DeepSeek's rapid rise in popularity has sparked a huge [surge](https://www.theinformation.com/articles/deepseek-spurs-black-market-demand-for-nvidia-gaming-chips-in-china) in demand for NVIDIA's gaming GPUs in China, as these cards can be repurposed to run DeepSeek's AI models – and provide a workaround for U.S. export restrictions on advanced GPUs.

(For more info on DeepSeek and how to run their LLM on our GPU cloud, see our previous post [here](https://vast.ai/article/deepseek-r1-open-source-disruptor-or-overhyped-upstart).)

With both gamers and AI developers vying for the same hardware, the strain on supply could push prices even higher in the near future. Some forecasts predict RTX 5090 prices could skyrocket to $5,000 or more.

### Final Thoughts: The Power of the RTX 5090 – Without the Hassle

Despite its hurdles, the RTX 5090 represents a major step forward in AI-driven graphics and cutting-edge performance, cementing its place as the most powerful consumer-grade GPU available today.

At the same time, it may only be worth purchasing if you really want the best of the best and you're willing to shell out for it. With a $1,999 price tag and relatively modest raw performance gains over the RTX 4090, the RTX 5090's true potential hinges on the wider adoption of Multi Frame Generation and neural rendering – features that, if fully embraced, could eventually make this flagship GPU a must-have for those willing to invest.

In the meantime, why wait? Skip the delays and high costs – [rent an RTX 5090](https://cloud.vast.ai/?gpu_option=RTX%205090) on Vast.ai for instant access to top-tier performance\! Given its limited availability on store shelves and its 575-watt power draw, our affordable rental options are the most practical way to access the RTX 5090's full potential without the upfront investment.

[Try it out](https://cloud.vast.ai/?gpu_option=RTX%205090) for yourself today\!

\*\*\*\*\*\*\*\*\*\*

#### Additional Specifications:

| Spec                                |   RTX 5090    |   RTX 5080    |  RTX 5070 Ti  |   RTX 5070   |    RTX 4090    |
| :---------------------------------- | :-----------: | :-----------: | :-----------: | :----------: | :------------: |
| **Architecture**                    |     GB202     |     GB203     |     GB203     |    GB205     |     AD102      |
| **VRAM**                            |  32GB GDDR7   |  16GB GDDR7   |  16GB GDDR7   |  12GB GDDR7  |  24GB GDDR6X   |
| **VRAM Speed**                      |    28Gbps     |    30Gbps     |    28Gbps     |    28Gbps    |     21Gbps     |
| **VRAM Bus Width**                  |      512      |      256      |      256      |     192      |      384       |
| **Bandwidth**                       |   1792GB/s    |    960GB/s    |    896GB/s    |   672GB/s    |    1.01TB/s    |
| **Streaming Multiprocessors (SMs)** |      170      |      84       |      70       |      48      |                |
| **Tensor Cores**                    |      680      |      336      |      280      |     192      |      512       |
| **Ray Tracing Cores**               |      170      |      84       |      70       |      48      |      128       |
| **Boost Clock**                     |   2407 MHz    |   2617 MHz    |   2452 MHz    |   2512 MHz   |    2520 MHz    |
| **Total Graphics Power (TGP)**      |     575W      |     360W      |     300W      |     250W     |      450W      |
| **Recommended Power Supply**        |     1000W     |     850W      |     750W      |     650W     |      850W      |
| **Launch Date**                     | Jan. 30, 2025 | Jan. 30, 2025 | Feb. 20, 2025 | Mar. 5, 2025 | Sept. 20, 2022 |
| **Launch Price**                    |    $1,999     |     $999      |     $749      |     $549     |     $1,599     |


NVIDIA’s RTX 5090 delivers cutting-edge AI and gaming performance, but launch issues, high costs, and stock shortages make it hard to get. Try it now on Vast.ai!

NVIDIA GeForce RTX 5090 Specs: Everything You Need to Know


February may be a short month, but we've made the most of it with plenty of updates!

Vast.ai is now offering NVIDIA RTX 5000-series GPUs for rent, including the powerful **RTX 5090**, bringing even more high-performance options to our platform. Plus, we've partnered with **SkyPilot** in an integration that allows you to rent Vast machines through SkyPilot's infrastructure and price optimization framework for even greater flexibility.

Of course, keeping everything running smoothly means staying on top of any technical hiccups. As always, we've been hard at work refining our platform and fixing bugs. Here are some of the latest bug fixes we've rolled out since our last update:

-   Fixed instance-to-instance copy issue.
-   Made various virtual machine (VM) enhancements.
-   Corrected an issue where machine is erroneously marked as "unavailable."
-   Fixed earnings display issue and template category display issue.

### NVIDIA Cloud GPU Updates

At Vast.ai, every update is aimed at making your experience smoother, faster, and more efficient. From expanding our GPU offerings to adding templates and updating guides, our latest improvements are designed to give you even more control and streamline your workflow.

#### Highlights of the latest enhancements:

-   NVIDIA RTX 5000-series GPUs are now available, including the [RTX 5090](https://cloud.vast.ai/?gpu_option=RTX+5090).
-   SkyPilot Integration: Rent Vast machines through SkyPilot’s infrastructure and price optimization framework.
-   Added [vLLM Template](https://cloud.vast.ai/?ref_id=62897&template_id=f4c3ccaf87393c5266748b3b42ba8567) and [Guide](https://docs.vast.ai/vllm-llm-inference-and-serving): Fast and easy-to-use library for LLM inference and serving.
-   Added [Unreal Pixel Streaming Template](https://cloud.vast.ai/?ref_id=62897&template_id=c1c84fd5a4d945fc16012e7ca02d1423): Stream high-quality, interactive 3D content directly to web browsers without requiring end users to install anything.
-   Added [Ollama + WebUI Guide](https://docs.vast.ai/ollama-webui) for running DeepSeek R1.
-   Added [Guide for Creating a Custom Template](https://docs.vast.ai/creating-a-custom-template) from Vast Base Image.
-   Updated Hosting Guide.

Please check out the full change log for our February 2025 updates at the bottom of this page.

### Our Commitment

Whether you're looking to try out the RTX 5090, take advantage of SkyPilot integration, explore our new and updated templates and guides, or just enjoy the many features we continue to refine and expand, we're here to help you get the most out of Vast.ai! We're always happy to support both new and longtime users with affordable, on-demand GPU and virtual machine rental.

Our easy-to-use platform is backed by a responsive customer support team and an active Discord community. Need assistance? Email us at [support@vast.ai](mailto:support@vast.ai) or join our Discord server for helpful resources today!

### Change Log

#### Features:

-   NVIDIA RTX 5000 Series GPUs: Rent Blackwell chips on Vast, including the [RTX 5090](https://cloud.vast.ai/?gpu_option=RTX+5090).
-   SkyPilot Integration: Rent Vast machines through SkyPilot’s infrastructure and price optimization framework.

#### Templates & Guides:

-   Added [vLLM Template](https://cloud.vast.ai/?ref_id=62897&template_id=f4c3ccaf87393c5266748b3b42ba8567) and [Guide](https://docs.vast.ai/vllm-llm-inference-and-serving): Fast and easy-to-use library for LLM inference and serving.
-   Added [Unreal Pixel Streaming Template](https://cloud.vast.ai/?ref_id=62897&template_id=c1c84fd5a4d945fc16012e7ca02d1423): Stream high-quality, interactive 3D content directly to web browsers without requiring end users to install anything.
-   Added [Ollama + WebUI Guide](https://docs.vast.ai/ollama-webui) for running DeepSeek R1.
-   Added [Guide for Creating a Custom Template](https://docs.vast.ai/creating-a-custom-template) from Vast Base Image.
-   Updated VM Desktop Template.
-   Updated Open WebUI Template: Now includes pre-built llama.cpp.
-   Updated ComfyUI & ComfyUI + FLUX.1 Templates.
-   Updated SD Swarm & SD Forge Templates.
-   Updated Fooocus Template.
-   Updated Ubuntu 22.04 VM Template: Updated drivers to CUDA 12.8, preinstalled Docker container toolkit, added Tailscale support.
-   Updated Kohya’s GUI.
-   Updated Oobabooga Template.
-   Updated Hosting Guide.

#### Bug Fixes:

-   Various virtual machine (VM) enhancements.
-   Instance-to-instance copy issue.
-   Machine marked as "unavailable" issue.
-   Earnings display issue.
-   Template category display issue.
-   Session key issue.

#### Miscellaneous Tasks:

-   Added API keys section into CLI page for CLI login command.
-   Added length validation to API keys.  


Rent NVIDIA RTX 5090 GPUs on Vast.ai! Now integrated with SkyPilot for better pricing. New templates, guides & fixes to improve your experience.

February 2025 Product Update


## Introduction

In the world of audio processing and speech recognition, identifying when someone is speaking versus when there's silence or background noise is a critical first step. This process, known as Voice Activity Detection (VAD), serves as the foundation for many speech-related applications, from transcription services to voice assistants. While conceptually simple, implementing an efficient and accurate VAD system can significantly improve downstream tasks and reduce computational costs.

[PyAnnote Audio](https://github.com/pyannote/pyannote-audio), an open-source toolkit built on PyTorch, offers state-of-the-art models for VAD that are both accurate and accessible. Running these models on Vast.ai provides a cost-effective solution for processing large audio datasets without investing in expensive hardware. This combination gives developers and researchers the tools they need to build sophisticated audio processing pipelines at a fraction of the cost of traditional cloud providers.

### Why Voice Activity Detection Matters

VAD provides several key benefits for speech processing pipelines:

1. **Reduced Computation Load**: By filtering out non-speech segments before running speech-to-text (STT) models, we significantly reduce the computational resources needed for transcription.

2. **Improved Accuracy**: Many STT models perform better when processing only speech segments rather than trying to interpret silence or background noise.

3. **Efficient Storage**: Extracting only the speech segments can reduce storage requirements for large audio datasets.

4. **Better User Experience**: For applications like voice assistants or transcription services, VAD helps eliminate unnecessary processing of silence.

### What This Guide Covers

In this guide, we will:

-   Set up the Pyannote Audio VAD pipeline
-   Process audio files to detect speech segments
-   Extract and save only the speech portions of the input audio
-   Visualize the results

The output will be a collection of audio files containing only the detected speech segments from the original recording, making them ready for further processing in speech-to-text pipelines.

### Why VAST.ai

VAST.ai offers a marketplace approach to GPU rentals that provides significant advantages for audio processing tasks. Unlike traditional cloud providers, VAST.ai allows you to:

-   Rent precisely the GPU capacity needed for your workload
-   Access GPU's at more affordable rates
-   Access more types of SKU's of GPU's, particularly lower RAM/more affordable GPUs
-   Avoid long-term commitments for experimental projects

For VAD specifically, VAST.ai offers an ideal balance of performance and cost-effectiveness, as these models benefit from GPU acceleration without requiring the most expensive hardware tiers.

## Download This Notebook

To follow along with this tutorial, you can download the complete Jupyter notebook:

[Notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/6q8akx/6b8dc846a00d27decad5a60a6159466813f6d445/files/pyannote_vad_vast.ipynb)

Having the notebook will allow you to execute the code blocks as you read through this guide, making it easier to understand and experiment with the VAD implementation.

## Choosing an Instance

For running the Pyannote Voice Activity Detection model on VAST.ai, you'll need a relatively modest GPU setup since VAD models are computationally efficient compared to larger AI tasks. Here are the recommended specifications:

-   GPU: A mid-range GPU like an RTX 3060 or 4060 would be sufficient.
-   VRAM: 6-8GB of VRAM should be adequate as the Pyannote VAD model is relatively small.
-   RAM: 8-16GB system RAM is recommended for processing audio files.
-   Storage: At least 10GB for the model, dependencies, and your audio files.
-   CUDA: Make sure the instance has CUDA installed (version 11.0+ recommended).
-   Python: Python 3.8+ with PyTorch installed.

## Selecting an Instance

Follow these steps to set up your environment:

1. Ensure that you have a Vast.ai account
2. Go to the Vast Templates in the Console https://cloud.vast.ai/templates/
3. Select the `PyTorch (CuDNN Runtime)` Template
4. Filter for an instance with:
    - 1 GPU
    - 6-8GB of VRAM
    - 8-16GB system RAM
    - 10GB of storage
5. Select an instance and click rent
6. Install the Vast TLS certificate in your browser to access the notebook server https://docs.vast.ai/instances/jupyter#1SmCz
7. Go to your Instances https://cloud.vast.ai/instances/ and click "Open" to access the jupyter server on your instance.
8. Upload the notebook to the server or create a new notebook

## Installing Dependencies

Let's start by installing the necessary Python packages:

```python
%%bash
pip install pyannote.audio
pip install pydub
pip install librosa
pip install yt-dlp
```

We also need to install FFmpeg for audio processing:

```python
%%bash
apt-get update && apt-get install -y ffmpeg
```

## Setting Up Your Hugging Face Token

Pyannote models are hosted on Hugging Face, so you'll need to set up authentication:

```python
# Make sure you've accepted the user conditions at:
# https://hf.co/pyannote/voice-activity-detection
# https://hf.co/pyannote/segmentation

HF_TOKEN = ""  # Add your token here
```

Ensure that you have accepted the terms for the models at the URLs above. The models are free to use, but you must agree to their terms of service.

## Generating Test Data

For this tutorial, we'll download a sample audio file from Vast.ai's YouTube channel. You can also use your own audio file if you prefer:

```bash
yt-dlp -f "bestaudio" --extract-audio --audio-format wav -o "test.wav" https://www.youtube.com/watch?v=542xENIxKFU
```

## Voice Activity Detection

First, let's set up the VAD pipeline:

```python
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
        "pyannote/voice-activity-detection",
        use_auth_token=HF_TOKEN
    )

# Move pipeline to appropriate device
pipeline = pipeline.to(device)
```

Next, we'll process our audio file to identify speech segments:

```python
# Process the audio file
audio_file = "test.wav"
output = pipeline(audio_file)

print(f"Processing {audio_file} on {device}")
print("Voice activity segments:")

# Get all speech segments
speech_segments = list(output.get_timeline().support())

for i, speech in enumerate(speech_segments):
    # active speech between speech.start and speech.end
    print(f"Segment {i+1}: Speech from {speech.start:.2f}s to {speech.end:.2f}s (duration: {speech.duration:.2f}s)")
```

The output will look like this when using the default audio file:

```
Processing test.wav on cuda
Voice activity segments:
Segment 1: Speech from 6.78s to 51.62s (duration: 44.84s)
Segment 2: Speech from 53.56s to 54.27s (duration: 0.71s)
Segment 3: Speech from 55.55s to 84.76s (duration: 29.21s)
Segment 4: Speech from 86.53s to 89.03s (duration: 2.50s)
...
```

## Saving Speech Segments

Now we'll create a function to extract the identified speech segments from our audio file:

```python
import os
import shutil
from pydub import AudioSegment


def split_audio_by_segments(audio_path, segments, output_dir="output_segments"):
    """
    Split an audio file into multiple files based on speech segments

    Parameters:
    -----------
    audio_path: str
        Path to the input audio file
    segments: list
        List of speech segments (with start and end attributes)
    output_dir: str
        Directory to save the output segments
    """
    # Clear the output directory if it exists
    if os.path.exists(output_dir):
        shutil.rmtree(output_dir)

    # Create output directory
    os.makedirs(output_dir, exist_ok=True)

    # Load the audio file
    audio = AudioSegment.from_file(audio_path)

    # Extract each segment
    for i, segment in enumerate(segments):
        # Convert seconds to milliseconds
        start_ms = int(segment.start * 1000)
        end_ms = int(segment.end * 1000)

        # Extract segment
        segment_audio = audio[start_ms:end_ms]

        # Generate output filename
        filename = os.path.basename(audio_path)
        name, ext = os.path.splitext(filename)
        output_path = os.path.join(output_dir, f"{name}_segment_{i+1:04d}_{start_ms:08d}ms-{end_ms:08d}ms{ext}")

        # Export segment
        segment_audio.export(output_path, format=ext.replace('.', ''))
        print(f"Saved segment {i+1} to {output_path}")
```

Let's apply this function to extract our speech segments:

```python
split_audio_by_segments(audio_file, speech_segments)
```

This will be the output when using the default audio file:

```
Saved segment 1 to output_segments/test_segment_0001_00006780ms-00051617ms.wav
Saved segment 2 to output_segments/test_segment_0002_00053558ms-00054267ms.wav
Saved segment 3 to output_segments/test_segment_0003_00055549ms-00084760ms.wav
Saved segment 4 to output_segments/test_segment_0004_00086532ms-00089029ms.wav
...
```

## Inspecting Results

To verify our results, we'll create a function to play audio files in our Jupyter environment:

```python
import librosa
from IPython.display import Audio, display

def play_audio(file_path, sr=None):
    """
    Play an audio file in a Jupyter notebook.

    Parameters:
    -----------
    file_path : str
        Path to the audio file to play
    sr : int, optional
        Sample rate to load the audio with. If None, uses the file's native sample rate.

    Returns:
    --------
    Audio widget that can be played in the notebook

    Example:
    --------
    >>> play_audio('path/to/audio.wav')
    """
    # Load the audio file
    y, sr = librosa.load(file_path, sr=sr)


    # Return an audio widget to play the sound
    audio_widget = Audio(data=y, rate=sr)
    display(audio_widget)
```

First, we'll play the original audio file. Listen to the first minute or so to get an idea of what it sounds like before VAD.

```python
play_audio(audio_file)
```

Next, we'll listen to the first three clips to verify that we have isolated the speech in our test file:

```python
import os
audio_dir = "./output_segments/"

audio_files = os.listdir(audio_dir)
audio_files.sort()

n_clips = 3

for fname in audio_files[0:n_clips]:
    play_audio(audio_dir + fname)
```

You'll see that the audio lengths match up with the `speech_segments` output:

```
Segment 1: Speech from 6.78s to 51.62s (duration: 44.84s)
Segment 2: Speech from 53.56s to 54.27s (duration: 0.71s)
Segment 3: Speech from 55.55s to 84.76s (duration: 29.21s)
...
```

Listening to the output files, we can see that we have effectively isolated the speech.

## Conclusion

With this implementation, you now have a working Voice Activity Detection system that can identify and extract speech segments from audio files. This forms an excellent foundation for more advanced audio processing tasks like speech recognition, speaker diarization, or audio content analysis.

Look out for more content from Vast about all these other types of tasks!


Run state-of-the-art Voice Activity Detection (VAD) with PyAnnote Audio on Vast.ai for cost-effective speech processing. Extract speech segments easily!

Voice Activity Detection (VAD) with Pyannote on VAST


## Introduction

With the release of o1 family of models, reasoning has taken the AI world by storm. Instead of immediately answering a prompt, reasoning is where a model is trained to first output it's "thinking" for its answer. This increases performance on the most difficult tasks today, and since it is trained into the model, it is better than asking it to record its "chain of thought" via a prompt.

R1 takes this even further. We now have Open Source models for this type of task, which means:

1. We can retain the "thinking" token outputs which we couldn't do beforehand. This lets us create datasets for finetuning.
2. We can easily fine-tune these models on our own tasks
3. We can run them on more affordable compute to drastically reduce the cost!

This implementation combines three key components: A distilled DeepSeek model for reasoning transparency, a Vast Template for vLLM's optimized inference server for efficient deployment, and Langchain for parsing reasoning tokens versus normal output tokens. Together, they create a production-ready system that can handle both the technical demands of serving a large language model and the practical needs of processing its unique output format.

Vast.ai provides an ideal platform for this setup, offering the necessary GPU resources at a fraction of traditional cloud costs. Its marketplace model and simple Docker integration make it particularly well-suited for deploying and scaling language models, allowing developers to focus on building applications rather than managing infrastructure. The Templates feature from Vast allows for a repeatable deployment with little additional configuration.

This guide demonstrates how to deploy [deepseek/DeepSeek-R1-Distill-Qwen-32B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B) with Vast's templates and integrate them with Langchain for advanced processing capabilities. We'll show you how to leverage vLLM's optimized inference server and create custom parsers to handle DeepSeek's distinctive output format. Feel free to follow along in this [companion notebook](https://nbviewer.org/urls/bitbucket.org/!api/2.0/snippets/jsbcannell/xq8zG5/2cf7d8758235fa5176d9aee8b8d3885128c0109c/files/serving_deepseek_on_vast.ipynb)

## Setting Up the Environment

Before we can deploy our model, we need to set up our Vast.ai environment. First, install the Vast SDK:

```python
%%bash
pip install --upgrade vastai
```

Set up your Vast API key (available from your [Account Page](https://cloud.vast.ai/account/)):

```python
%%bash
# Here we will set our api key
export VAST_API_KEY="<your-key-here>"
vastai set api-key $VAST_API_KEY
```

### Choosing the Right Hardware

Now we are going to search for GPUs on Vast.ai to run the DeepSeek-R1-Distill-Qwen-32B model. This model requires specific hardware capabilities to run efficiently with vLLM's optimizations. Here are our requirements:

1. A minimum of 80GB GPU RAM to accommodate:

    - DeepSeek model weights (32B Parameters)
    - KV Cache for handling of extra long output token lengths

2. A single GPU configuration, as DeepSeek-R1-Distill-Qwen-32B can be efficiently served on one GPU:
   Note: Multi-GPU configurations are supported if higher throughput is needed.

3. A static IP address for:

    - Stable API endpoint hosting
    - Consistent client connections
    - Reliable Langchain integration

4. At least one direct port that we can forward for:

    - vLLM's OpenAI-compatible API server
    - External access to the model endpoint
    - Secure request routing

5. At least 120GB of disk space to hold the model and other things we might like to download

Here's how to search for suitable instances on Vast.ai:

```python
%%bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 80 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 120 \
rentable = true"
```

### Deploying the Server via Vast Template

We'll use vLLM's OpenAI-compatible server to deploy the DeepSeek model. This setup provides an OpenAI-compatible API endpoint that works seamlessly with existing tools and libraries.

We will do this with a template that:

1. Uses `vllm/vllm-openai:latest` docker image. This gives us an OpenAI-compatible server.
2. Forwards port `8000` to the outside of the container, which is the default OpenAI server port
3. Forwards `--model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --max-model-len 8192  --enforce-eager` on to the default entrypoint (the server itself)
4. Uses `--tensor-parallel-size 1` by default.
5. Uses `--gpu-memory-utilization 0.90` by default
6. Ensures that we have 120 GB of Disk space

```python
%%bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --disk 120 --template_hash eda062b3e0c9c36f09d9d9a294405ded
```

### Verify Setup

After deployment, verify that your server is running correctly with a simple curl test:

```python
%%bash
export VAST_IP_ADDRESS="<your-ip-here>"
export VAST_PORT="<your-port-here>"
curl -X POST http://$VAST_IP_ADDRESS:$VAST_PORT/v1/completions \
     -H "Content-Type: application/json" \
     -d '{
           "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
           "prompt": "Hello, how are you?",
           "max_tokens": 50
         }'
```

Note: You can find your instance's IP address and port in the [Instances tab](https://cloud.vast.ai/instances/) of the Vast AI Console.

## Implementing Custom Output Parsing

### Install Required Dependencies

```python
%%bash
pip install --upgrade langchain langchain-openai openai
```

### Creating a DeepSeek Output Parser

DeepSeek models output responses in a unique format with separate thinking and response sections. The thinking section, wrapped in `<think>` tags, contains the model's reasoning process, while everything after the closing tag represents the final response. This separation is valuable for understanding how the model reaches its conclusions.

Let's create a custom parser that can handle this format:

```python
from typing import Optional, Tuple
from langchain.schema import BaseOutputParser

class R1OutputParser(BaseOutputParser[Tuple[Optional[str], str]]):
    """Parser for DeepSeek R1 model output that includes thinking and response sections."""

    def parse(self, text: str) -> Tuple[Optional[str], str]:
        """Parse the model output into thinking and response sections.

        Args:
            text: Raw text output from the model

        Returns:
            Tuple containing (thinking_text, response_text)
            - thinking_text will be None if no thinking section is found
        """
        if "</think>" in text:
            # Split on </think> tag
            parts = text.split("</think>")
            # Extract thinking text (remove <think> tag)
            thinking_text = parts[0].replace("<think>", "").strip()
            # Get response text
            response_text = parts[1].strip()
            return thinking_text, response_text

        # If no thinking tags found, return None for thinking and full text as response
        return None, text.strip()

    @property
    def _type(self) -> str:
        """Return type key for serialization."""
        return "r1_output_parser"
```

Our parser does several important things:

1. It looks for the `</think>` tag to identify the boundary between thinking and response
2. It extracts and cleans up the thinking section by removing the `<think>` tags
3. It handles cases where no thinking section is present by returning None
4. It ensures both sections are properly stripped of extra whitespace

### Setting Up the Model Chain

Now that we have our parser, we need to set up a processing chain that connects our model to the parser. This chain will:

-   Send requests to our deployed model
-   Process the responses through our custom parser
-   Return the separated thinking and response sections

Here's how we set it up:

```python
from langchain_openai import ChatOpenAI
from langchain.prompts import ChatPromptTemplate
from langchain.schema.runnable import RunnablePassthrough

# Initialize the model with your Vast instance details
VAST_IP_ADDRESS="<your-ip-here>"
VAST_PORT="<your-port-here>"

openai_api_key = "EMPTY"
openai_api_base = f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"

model = ChatOpenAI(
    base_url=openai_api_base,
    api_key=openai_api_key,
    model_name="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
    max_tokens=8000,
    temperature=0.7
)

# Create prompt template
prompt = ChatPromptTemplate.from_messages([
    ("user", "{input}")
])

# Create parser
parser = R1OutputParser()

# Create chain
chain = (
    {"input": RunnablePassthrough()}
    | prompt
    | model
    | parser
)
```

Let's break down what each component does:

-   The `ChatOpenAI` instance connects to our deployed model using the OpenAI-compatible API
-   The `ChatPromptTemplate` formats our input messages
-   The `RunnablePassthrough` ensures our inputs flow through the chain correctly
-   Finally, our custom parser processes the model's output

### Example Usage

Let's test our setup with a prompt that can demonstrate the model's reasoning capabilities:

```python
prompt_text = "Explain quantum computing to a curious 10-year-old who loves video games."

thinking, response = chain.invoke(prompt_text)
print("\nTHINKING:\n")
print(thinking)
print("\nRESPONSE:\n")
print(response)
```

#### Output:

```
THINKING:

Okay, so I need to explain quantum computing to a 10-year-old who loves video games. Hmm, let's break this down. First, I should think about what a 10-year-old knows. They understand basic concepts like computers, games, maybe some video game terms like pixels or bits. They also enjoy stories or analogies that relate to their interests, so using video games as a context makes sense.

Quantum computing is a complex topic, so I need to simplify it. I remember that classical computers use bits, which are 0s and 1s. Quantum computers use qubits, which can be both 0 and 1 at the same time. That's a key difference. Maybe I can compare it to something in a video game. Maybe like a character that can be in two places at once, or a power-up that gives multiple abilities simultaneously.

I should also mention superposition and entanglement. Superposition is when a qubit can be in multiple states at once, and entanglement is when qubits are connected, so the state of one affects the others. Maybe relate that to something like a power-up that affects multiple things at once or a team of characters working together.

Another important point is that quantum computers can solve certain problems much faster than classical computers. Maybe compare it to a game where you can unlock a level or defeat a boss much quicker with a special power-up. But I should also note that not all problems are better solved by quantum computers, just specific ones.

Putting it all together, the explanation should start with what a classical computer does, then introduce qubits as special bits, explain their unique properties using video game analogies, and then talk about the advantages and limitations. I should keep the language simple, use terms they're familiar with, and make it engaging.

Let me think of an example. Maybe a game where you have to find a key in a maze. A classical computer would check each path one by one, while a quantum computer could check all paths at the same time, making it faster. Or maybe a power-up that lets you see all possible paths at once.

I also want to make it clear that it's not just faster, but it's a different way of computing. Maybe use the idea of magic or special abilities in a game that allow for unique capabilities.

I should avoid technical jargon and keep sentences short. Maybe use a narrative style, like telling a story about a game where quantum computing helps the player win. That could make it more relatable and fun.

So, to sum up, I'll start by comparing classical computers to video game rules, then introduce qubits as special bits that can do multiple things at once, explain superposition and entanglement with game-related examples, and discuss the speed advantages and specific applications. I'll wrap it up by connecting it back to video games, showing how quantum computing could make games more complex or fun in the future.

I think that's a solid approach. Now, I'll structure this into a clear, concise explanation using these points and analogies.

RESPONSE:

Quantum computing is like having a supercharged video game character! Imagine you're playing a game where you need to find a hidden treasure in a maze. A regular computer (like a normal game character) would check each path one by one, which could take a long time. But a quantum computer is like a special character that can check all paths at the same time, making it super fast to find the treasure!

In video games, you sometimes get power-ups that give you special abilities. Quantum computers use something called qubits, which are like those power-ups. Unlike regular bits in a computer that can only be 0 or 1, qubits can be both at the same time. This is called superposition. It's like having a character that can be in two places at once!

Also, qubits can work together in a magical way called entanglement. If one qubit changes, the others change too, no matter how far apart they are. It's like having a team of characters where each member's actions affect the others instantly.

Quantum computers are especially good at solving tricky puzzles or problems that would take a regular computer forever. But they're not better at everything—just certain tasks. So, in the game, your special character would shine in specific levels, making them super fun and challenging!

One day, quantum computing might help create even more exciting games, with puzzles that seem impossible to solve without those special powers. So, think of quantum computing as the ultimate power-up for solving complex problems in the blink of an eye!
```

This example shows how the model:

1. First works through its reasoning in the thinking section
2. Then provides a clear, final response
3. Keeps these two aspects separate for better analysis and control

## Key Features

-   **Separated Thinking and Response**: DeepSeek's unique output format provides insight into the model's reasoning process
-   **Custom Output Parsing**: Langchain integration enables structured parsing of the model's unique output format

## Conclusion

This guide has demonstrated how to deploy and serve the DeepSeek-R1-Distill-Qwen-32B model on Vast.ai. We've shown how to:

-   Set up a cost-effective GPU instance with the right specifications
-   Easilly deploy the model using Vast's Templates with vLLM
-   Create a custom Langchain parser to handle DeepSeek's unique thinking/response format
-   Integrate everything through an OpenAI-compatible API

With this setup on Vast.ai's GPU marketplace, you can now build powerful AI applications with DeepSeek while minimizing both infrastructure complexity and compute costs. The platform's combination of flexible GPU options and simple Docker-based deployment makes it an ideal foundation for serving and scaling language models without the expensive cost model of traditional cloud providers.


Deploy DeepSeek-R1-Distill-Qwen-32B on Vast.ai for cost-effective AI reasoning. Use vLLM, Langchain, and custom parsing to extract structured model insights efficiently.

Vast.ai

Implementing Speech-to-Text with Speaker Diarization: Comparing Pyannote and Sortformer on VAST.ai

NVIDIA GeForce RTX 5090 Specs: Everything You Need to Know

February 2025 Product Update

Voice Activity Detection (VAD) with Pyannote on VAST

Serving DeepSeek Models on Vast.ai with vLLM and Langchain!