
Deep Cogito models are hybrid reasoning Large Language Models (LLMs) uniquely designed to operate in two distinct modes within a single deployment: standard direct answering and detailed step-by-step reasoning. Unlike traditional models that require separate deployments for different reasoning capabilities, Deep Cogito can seamlessly toggle between these modes using simple prompt engineering.

This guide demonstrates how to deploy `deepcogito/cogito-v1-preview-llama-8B` on Vast.ai and leverage its dual reasoning capabilities through vLLM's OpenAI-compatible API.

## Deploying on Vast.ai

First, install the Vast.ai SDK and configure your API key:

```bash
pip install --upgrade vastai
export VAST_API_KEY="YOUR_KEY_HERE"
vastai set api-key $VAST_API_KEY
```

Search for a suitable instance that meets the model's requirements:

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 60 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

Deploy the model using vLLM's OpenAI-compatible API:

```bash
vastai create instance YOUR_INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 80 --args --model deepcogito/cogito-v1-preview-llama-8B
```

## Using Deep Cogito with OpenAI SDK

Install the OpenAI SDK:

```bash
pip install --upgrade openai
```

### Enabling Thinking Mode

Deep Cogito provides two methods to toggle thinking mode:

1. **Hugging Face Transformers Library Method**:
   When using the Hugging Face Transformers library directly, you can enable thinking mode using the `enable_thinking=True` flag with the tokenizer's `apply_chat_template` method. This activates the model's reasoning process, displaying its thought process within `<thinking>` tags before providing the final answer.

2. **vLLM OpenAI-compatible API Method**:
   When using vLLM's OpenAI-compatible API (as in our Vast.ai deployment), use a system prompt instead. The official prompt is:
   ```
   "Enable deep thinking subroutine."
   ```
   
   For the 8B model specifically, we recommend using this enhanced version for more consistent results:
   ```
   "Enable deep thinking subroutine. \n\n YOU MUST USE THE THINKING SUBROUTINE"
   ```

Here's an implementation example using the OpenAI SDK:

```python
from openai import OpenAI

# Vast.ai instance configuration
VAST_IP_ADDRESS = "YOUR_VAST_IP"
VAST_PORT = "YOUR_VAST_PORT"

# Initialize the OpenAI client
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

# Function for reasoning mode
def chat_with_model_reasoning(prompt):
    response = client.chat.completions.create(
        model="deepcogito/cogito-v1-preview-llama-8B",
        messages=[
            {"role": "system", "content": "Enable deep thinking subroutine. \n\n YOU MUST USE THE THINKING SUBROUTINE"},
            {"role": "user", "content": prompt}
        ],
    )
    return response.choices[0].message.content

# Function for standard mode
def chat_with_model(prompt):
    response = client.chat.completions.create(
        model="deepcogito/cogito-v1-preview-llama-8B",
        messages=[
            {"role": "user", "content": prompt}
        ],
    )
    return response.choices[0].message.content
```

## Response Comparison

The following examples demonstrate how the model's responses differ between thinking mode and standard mode when asked to explain Large Language Models:

### With Thinking Mode:

```
<think>
Okay, I need to give a short introduction to Large Language Models (LLMs). Let me start by recalling what I know about them. LLMs are a type of artificial intelligence that uses neural networks to process and generate human-like text. They're trained on massive amounts of text data from the internet, books, and other sources. This training allows them to understand and generate contextually relevant responses.

Wait, I should probably mention that the training process involves machine learning, specifically deep learning. They use transformer architectures, which were introduced by Google in 2017. Transformers revolutionized NLP by allowing the model to process input in parallel, which makes them more efficient than previous sequential models.

But maybe I'm getting a bit too technical. The user might just want a basic overview. Let me focus on the key aspects. LLMs are known for their ability to understand natural language and generate coherent text. They can engage in conversations, answer questions, write stories, translate languages, and even help with coding. But they also have limitations, like sometimes providing incorrect or biased information.

I should also mention some popular examples, like GPT-3, BERT, and the more recent ones like GPT-4. These models vary in size and capability. For instance, GPT-3 has 175 billion parameters, making it one of the largest models. GPT-4 is even bigger and claimed to have more capabilities, though it's not publicly available yet.

Also, LLMs are used in various applications, such as chatbots, virtual assistants, and content generation tools. They can help with tasks like summarization, question answering, and sentiment analysis. But it's important to note that they're not perfect and can have issues like hallucinations (making up information) or biases inherited from the training data.

Hmm, I might need to simplify this a bit. Maybe start with a basic definition and then highlight their key features and applications. Avoid too much technical jargon unless necessary.

Wait, I should also mention the training data aspect. LLMs learn from enormous datasets, which means they can pick up on language patterns, idioms, and even cultural references. But this also means they can reflect biases present in the data, which is a concern.

In summary, LLMs are advanced AI models that can understand and generate human-like text using deep learning techniques. They excel at language tasks but have limitations. They're widely used in various applications but also require careful consideration of their ethical implications.
</think>

Large Language Models (LLMs) are advanced AI systems that can process and understand language in ways similar to human intelligence. These models use neural networks and are trained on vast amounts of text data from the internet, books, and other sources.

Here are the key things you should know about LLMs:

1. **How they work**: LLMs use a technology called transformer architecture, which allows them to process language in parallel and maintain context in conversations. This means they can understand what you're saying and respond appropriately.

2. **Capabilities**: LLMs are incredibly versatile, able to:
   - Engage in conversations
   - Answer questions
   - Write stories and articles
   - Help with coding
   - Translate languages
   - Summarize text
   - Create content

3. **Examples**: Popular LLMs include GPT-3, BERT, and ChatGPT. These models vary in size and capability, but they all share the ability to understand and generate human-like text.

4. **Important limitations**: While powerful, LLMs aren't perfect. They can:
   - Make up information (hallucinations)
   - Reflect biases from their training data
   - Sometimes provide incorrect answers

5. **Applications**: LLMs are used in:
   - Chatbots
   - Virtual assistants
   - Content generation tools
   - Language translation services
   - Text analysis and processing

As these models continue to evolve, they're becoming increasingly important in our digital world, though we need to remain mindful of their limitations and potential biases.
```

### Without Thinking Mode:

```
Large Language Models (LLMs) are AI systems that use neural networks to process and generate human-like text. They're pre-trained on massive datasets of text and can understand context, generate coherent responses, and perform various language tasks like translation, summarization, and question-answering.

The most well-known LLMs include GPT (Generative Pre-trained Transformer) models created by OpenAI, BERT (Bidirectional Encoder Representations from Transformers) created by Google, and others. They've become increasingly useful for applications like chatbots, content generation, and language understanding tasks.

Key characteristics of LLMs include their ability to understand context, generate human-like text, and adapt to different tasks, though they can also sometimes produce incorrect or biased information.
```

## Conclusion

Deep Cogito on Vast.ai offers a cost-effective solution for accessing a powerful reasoning LLM. The system prompt technique enables visibility into the model's reasoning process, making it particularly valuable for complex tasks such as coding, mathematical problems, and detailed analysis.

While the `enable_thinking` flag isn't available through vLLM's API, the system prompt approach can be highly effective with proper configuration. Users should experiment with prompt variations to achieve optimal results, as factors such as emphasis and repetition can significantly impact the consistency of the thinking mode, especially with the 8B model.

For production environments where understanding the model's reasoning process is crucial, this approach provides a robust and economical solution. We recommend testing different prompt variations to find the optimal configuration for your specific use case, as the effectiveness of thinking mode can vary depending on the complexity of the task and the chosen model size.


Learn how to deploy and interact with Deep Cogito’s hybrid reasoning LLM (cogito-v1-preview-llama-8B) on Vast.ai. This guide covers both standard and deep thinking modes via vLLM’s OpenAI-compatible API.

Running Deep Cogito on Vast.ai


## Introduction

In multi-speaker audio recordings like meetings, podcasts, or interviews, knowing who spoke when is crucial for many applications. This process, known as Speaker Diarization, partitions an audio stream into segments according to speaker identity.

[PyAnnote Audio](https://github.com/pyannote/pyannote-audio), an open-source toolkit built on PyTorch, offers state-of-the-art models for speaker diarization that are both accurate and accessible. Running these models on Vast.ai provides a cost-effective solution for processing large audio datasets without investing in expensive hardware. This combination gives developers and researchers the tools they need to build sophisticated audio processing pipelines at a fraction of the cost of traditional cloud providers.

### Why Speaker Diarization Matters

Speaker Diarization provides several key benefits for audio processing pipelines:

1. **Speaker Identification**: It identifies different speakers in a conversation, meeting, or any multi-speaker audio recording.

2. **Improved Transcription**: When combined with speech-to-text systems, diarization allows for speaker-attributed transcripts, making it clear who said what.

3. **Processing Efficiency**: By segmenting audio by speaker and removing non-speech portions, diarization can significantly reduce the computational load for downstream tasks like speech recognition, allowing these systems to process only relevant speech segments rather than the entire audio file.

4. **Audio Indexing**: Makes audio content searchable by speaker, allowing users to find all segments where a specific person speaks.

### What This Guide Covers

In this guide, we will:
- Set up the Pyannote Audio Speaker Diarization pipeline
- Process audio files to detect different speakers and their speaking turns
- Calculate speaking time for each identified speaker
- Identify regions with overlapping speech
- Extract and save speaker-specific segments from the input audio
- Play and verify the diarization results

The output will be a collection of audio files separated by speaker, making them ready for further processing in speech-to-text pipelines or speaker-specific analysis.

### Why VAST.ai

VAST.ai offers a marketplace approach to GPU rentals that provides significant advantages for audio processing tasks. Unlike traditional cloud providers, VAST.ai allows you to:
- Rent precisely the GPU capacity needed for your workload
- Access GPUs at more affordable rates
- Access more types of GPU SKUs, particularly lower RAM/more affordable GPUs
- Avoid long-term commitments for experimental projects

For speaker diarization specifically, VAST.ai is particularly well-suited as these models typically require GPU acceleration but don't demand extensive resources. Users can rent just the right amount of computing power for this specific task without overpaying for unused capacity, making it an economical choice for audio processing workflows that might otherwise be cost-prohibitive on traditional cloud platforms.

## Download This Notebook

To follow along with this tutorial, you can download the complete Jupyter notebook:

[Notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/q7MzLr/ee586f7e9bfbcdb2fb48f059b7bf64caf92fa3bd/files/pyannote_diarization_vast.ipynb)

Having the notebook will allow you to execute the code blocks as you read through this guide, making it easier to understand and experiment with the diarization implementation.

## Choosing an Instance

For running the Pyannote Speaker Diarization model on VAST.ai, you'll need a relatively modest GPU setup. The pyannote/speaker-diarization-3.1 model runs in pure PyTorch and is designed to be efficient. Here are the recommended specifications:

- GPU: A low-end GPU like an RTX 3060 or 4060 would be sufficient.
- VRAM: 6-8GB of VRAM should be adequate as the Pyannote diarization pipeline is relatively efficient.
- RAM: 8-16GB system RAM is recommended for processing audio files.
- Storage: At least 10GB for the model, dependencies, and your audio files.
- CUDA: Make sure the instance has CUDA installed (version 11.0+ recommended).
- Python: Python 3.8+ with PyTorch installed.

## Selecting an Instance

Follow these steps to set up your environment:

1. Ensure that you have a Vast.ai account
2. Go to the Vast Templates in the Console https://cloud.vast.ai/templates/
3. Select the `PyTorch (CuDNN Runtime)` Template
4. Filter for an instance with:
   - 1 GPU
   - 6-8GB of VRAM     
   - 8-16GB system RAM
   - 10GB of storage
5. Select an instance and click rent
6. Install the Vast TLS certificate in your browser to access the notebook server https://docs.vast.ai/instances/jupyter#1SmCz
7. Go to your Instances https://cloud.vast.ai/instances/ and click "Open" to access the jupyter server on your instance.
8. Upload the notebook to the server or create a new notebook

## Installing Dependencies

Let's start by installing the necessary Python packages:

```python
%%bash
pip install pyannote.audio
pip install pydub
pip install librosa
pip install datasets
```

We also need to install FFmpeg for audio processing:

```python
%%bash
apt-get update && apt-get install -y ffmpeg
```

## Setting Up Your Hugging Face Token

Pyannote models are hosted on Hugging Face, so you'll need to set up authentication:

```python
# Make sure you've accepted the user conditions at:
# https://huggingface.co/pyannote/speaker-diarization-3.1
# https://huggingface.co/pyannote/segmentation-3.0

HF_TOKEN = ""  # Add your token here
```

Ensure that you have accepted the terms for the models at the URLs above. The models are free to use, but you must agree to their terms of service.

## Downloading Test Data

For this tutorial, we'll use a sample file from the AMI Meeting Corpus dataset, which is a collection of 100 hours of meeting recordings. This dataset is perfect for testing speaker diarization as it contains natural multi-speaker conversations.

```python
from datasets import load_dataset
import os
import soundfile as sf

# Create a directory to save the files
os.makedirs("ami_samples", exist_ok=True)

# Load the dataset with the correct split
dataset = load_dataset("diarizers-community/ami", "ihm", split="train", streaming=True)

# Load just one sample
n_samples = 1
samples = list(dataset.take(n_samples))

for i, sample in enumerate(samples):
    audio = sample["audio"]
    audio_array = audio["array"]
    sampling_rate = audio["sampling_rate"]
    
    # Calculate duration in seconds
    duration = len(audio_array) / sampling_rate
    
    # Use soundfile to save the audio
    output_path = f"ami_samples/sample_{i}.wav"
    sf.write(output_path, audio_array, sampling_rate)
    
    print(f"Saved {output_path} - Speaker: {sample['speakers']} - Duration: {duration:.2f} seconds")
```

## Speaker Diarization

First, let's set up the Speaker Diarization pipeline:

```python
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
        "pyannote/speaker-diarization-3.1",
        use_auth_token=HF_TOKEN
)

# Move pipeline to appropriate device
pipeline = pipeline.to(device)
```

Next, we'll process our audio file to identify different speakers and their speaking turns:

```python
# Process the audio file
audio_file = "./ami_samples/sample_0.wav"
print(f"Processing {audio_file} on {device}")
output = pipeline(audio_file)

print("Voice activity segments:")

for segment, _, speaker in output.itertracks(yield_label=True):
    result = f"{segment.start:.2f} --> {segment.end:.2f} (duration: {segment.duration:.2f}s) Speaker: {speaker}"
    print(result)
```

The output will look something like this:

```
Processing ./ami_samples/sample_0.wav on cuda
Voice activity segments:
18.36 --> 18.42 (duration: 0.07s) Speaker: SPEAKER_03
23.01 --> 25.63 (duration: 2.62s) Speaker: SPEAKER_03
27.08 --> 27.64 (duration: 0.56s) Speaker: SPEAKER_05
...
```

## Additional Analytics

Now that we have processed our file, let's explore some useful features of the Pyannote SDK:

### Speaker Time

Here we calculate the total speaking time for each speaker:

```python
for speaker in output.labels():
    speaking_time = output.label_duration(speaker)
    print(f"Speaker {speaker} total speaking time: {speaking_time:.2f}s")
```

This will output something like:

```
Speaker SPEAKER_00 total speaking time: 558.98s
Speaker SPEAKER_01 total speaking time: 18.98s
Speaker SPEAKER_02 total speaking time: 22.88s
Speaker SPEAKER_03 total speaking time: 469.68s
Speaker SPEAKER_04 total speaking time: 698.02s
Speaker SPEAKER_05 total speaking time: 190.70s
Speaker SPEAKER_06 total speaking time: 5.74s
...
```

### Speaker Overlap

Pyannote can identify regions where multiple speakers are talking simultaneously:

```python
overlap = output.get_overlap()
print(f"Overlapping speech regions: {overlap}")
```

This will output the segments where more than one speaker is speaking.

```
Overlapping speech regions: 
[[ 00:00:27.672 -->  00:00:27.689]
 [ 00:00:38.337 -->  00:00:38.860]
 [ 00:00:40.395 -->  00:00:40.463]
```

### Filter by Speaker

We can also filter the diarization output to focus on a specific speaker:

```python
speaker = "SPEAKER_06"
speaker_turns = output.label_timeline(speaker)
print(f"Speaker {speaker} speaks at:")
for speaker_turn in speaker_turns:
    print(speaker_turn)
```

This will show the segments where this speaker is speaking.

```
Speaker SPEAKER_06 speaks at:
[ 00:03:45.767 -->  00:03:45.852]
[ 00:03:55.386 -->  00:03:55.521]
[ 00:05:07.257 -->  00:05:07.274]
...
```

## Extracting Speaker Segments

To verify our results and prepare the audio for further processing, let's split the original audio into segments by speaker:

```python
import shutil
from pydub import AudioSegment

def split_audio_by_segments(audio_path, diarization_output, output_dir="output_segments"):
    """
    Split an audio file into multiple files based on diarization output
    
    Parameters:
    -----------
    audio_path: str
        Path to the input audio file
    diarization_output: Annotation
        Pyannote diarization output
    output_dir: str
        Directory to save the output segments
    """
    # Clear the output directory if it exists
    if os.path.exists(output_dir):
        shutil.rmtree(output_dir)
    
    # Create output directory
    os.makedirs(output_dir, exist_ok=True)
    
    # Load the audio file
    audio = AudioSegment.from_file(audio_path)
    
    # Extract each segment with speaker information
    for i, (segment, _, speaker) in enumerate(diarization_output.itertracks(yield_label=True)):
        # Convert seconds to milliseconds
        start_ms = int(segment.start * 1000)
        end_ms = int(segment.end * 1000)
        
        # Extract segment
        segment_audio = audio[start_ms:end_ms]
        
        # Generate output filename with speaker information
        filename = os.path.basename(audio_path)
        name, ext = os.path.splitext(filename)
        output_path = os.path.join(output_dir, f"{name}_segment_{i+1:04d}_{start_ms:08d}ms-{end_ms:08d}ms_speaker_{speaker}{ext}")
        
        # Export segment
        segment_audio.export(output_path, format=ext.replace('.', ''))
        print(f"Saved segment {i+1} to {output_path} (Speaker: {speaker})")

# Apply the function to our audio file
split_audio_by_segments(audio_file, output)
```

## Inspecting Results

To verify our results, we'll create a function to play audio files in our Jupyter environment:

```python
import librosa
from IPython.display import Audio, display

def play_audio(file_path, sr=None):
    """
    Play an audio file in a Jupyter notebook.
    """
    # Load the audio file
    y, sr = librosa.load(file_path, sr=sr)
    
    # Display an audio widget to play the sound
    audio_widget = Audio(data=y, rate=sr)
    display(audio_widget)
```

Now, let's listen to a few clips to verify that the speakers were correctly identified and isolated.


```python
import os
audio_dir = "./output_segments/"

audio_files = os.listdir(audio_dir)
audio_files.sort()

n_offset = 21
n_clips = 5

for fname in audio_files[n_offset:n_clips + n_offset]:
    print(f"File: {fname}")
    
    # Extract speaker info if present in filename
    if "_speaker_" in fname:
        speaker_part = fname.split("_speaker_")[1].split(".")[0]
        print(f"Speaker: {speaker_part}")
    
    play_audio(audio_dir + fname)
```


Around file 21 or so, we see a 14-second clip of `SPEAKER_00` speaking:

`File: sample_0_segment_0025_00055364ms-00070045ms_speaker_SPEAKER_00.wav Speaker: SPEAKER_00`

A few files later we see a 10-second clip of `SPEAKER_00` speaking again:

`File: sample_0_segment_0029_00071530ms-00081840ms_speaker_SPEAKER_00.wav Speaker: SPEAKER_00`

Note: The filenames may change. Each time we run the model there may be a different number of clips. The diarization model sometimes captures <1 second of audio and labels it as a speaker.

The 10-second clip is audio of `SPEAKER_00` about the meeting agenda. It is a file of just `SPEAKER_00` talking.

The 14-second clip has two other speakers speaking. This seems like an error at first, but we will notice two other files that capture the additional speakers:

`sample_0_segment_0026_00059228ms-00060325ms_speaker_SPEAKER_02.wav`
`sample_0_segment_0027_00061793ms-00062924ms_speaker_SPEAKER_02.wav`

With further processing, we could remove these from the `SPEAKER_00` file if necessary for our application.

We can also use the Pyannote overlap function to verify that we did have overlapping speech at this time in the audio file.

## Verifying Speaker Overlap

Finally, let's check for regions where multiple speakers are talking simultaneously:

```python
overlap = output.get_overlap()
for overlap_ts in overlap:
    print(f"Overlapping speech regions: {overlap_ts}")
```

Here we see two overlapping segments that match up with the overlapping files we found above.

`sample_0_segment_0026_00059228ms-00060325ms_speaker_SPEAKER_02.wav`
`sample_0_segment_0027_00061793ms-00062924ms_speaker_SPEAKER_02.wav`

```
Overlapping speech regions: [ 00:00:59.228 -->  00:01:00.325]
Overlapping speech regions: [ 00:01:01.793 -->  00:01:02.924]
```

## Conclusion

This tutorial demonstrates how Pyannote's speaker diarization on VAST.ai provides a powerful solution for identifying "who spoke when" in multi-speaker recordings. The implementation offers several advantages:

- **Accuracy and Efficiency**: Pyannote accurately identifies different speakers and their speaking times, even detecting overlapping speech
- **Practical Applications**: Enables speaker-attributed transcription, conversation analytics, and content indexing for meetings, podcasts, and interviews
- **Cost-Effective Processing**: VAST.ai's affordable GPU rentals make processing large audio datasets accessible without expensive hardware investments

The Pyannote diarization model provides impressive results out of the box, and running it on VAST.ai makes it accessible and affordable for a wide range of applications. Whether you're building a meeting transcription service, analyzing call center interactions, or researching conversation dynamics, this approach gives you a solid starting point.

Look out for more content from Vast about other audio processing tasks!


Run speaker diarization with Pyannote on VAST.ai to detect who spoke when in audio files—cost-effective, accurate, and easy to integrate with your pipeline.

Speaker Diarization with Pyannote on VAST


We're pleased to announce that Vast.ai has successfully completed the SOC 2 Type I audit, a significant milestone as we continue to deliver secure and scalable GPU compute to a global user base.

Security remains central to everything we do here at Vast.ai. We've spent the past 6+ years delivering reliable service worldwide while meeting the [highest standards](https://vast.ai/article/security-and-compliance-at-vast-ai) of regulatory compliance. 

### Vast.ai’s Secure Cloud Offering

Our [Trust Center](https://trust.vast.ai/) includes additional information about our controls, compliance stance, FAQs and Subprocessors.

For customers with heightened security and compliance requirements, our [**Secure Cloud**](https://cloud.vast.ai/?secure_cloud=true) offering provides access to GPUs hosted exclusively by our certified datacenter partners. This environment already delivers an added layer of assurance – now further backed by an independent SOC 2 audit on our platform infrastructure.

Achieving **SOC 2 Type I certification** is a clear validation of the controls we've built to protect customer data and ensure the integrity of our infrastructure.

Here's a brief overview of SOC 2 Type I and what it entails.

### What Is SOC 2 Type I?

Put simply, SOC 2 (System and Organization Controls 2\) compliance assesses an organization's security measures. The audit determines whether internal controls are appropriately designed to meet [trust services criteria](https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-2) – covering areas like security, confidentiality, privacy, and processing integrity – in order to safeguard customer data.

Developed by the American Institute of Certified Public Accountants (AICPA), the SOC 2 Type I report is essentially a detailed snapshot evaluating the design and implementation of these controls at a specific point in time. Its goal is to ensure that security measures meet rigorous industry standards.

### Why This Matters

Our SOC 2 Type I certification affirms to our customers and partners that we've put strong, well-designed controls in place to protect their data and maintain operational reliability. We take data security seriously and follow industry-recognized best practices.

The Type I audit is only the first step in a broader compliance roadmap. We are currently preparing for the SOC 2 Type II audit, which evaluates how effectively internal controls perform over an extended period. That assessment will validate that we maintain our security practices consistently, both in design and in day-to-day execution.

We're also advancing our compliance efforts across other standards, to help us better serve organizations with strict regulatory requirements.

### Our Ongoing Commitment

Earning this SOC 2 Type I certification reflects our commitment to protecting the privacy and integrity of the data entrusted to us by developers, researchers, and organizations worldwide. As a distributed peer-to-peer platform powering global compute, we take our responsibility seriously. This certification is one step in our ongoing effort to provide users with infrastructure they can trust. 

For more about security and compliance at Vast.ai as well as our datacenter partners, please see our previous post [here](https://vast.ai/article/security-and-compliance-at-vast-ai). To request a copy of our SOC 2 report, feel free to contact us anytime at [compliance@vast.ai](mailto:compliance@vast.ai).

A huge thank-you to our team at Vast.ai and the audit partners who helped us reach this milestone\! 

We'll keep raising the bar – and keep you updated as we do.



Vast.ai has achieved SOC 2 Type I certification, reinforcing its commitment to data security and trusted infrastructure. Learn what this means for users and what’s next on their compliance roadmap.

Vast.ai Achieves SOC 2 Type I Certification


Google has recently released Gemma 3, their latest open-source language model family. The model, designed to run efficiently on a single GPU, brings state-of-the-art capabilities including advanced multimodal understanding, multilingual support, and an impressive 128K token context window. In this post, we'll explore the key features of Gemma 3 and demonstrate how to deploy it on Vast.ai to leverage one of its new features, comparing images.

## What makes Gemma 3 Special

Gemma 3 is available in four model sizes: 1B, 4B, 12B, and 27B parameters. Each size comes in both instruction-tuned (IT) and pre-trained (PT) variants, allowing developers to choose the optimal model for their specific applications.

What makes Gemma 3 particularly accessible is its efficient architecture - all models are optimized to run on a single GPU, with the smaller variants capable of running on consumer-grade GPUs through Vast.ai's platform.

Gemma 3 offers multiple state-of-the-art capabilities:

### 1. Enhanced Context Window

-   Supports up to 128K tokens context window
-   Improved long-range understanding and coherence
-   Better handling of complex, multi-turn conversations

### 2. Multi-Modal Capabilities

-   Native image understanding and analysis (available in 4B, 12B, and 27B models only)
-   Support for both text and image inputs in the same prompt
-   Advanced ability to analyze images, text, and short videos

### 3. Improved Performance

-   State-of-the-art performance for its size
-   Enhanced reasoning capabilities
-   Better code generation and understanding

### 4. Multilingual Support

-   Out-of-the-box support for over 35 languages
-   Pretrained support for over 140 languages

### 5. Additional Features

-   Function calling support for automated workflows
-   Official quantized versions for faster performance
-   Optimized for various hardware platforms (NVIDIA GPUs, Google Cloud TPUs, AMD GPUs)

## Using Gemma 3 For Image Comparison On Vast

Gemma 3 supports multimodal inputs, including an impressive image comparison capability. In this guide, we'll demonstrate this feature using images from the Open Image Preferences dataset, available [here](https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1-binarized).

### Image Selection

For this post, we've selected two contrasting astronaut images from the Open Image Preferences dataset:

<div style={{ display: "flex", gap: "20px", marginBottom: "2rem" }}>
    <div style={{ flex: 1 }}>
        <img
            src="/uploads/image-1.jpg"
            alt="Astronaut 1"
            style={{ width: "100%" }}
        />
        <p style={{ textAlign: "center", fontWeight: "bold" }}>Astronaut 1</p>
    </div>
    <div style={{ flex: 1 }}>
        <img
            src="/uploads/image-2.jpg"
            alt="Astronaut 2"
            style={{ width: "100%" }}
        />
        <p style={{ textAlign: "center", fontWeight: "bold" }}>Astronaut 2</p>
    </div>
</div>

### Launch an Instance

First, let's set up a Vast.ai instance with the necessary configurations to run Gemma 3:

```bash
vastai create instance $INSTANCE_ID \
    --image vllm/vllm-openai:latest \
    --disk 40 \
    --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=hf-token' \
    --args --model google/gemma-3-4b-it --gpu-memory-utilization 0.9 --max-model-len 4096 --limit-mm-per-prompt "image=2"
```

### Call Gemma 3 Using the OpenAI SDK

Next, we'll write a script to call Gemma 3 on our instance. vLLM exposes an OpenAI-compatible API endpoint, allowing us to interact with Gemma 3 using the OpenAI SDK:

```python
from openai import OpenAI
import base64

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client
client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

# Load and convert local images to base64
def image_to_base64(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

# Convert images to base64
image1_base64 = image_to_base64("/path/to/image-1.jpg")
image2_base64 = image_to_base64("/path/to/image-2.jpg")

response = client.chat.completions.create(
    model="google/gemma-3-4b-it",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Compare these two images"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image1_base64}"
                    }
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image2_base64}"
                    }
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)
```

Here is our output from Gemma 3:

```
Okay, let's compare these two images of astronauts. Here's a breakdown of the key differences and similarities:

**Similarities:**

* **Subject:** Both images feature a spacesuit and the astronaut's helmet.
* **Style:**  Both are highly stylized and appear to be digitally rendered images, likely created with a focus on aesthetic appeal and a sense of wonder. They evoke a futuristic and somewhat surreal feeling.
* **Helmet:** The overall shape and design of the helmet are fairly consistent between the two – a classic, bulbous, and slightly vintage-looking design.



**Differences:**

1. **Background:** This is the most significant difference.
   * **Image 1:** Features a vibrant, complex, and detailed space scene with a blue and gold starry sky, nebulas, and a visible horizon line suggesting a planet or moon. The background is rich in color and texture.
   * **Image 2:** Shows a much simpler and muted background. It’s a dusky blue sky with scattered stars and a faint suggestion of a horizon with mountains. There's no nebula or dramatic planetary features.

2. **Suit Detail & Appearance:**
    * **Image 1:** The spacesuit is more elaborate and brightly colored. It has intricate gold detailing on the arms and chest and a much more modern and polished look. It also appears to be a little more reflective.
    * **Image 2:** The suit is weathered and looks older. There's a visible rust and decay on the metal parts, suggesting it's been used and exposed to the elements. It’s a more rugged and worn appearance.

3. **Pose and Composition:**
   * **Image 1:** The astronaut is standing and facing slightly towards the camera, but the composition is broader, showing more of the suit and the expansive space.
   * **Image 2:** The astronaut is positioned at a three-quarter angle, focusing primarily on the helmet and upper body. The composition is more intimate and tightly framed.

4. **Lighting & Tone:**
   * **Image 1:**  The lighting is dramatic and highlights the spacesuit with warm, glowing lights. It's a brighter and more vibrant image overall.
   * **Image 2:** The lighting is softer and cooler, creating a more subdued and melancholic atmosphere.

**In essence:**

* **Image 1** is a grand, visually stunning depiction of an astronaut experiencing the majesty of space.
* **Image 2** is a more intimate and evocative portrayal, suggesting a sense of isolation, exploration, and the passage of time.



Would you like me to elaborate on any specific aspect of the comparison, such as the artistic style or the possible symbolism of the images?
```

### Image Comparison Analysis

The model provides detailed, structured analysis of the images, demonstrating its sophisticated visual understanding capabilities. The analysis includes:

1. **Comprehensive Comparison**: The model systematically breaks down both similarities and differences between the images
2. **Structured Output**: Results are organized into clear categories (Similarities, Differences, In essence)
3. **Detailed Observations**: The model captures both obvious and subtle details, from overall composition to specific visual elements
4. **Contextual Understanding**: The analysis includes interpretation of artistic style, mood, and potential symbolism
5. **Interactive Engagement**: The model offers to elaborate on specific aspects of the comparison

This level of analysis showcases Gemma 3's advanced multimodal capabilities, particularly its ability to:

-   Process and compare multiple images simultaneously
-   Provide nuanced visual analysis
-   Generate structured, coherent responses
-   Maintain context throughout the analysis

## Next Steps

The image comparison capabilities we've demonstrated can be applied to several real-world use cases:

-   **Visual Quality Control**: Compare product images for manufacturing defects
-   **Design Evolution**: Analyze changes between design iterations
-   **Medical Imaging**: Compare diagnostic images for changes or anomalies

### Beyond Image Analysis: Additional Capabilities

While image comparison showcases Gemma 3's visual understanding, the model's versatility extends far beyond visual tasks. Here are several other powerful applications you can build:

-   **Multilingual Chatbots**: Create AI assistants that can communicate in 140+ languages
-   **Document Analysis**: Leverage the 128K token context window for processing extensive documents
-   **Educational Tools**: Build interactive learning systems with multilingual support
-   **Visual Search**: Combine text and image understanding for sophisticated search applications

These applications can be deployed using the same Vast.ai infrastructure we've explored, allowing for efficient development and scaling.

## Conclusion

Gemma 3 represents a significant advancement in open-source AI models, offering state-of-the-art performance while being designed to run efficiently on a single GPU. Vast.ai's infrastructure provides an ideal platform for deploying Gemma 3 in production environments, making it accessible for both development and scaling applications.


Google’s open-source Gemma 3 model brings multimodal reasoning, 128K context, and multilingual support—all runnable on a single GPU. Deploy it on Vast.ai to compare images or build advanced AI apps.

Running Google's Gemma 3 on Vast.ai


A new NVIDIA Blackwell GPU with 96GB of VRAM? If rumors are to be believed, that could be exactly what's coming. 

A recent shipping manifest [revealed](https://wccftech.com/nvidia-rtx-pro-6000-x-blackwell-leak-96-gb-gddr7-600w/) an entry for the "**RTX PRO 6000 X Blackwell**," evidently being sent to India "for testing purposes." At least two different shippers have sent the GPUs for testing.

This isn't your typical consumer-grade gaming card, however. The as-yet-unannounced Blackwell-based GPU appears to be aimed at AI and data-intensive workloads. Here's what we know so far about this new entry in NVIDIA's professional lineup.

### NVIDIA RTX Pro 6000 X: What We Know So Far

With the impressive [RTX 5090](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know) expected to remain at the top of the GeForce 50-series lineup, the professional segment is where we'll likely see ultra-high-end specs that push the limits on VRAM – and this latest shipping log discovery suggests that will be the case. 

According to reports, as mentioned, the RTX Pro 6000 X features a staggering **96GB of GDDR7 memory** on a 512-bit memory bus. That's significantly higher than even the most powerful gaming cards. For example, the RTX 5090 has 32GB of GDDR7 memory – which is more than enough for most purposes – but higher VRAM is obviously beneficial for workloads that require handling massive datasets, such as foundation AI model training and large-scale simulations.

The RTX Pro 6000 X also features a **GB202 die** and **600 watts** of total board power (TBP). This aligns with previous high-end Blackwell architecture designs, like the RTX 5090, which also has the GB202 die and comes in at 575 watts of TBP.

Interestingly, the discovery of the RTX Pro 6000 X on the shipping manifest follows an [earlier sighting](https://wccftech.com/nvidia-rtx-4090-with-96gb-vram-reportedly-exists/) of a 96GB RTX 4090 GPU – perhaps an engineering sample. While NVIDIA hasn't made any announcements, these repeated sightings nonetheless suggest a broader push for high-VRAM GPUs.

### Comparison to Existing RTX Workstation GPUs

We've seen professional-grade GPUs with massive VRAM configurations before, such as the RTX 6000 Ada with 48GB of memory, as well as the A100/A800 series. However, the RTX Pro 6000 X's 96GB of GDDR7 memory would make it one of the highest-capacity workstation GPUs to date. 

Assuming we see the RTX Pro 6000 X emerge, it would set a new benchmark for professional GPUs, doubling the memory of the RTX 6000 Ada while offering the faster speeds and efficiency of GDDR7 memory.

Plus, unlike the A100 and A800 series, which are designed primarily for data centers and enterprise AI workloads, the RTX Pro 6000 X could serve as a more flexible solution. Its ultra-high VRAM would allow professionals to work with larger datasets without necessarily relying on multi-GPU setups. 

### Looking Ahead

NVIDIA hasn't officially announced the RTX Pro 6000 X, so all information should be taken with a grain of salt, but the repeated mentions of this new GPU in shipping logs suggest that an announcement may be imminent. 

For Vast.ai users, this could mean more powerful GPU rental options in the near future. Stay tuned for more developments\! In the meantime, check out our current selection of high-performance GPUs available on Vast.ai for flexible and cost-effective access to cutting-edge hardware today.  

NVIDIA’s rumored RTX Pro 6000 X packs 96GB of GDDR7 VRAM, a GB202 die, and 600W TBP—ideal for AI, data science, and high-end workstation workloads.

Vast.ai

Running Deep Cogito on Vast.ai

Speaker Diarization with Pyannote on VAST

Vast.ai Achieves SOC 2 Type I Certification

Running Google's Gemma 3 on Vast.ai

NVIDIA RTX Pro 6000 X Rumors: Blackwell GPU with 96GB VRAM Spotted in Shipping Log