
Google has recently released Gemma 3, their latest open-source language model family. The model, designed to run efficiently on a single GPU, brings state-of-the-art capabilities including advanced multimodal understanding, multilingual support, and an impressive 128K token context window. In this post, we'll explore the key features of Gemma 3 and demonstrate how to deploy it on Vast.ai to leverage one of its new features, comparing images.

## What makes Gemma 3 Special

Gemma 3 is available in four model sizes: 1B, 4B, 12B, and 27B parameters. Each size comes in both instruction-tuned (IT) and pre-trained (PT) variants, allowing developers to choose the optimal model for their specific applications.

What makes Gemma 3 particularly accessible is its efficient architecture - all models are optimized to run on a single GPU, with the smaller variants capable of running on consumer-grade GPUs through Vast.ai's platform.

Gemma 3 offers multiple state-of-the-art capabilities:

### 1. Enhanced Context Window

-   Supports up to 128K tokens context window
-   Improved long-range understanding and coherence
-   Better handling of complex, multi-turn conversations

### 2. Multi-Modal Capabilities

-   Native image understanding and analysis (available in 4B, 12B, and 27B models only)
-   Support for both text and image inputs in the same prompt
-   Advanced ability to analyze images, text, and short videos

### 3. Improved Performance

-   State-of-the-art performance for its size
-   Enhanced reasoning capabilities
-   Better code generation and understanding

### 4. Multilingual Support

-   Out-of-the-box support for over 35 languages
-   Pretrained support for over 140 languages

### 5. Additional Features

-   Function calling support for automated workflows
-   Official quantized versions for faster performance
-   Optimized for various hardware platforms (NVIDIA GPUs, Google Cloud TPUs, AMD GPUs)

## Using Gemma 3 For Image Comparison On Vast

Gemma 3 supports multimodal inputs, including an impressive image comparison capability. In this guide, we'll demonstrate this feature using images from the Open Image Preferences dataset, available [here](https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1-binarized).

### Image Selection

For this post, we've selected two contrasting astronaut images from the Open Image Preferences dataset:

<div style={{ display: "flex", gap: "20px", marginBottom: "2rem" }}>
    <div style={{ flex: 1 }}>
        <img
            src="/uploads/image-1.jpg"
            alt="Astronaut 1"
            style={{ width: "100%" }}
        />
        <p style={{ textAlign: "center", fontWeight: "bold" }}>Astronaut 1</p>
    </div>
    <div style={{ flex: 1 }}>
        <img
            src="/uploads/image-2.jpg"
            alt="Astronaut 2"
            style={{ width: "100%" }}
        />
        <p style={{ textAlign: "center", fontWeight: "bold" }}>Astronaut 2</p>
    </div>
</div>

### Launch an Instance

First, let's set up a Vast.ai instance with the necessary configurations to run Gemma 3:

```bash
vastai create instance $INSTANCE_ID \
    --image vllm/vllm-openai:latest \
    --disk 40 \
    --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=hf-token' \
    --args --model google/gemma-3-4b-it --gpu-memory-utilization 0.9 --max-model-len 4096 --limit-mm-per-prompt "image=2"
```

### Call Gemma 3 Using the OpenAI SDK

Next, we'll write a script to call Gemma 3 on our instance. vLLM exposes an OpenAI-compatible API endpoint, allowing us to interact with Gemma 3 using the OpenAI SDK:

```python
from openai import OpenAI
import base64

# Your Vast.ai instance details
VAST_IP_ADDRESS = ""
VAST_PORT = ""

# Initialize the client
client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

# Load and convert local images to base64
def image_to_base64(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

# Convert images to base64
image1_base64 = image_to_base64("/path/to/image-1.jpg")
image2_base64 = image_to_base64("/path/to/image-2.jpg")

response = client.chat.completions.create(
    model="google/gemma-3-4b-it",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Compare these two images"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image1_base64}"
                    }
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image2_base64}"
                    }
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)
```

Here is our output from Gemma 3:

```
Okay, let's compare these two images of astronauts. Here's a breakdown of the key differences and similarities:

**Similarities:**

* **Subject:** Both images feature a spacesuit and the astronaut's helmet.
* **Style:**  Both are highly stylized and appear to be digitally rendered images, likely created with a focus on aesthetic appeal and a sense of wonder. They evoke a futuristic and somewhat surreal feeling.
* **Helmet:** The overall shape and design of the helmet are fairly consistent between the two – a classic, bulbous, and slightly vintage-looking design.



**Differences:**

1. **Background:** This is the most significant difference.
   * **Image 1:** Features a vibrant, complex, and detailed space scene with a blue and gold starry sky, nebulas, and a visible horizon line suggesting a planet or moon. The background is rich in color and texture.
   * **Image 2:** Shows a much simpler and muted background. It’s a dusky blue sky with scattered stars and a faint suggestion of a horizon with mountains. There's no nebula or dramatic planetary features.

2. **Suit Detail & Appearance:**
    * **Image 1:** The spacesuit is more elaborate and brightly colored. It has intricate gold detailing on the arms and chest and a much more modern and polished look. It also appears to be a little more reflective.
    * **Image 2:** The suit is weathered and looks older. There's a visible rust and decay on the metal parts, suggesting it's been used and exposed to the elements. It’s a more rugged and worn appearance.

3. **Pose and Composition:**
   * **Image 1:** The astronaut is standing and facing slightly towards the camera, but the composition is broader, showing more of the suit and the expansive space.
   * **Image 2:** The astronaut is positioned at a three-quarter angle, focusing primarily on the helmet and upper body. The composition is more intimate and tightly framed.

4. **Lighting & Tone:**
   * **Image 1:**  The lighting is dramatic and highlights the spacesuit with warm, glowing lights. It's a brighter and more vibrant image overall.
   * **Image 2:** The lighting is softer and cooler, creating a more subdued and melancholic atmosphere.

**In essence:**

* **Image 1** is a grand, visually stunning depiction of an astronaut experiencing the majesty of space.
* **Image 2** is a more intimate and evocative portrayal, suggesting a sense of isolation, exploration, and the passage of time.



Would you like me to elaborate on any specific aspect of the comparison, such as the artistic style or the possible symbolism of the images?
```

### Image Comparison Analysis

The model provides detailed, structured analysis of the images, demonstrating its sophisticated visual understanding capabilities. The analysis includes:

1. **Comprehensive Comparison**: The model systematically breaks down both similarities and differences between the images
2. **Structured Output**: Results are organized into clear categories (Similarities, Differences, In essence)
3. **Detailed Observations**: The model captures both obvious and subtle details, from overall composition to specific visual elements
4. **Contextual Understanding**: The analysis includes interpretation of artistic style, mood, and potential symbolism
5. **Interactive Engagement**: The model offers to elaborate on specific aspects of the comparison

This level of analysis showcases Gemma 3's advanced multimodal capabilities, particularly its ability to:

-   Process and compare multiple images simultaneously
-   Provide nuanced visual analysis
-   Generate structured, coherent responses
-   Maintain context throughout the analysis

## Next Steps

The image comparison capabilities we've demonstrated can be applied to several real-world use cases:

-   **Visual Quality Control**: Compare product images for manufacturing defects
-   **Design Evolution**: Analyze changes between design iterations
-   **Medical Imaging**: Compare diagnostic images for changes or anomalies

### Beyond Image Analysis: Additional Capabilities

While image comparison showcases Gemma 3's visual understanding, the model's versatility extends far beyond visual tasks. Here are several other powerful applications you can build:

-   **Multilingual Chatbots**: Create AI assistants that can communicate in 140+ languages
-   **Document Analysis**: Leverage the 128K token context window for processing extensive documents
-   **Educational Tools**: Build interactive learning systems with multilingual support
-   **Visual Search**: Combine text and image understanding for sophisticated search applications

These applications can be deployed using the same Vast.ai infrastructure we've explored, allowing for efficient development and scaling.

## Conclusion

Gemma 3 represents a significant advancement in open-source AI models, offering state-of-the-art performance while being designed to run efficiently on a single GPU. Vast.ai's infrastructure provides an ideal platform for deploying Gemma 3 in production environments, making it accessible for both development and scaling applications.


Google’s open-source Gemma 3 model brings multimodal reasoning, 128K context, and multilingual support—all runnable on a single GPU. Deploy it on Vast.ai to compare images or build advanced AI apps.

Running Google's Gemma 3 on Vast.ai


Analyzing lengthy documents like SEC filings has traditionally required complex chunking strategies and careful management of context. However, with Mistral AI's [Mistral-Small-3.1-24B-Instruct-2503](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503) model and its 128k token context window, we can now process entire documents at once. This post demonstrates how to leverage this capability using Vast.ai's cloud GPU platform.

The Mistral Small 3.1 model represents a significant advancement in open-source language models. With 24 billion parameters, it combines state-of-the-art vision capabilities with an extended context window that makes it particularly suited for document analysis tasks.

Vast.ai's marketplace provides GPU configurations needed for optimal performance at cost-effective prices.

In this post, we will:
1. Set up a Vast.ai instance with the right GPU specifications
2. Deploy the model using vLLM for optimal serving performance
3. Utilize the model's native chat completions API
4. Leverage the long context window to analyze full public company annual reports (10-K filings)

This setup demonstrates how Mistral Small 3.1 with extended context windows can transform document analysis workflows, making it possible to process and understand complex documents in their entirety.

## Deploy Mistral Small 3.1 on Vast

Let's start by getting our environment set up. First, we need to install the Vast AI client and configure our access.

### Install Vast

Head over to the [Vast.ai Account Page](https://cloud.vast.ai/account/) to grab your API key. Once you have it, we'll use it to set up the client:

```bash
#In an environment of your choice
pip install --upgrade vastai
```

With the client installed, we can configure our API key:

```bash
# Here we will set our api key
export VAST_API_KEY="VAST_API_KEY" #Your key here
vastai set api-key $VAST_API_KEY
```

### Choosing the Right Hardware

Now we are going to search for GPUs on Vast.ai to run the Mistral-Small-3.1-24B-Instruct-2503 model. This model requires specific hardware capabilities to run efficiently with vLLM's optimizations. Here are our requirements:

1. A minimum of 60GB GPU RAM to accommodate:
   - Mistral model weights (24B Parameters)
   - KV Cache for handling the 128k token context window

2. A single GPU configuration, as Mistral-Small-3.1-24B-Instruct-2503 can be efficiently served on one GPU:
   Note: Multi-GPU configurations are supported if higher throughput is needed.

3. A static IP address for:
   - Stable API endpoint hosting
   - Consistent client connections
   - Reliable Langchain integration

4. At least one direct port that we can forward for:
   - vLLM's OpenAI-compatible API server
   - External access to the model endpoint
   - Secure request routing

5. At least 80GB of disk space to hold the model and other dependencies

We'll use `vast ai search offers` to find the right instance.

```bash
vastai search offers "compute_cap >= 750 \
gpu_ram >= 60 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

### Deploying the Server via Vast

Select a machine from the output above and copy its ID into the `INSTANCE_ID` variable below.

We will use `vastai create instance` to create an instance that:
1. Uses `vllm/vllm-openai:latest` docker image. This gives us an OpenAI-compatible server.
2. Forwards port `8000` to the outside of the container, which is the default OpenAI server port.
3. Forwards `--model mistralai/Mistral-Small-3.1-24B-Instruct-2503` to serve the Mistral model.
4. Uses Mistral-specific parameters:
   - `--tokenizer_mode mistral` to use the Mistral tokenizer.
   - `--config_format mistral` for the correct model config.
   - `--load_format mistral` to properly load the model weights.
5. Uses `--gpu-memory-utilization 0.95` to maximize context window.
6. Ensures that we have 80 GB of disk space.

Note: Ensure that you fill in your huggingface token `HUGGING_FACE_HUB_TOKEN` to access the model. You'll need to accept the terms on the model's huggingface page https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503.

```bash
export INSTANCE_ID= #insert instance ID
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e HUGGING_FACE_HUB_TOKEN=HUGGING_FACE_HUB_TOKEN' --disk 80 --args --model mistralai/Mistral-Small-3.1-24B-Instruct-2503   --tokenizer_mode mistral   --config_format mistral   --load_format mistral   --gpu-memory-utilization 0.95
```

### Verify Setup and Get Instance IP Address and Port

Now, we need to verify that our setup is working. We first need to wait for our machine to download the image and the model and start serving. This will take a few minutes. The logs will show you when it's done. 

Next, go to the [Instances tab](https://cloud.vast.ai/instances/) in the Vast AI Console and find the instance you just created.

At the top of the instance, there is a button with an IP address in it. Click this and a panel will show up of the IP address and the forwarded ports. 
You should see something like: 
```
Open Ports
XX.XX.XXX.XX:YYYY -> 8000/tcp
``` 
Copy and paste the IP address (XX.XX.XXX.XX) into `VAST_IP_ADDRESS` and the port (YYYY) into `VAST_PORT` as inputs to the curl command below.

This curl command sends an OpenAI compatible request to your `vLLM` server. You should see the response if everything is set up correctly. 

Note: It may take a few minutes for the OpenAI server to initialize.

```bash
export VAST_IP_ADDRESS="VAST_IP_ADDRESS"
export VAST_PORT="VAST_PORT"
curl -X POST http://$VAST_IP_ADDRESS:$VAST_PORT/v1/completions -H "Content-Type: application/json"  -d '{"model" : "mistralai/Mistral-Small-3.1-24B-Instruct-2503", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

## Set Up Our Model

### Setup Dependencies

First, we'll install the OpenAI SDK to communicate with our model on the Vast server.

```bash
pip install --upgrade openai
```

### Set up Our Model

Next, let's configure our model using the IP address and port from the Vast.ai Console. We'll create a simple function to interact with the model, using Mistral's recommended temperature of 0.15.

```python
from openai import OpenAI

# Your Vast.ai instance details
VAST_IP_ADDRESS = "VAST_IP_ADDRESS"
VAST_PORT = "VAST_PORT"

# Initialize the client with your server URL
client = OpenAI(
    api_key="EMPTY",  # vLLM doesn't require an actual API key
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

# Simple function to chat with the model
def chat_with_model(prompt):
    response = client.chat.completions.create(
        model="mistralai/Mistral-Small-3.1-24B-Instruct-2503",
        messages=[
            {"role": "user", "content": prompt}
        ],
        temperature=0.15
    )
    return response.choices[0].message.content
```

#### Test our Model

Before we go on to long document analysis, let's test our function to make sure everything is working correctly.

```python
response = chat_with_model("what is the capital of France?")
print(response)
```

## Analyze Macy's Annual Reports to Track Store Numbers

The retail landscape has undergone significant changes, with major department stores closing across the US. Using our model's long context window, we'll analyze Macy's annual reports (10-K filings) to track their store count over the years.

### Download Filings

First, we need to download our data. We've gathered a list of Macy's annual report URLs from the SEC's EDGAR database (https://www.sec.gov/edgar/search/). Let's create a `macys_filings` list to store these URLs:

```python
import requests
from bs4 import BeautifulSoup
import re

# List of Macy's 10-K filing URLs
macys_filings = [
    "https://www.sec.gov/Archives/edgar/data/794367/000162828024012734/m-20240203.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000162828023009154/m-20230128.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000156459022011726/m-10k_20220129.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000156459021016119/m-10k_20210130.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436720000040/m-0201202010xk.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436719000038/m-0202201910xk.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436718000036/m-0203201810k.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436717000041/m-0128201710k.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436716000221/m-0130201610k.htm",
    "https://www.sec.gov/Archives/edgar/data/794367/000079436715000073/m-0131201510k.htm"
]
```

Next, we'll create a function that downloads and cleans the text from each filing, removing HTML formatting and other unnecessary elements:

```python
# Simple SEC headers
headers = {
    "User-Agent": "Your Name (your.email@example.com)",
    "Accept-Encoding": "gzip, deflate",
    "Host": "www.sec.gov"
}

def download_filing_text(url):
    """Download a filing and extract its text."""
    try:
        # Download the filing
        response = requests.get(url, headers=headers, timeout=30)
        response.raise_for_status()
        
        # Parse the HTML and extract text
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # Remove script and style elements
        for script in soup(["script", "style"]):
            script.decompose()
        
        # Get all text with minimal processing
        text = soup.get_text(separator=' ')
        
        # Basic cleaning - collapse whitespace
        text = re.sub(r'\s+', ' ', text).strip()
        
        return text
        
    except Exception as e:
        print(f"Error downloading filing: {str(e)}")
        return ""
```

Now that we have our download function ready, let's use it to process all ten years of Macy's reports. We'll loop through each URL, download the content, and store the cleaned text:

```python
# Process all filings
filings_text = []

for i, url in enumerate(macys_filings):
    
    # Download and process filing
    print(f"Downloading {url} filing...")
    text = download_filing_text(url)
    
    if text:  # If we got text back (not empty string)
        print(f"✓ Successfully processed filing ({len(text)} characters)")
        filings_text.append(text)
    else:
        print(f"✗ Failed to process filing")

# Print summary
print("\n=== PROCESSING SUMMARY ===")
print(f"Successfully processed: {len(filings_text)}/{len(macys_filings)} filings")
```

### Extract Store Counts from Filings

Now comes the exciting part - analyzing the filings to track Macy's retail footprint over time. We'll use our model's long context window to examine each document in its entirety, asking a simple but powerful question: "How many stores did Macy's operate this fiscal year?" This approach demonstrates the advantage of processing complete documents at once - we don't need to worry about missing context or searching through different sections manually.

For each filing, we'll store the model's response in an `all_responses` list, building a comprehensive picture of Macy's store count evolution:

```python
all_responses = []
for i, text in enumerate(filings_text):
    print(f"\nProcessing file {i+1}/{len(filings_text)}")
    
    try:
        prompt = f"How many stores did Macy's operate this fiscal year? DATA: {text}"
        response = chat_with_model(prompt)
        all_responses.append(f"File {i+1} analysis: {response}")
        print(f"✓ File {i+1} processed successfully")

        print(response)
        
    except Exception as e:
        print(f"✗ Error processing file {i+1}: {str(e)}")
        all_responses.append(f"File {i+1} error: {str(e)}")
        
# Print all responses
print("\n=== ANALYSIS OF ALL FILES ===")
for response in all_responses:
    print(f"\n{response}\n")
```

The model's responses show detailed information extraction - accurately identifying store counts while providing precise dates (e.g., "February 3, 2024"), section references (e.g., "Item 1. Business. General"), and geographical context (e.g., "43 states, the District of Columbia, Puerto Rico and Guam") from each filing:

```
=== ANALYSIS OF ALL FILES ===

File 1 analysis: As of February 3, 2024, Macy's operated 718 store locations.


File 2 analysis: As of January 28, 2023, Macy's operated 722 store locations. This information is explicitly stated in the document under the section "Item 1. Business. General" where it mentions:

"As of January 28, 2023, the Company operated 722 store locations in 43 states, the District of Columbia, Puerto Rico and Guam."


File 3 analysis: As of January 29, 2022, Macy's operated 725 store locations. This information is explicitly stated in the "Item 2. Properties" section of the 10-K filing:

"As of January 29, 2022, the operations of the Company included 725 store locations in 43 states, the District of Columbia, Puerto Rico and Guam, comprising a total of approximately 112 million square feet."


File 4 analysis: Macy's, Inc. operated 727 store locations in 43 states, the District of Columbia, Puerto Rico, and Guam as of January 30, 2021.


File 5 analysis: As of February 1, 2020, Macy's operated 775 store locations in 43 states, the District of Columbia, Puerto Rico, and Guam.


File 6 analysis: As of February 2, 2019, Macy's operated a total of **867 stores**. This information is detailed in the 10-K filing under the section discussing the company's properties and store count activity.


File 7 analysis: As of February 3, 2018, Macy's operated approximately **852 stores** in 44 states, the District of Columbia, Puerto Rico, and Guam.


File 8 analysis: As of January 28, 2017, Macy's operated a total of **829 stores**. This number includes stores across various brands such as Macy's, Bloomingdale's, and Bluemercury, located in 45 states, the District of Columbia, Guam, and Puerto Rico.


File 9 analysis: As of January 30, 2016, Macy's operated a total of **870 stores**. This included Macy's, Macy's Backstage, Bloomingdale's, Bloomingdale's Outlet, and Bluemercury stores across 45 states, the District of Columbia, Guam, and Puerto Rico.


File 10 analysis: As of January 31, 2015, Macy's operated a total of 823 stores.
```

### Summarize Store Count Findings

Now that we have the store counts for each year, we will have Mistral summarize the results for us.

```python
prompt = "Give me a list of macys store numbers from 2015-2024 DATA: " + " ".join(all_responses)
response = chat_with_model(prompt)
```

```python
print(response)
```

We now have a concise summary of the number of stores that Macy's operated in each year.

```
Based on the provided data, here is the list of Macy's store numbers from 2015 to 2024:

- **2015:** 823 stores
- **2016:** 870 stores
- **2017:** 829 stores
- **2018:** 852 stores
- **2019:** 867 stores
- **2020:** 775 stores
- **2021:** 727 stores
- **2022:** 725 stores
- **2023:** 722 stores
- **2024:** 718 stores

This list reflects the number of store locations Macy's operated as of the specified dates each year.
```

## Conclusion

In this post, we demonstrated how Mistral Small 3.1's 128k token context window transforms SEC filing analysis. We:

- Deployed Mistral's 24B parameter model on a cost-effective Vast.ai GPU instance
- Processed complete 10-K filings without splitting them into smaller chunks
- Extracted store count data from various sections of Macy's annual reports
- Maintained the full document context throughout the analysis process

With this foundation in place, we can tackle more sophisticated document analysis tasks. While tracking store counts offers a simple demonstration, the real value lies in analyzing complex relationships across entire documents. Financial analysts can:

- Extract specific metrics while understanding their broader context
- Connect information from management discussions with financial data
- Assess risk factors in relation to company performance
- Identify patterns across different sections of a single document

This capability extends beyond financial analysis. The same approach works for legal contracts, academic papers, technical documentation, and any other domain where processing long documents as complete units provides better insights.

By combining Mistral's context window capabilities with Vast.ai's accessible GPU infrastructure, comprehensive document analysis becomes more practical and accessible for a wider range of users and use cases.


Deploy Mistral Small 3.1 on Vast.ai to analyze SEC filings with a 128k token context window, extracting insights like Macy’s store count trends from 2015-2024.

SEC Filing Analysis Using Mistral Small 3.1's Long Context Window


DeepSeek may be getting a [lot of hype](https://vast.ai/article/deepseek-r1-open-source-disruptor-or-overhyped-upstart) recently, but there's a new rival on the scene. Alibaba Cloud's Qwen series of AI models is setting a new standard for intelligence in the open-source world.

Earlier this year, Alibaba Cloud launched **Qwen 2.5-Max** – and it reportedly [outperforms](https://www.reuters.com/technology/artificial-intelligence/alibaba-releases-ai-model-it-claims-surpasses-deepseek-v3-2025-01-29/) other foundation models like DeepSeek-V3, GPT-4o, and Llama-3.1-405B "almost across the board," according to the company. Not only that, but it also [powered](https://www.scmp.com/tech/big-tech/article/3298233/alibabas-qwen-powers-top-10-open-source-models-china-ai-know-how-goes-beyond-deepseek) the top 10 open-source LLMs on Hugging Face's rankings last month. All of the top-ranked models on the Open LLM Leaderboard were trained and developed on the updated open-source versions of Qwen.

So what is Qwen, and what can you do with it? Let's take a closer look\!

### Qwen 2.5: Advancements and Features

The Qwen model series includes Qwen (an LLM), Qwen-VL (a vision-language model), Qwen-Audio, Qwen-Coder, and Qwen-Math. The latest Qwen 2.5 models have been pre-trained on a large-scale dataset of up to 18 trillion tokens, resulting in a substantial knowledge base as well as much-improved capabilities in coding and math.

In terms of performance metrics, Qwen 2.5 is quite impressive. It [achieved](https://www.alibabacloud.com/en/solutions/generative-ai/qwen) a score of 85+ on the Massive Multitask Language Understanding (MMLU) benchmark, a HumanEval score of 85+ in coding, and a MATH benchmark score above 80\.

Beyond these core competencies, Qwen 2.5 has highly advanced abilities in following instructions and understanding and generating structured data. It can handle diverse system prompts, demonstrating an adaptability that's well suited for role-based interactions and condition-setting for chatbots.

The models can even interact with software on PCs and mobile devices. One user posted a video online [showing](https://techcrunch.com/2025/01/27/alibabas-qwen-team-releases-ai-models-that-can-control-pcs-and-phones/) Qwen 2.5-VL opening the Booking.com app for Android and booking a flight from Chongqing to Beijing\!

### Specialized Models in the Qwen Family

Each open-source model in the Qwen series is designed for a specific domain. The following are the main options in the Qwen 2.5 lineup:

-   **Qwen 2.5-VL** – This large vision-language model processes images, text, and bounding boxes to recognize and analyze content. It can read text in Chinese and English, compare visuals, create stories, solve math problems, and answer questions.

-   **Qwen 2.5-Audio** – Designed to process audio and text, this audio language model accepts a variety of audio formats (including speech, music, and natural sounds) and generates text responses.

-   **Qwen 2.5-Max** – Pre-trained on over 20 trillion tokens and post-trained with curated Supervised Fine-Tuning (SFT), this high-performing Mixture-of-Expert (MoE) model has surpassed DeepSeek V3, Llama 3.1, and others in benchmarks like Arena-Hard, LiveBench, LiveCodeBench, and GPQA-Diamond.

-   **Qwen 2.5-Coder** – This coding model supports up to 128K tokens of context, covers 92 programming languages, and delivers competitive performance against larger language models in code generation, multi-language coding, completion, and repair.

-   **Qwen 2.5-Math** – Pre-trained and fine-tuned on synthesized data, this mathematical LLM supports English and Chinese queries and excels in Chain-of-Thought (CoT), Program-of-Thought (PoT), and Tool-Integrated Reasoning (TIR) while outperforming most 70B math models.

Furthermore, Qwen APIs enable users to develop generative AI applications for tasks like writing, image generation, and audio analysis, boosting efficiency and transforming the customer experience for businesses.

### Powering Qwen with Vast.ai

Given how competitive the Qwen models are with larger, more established AI systems – and the fact that they're open source – they offer an exciting opportunity for AI researchers, developers, and enthusiasts to experiment and innovate at minimal cost.

However, deploying the models on high-performance hardware is crucial for optimal performance. That's where Vast.ai comes in. Our platform makes this possible with **on-demand [RTX 5090 GPU](https://vast.ai/article/nvidia-geforce-rtx-5090-specs-everything-you-need-to-know) rental**, for instance, providing the power needed to run Qwen models efficiently without the overhead of expensive infrastructure.

With many other GPU options in our marketplace, you can choose the right balance of performance and cost for your specific needs. Get started with Vast today and experience the full potential of Qwen with high-performance GPUs at your fingertips\!


Alibaba Cloud’s Qwen 2.5 AI models outperform rivals like DeepSeek and GPT-4o, excelling in coding, math, and vision. Available open-source, powered by Vast.ai GPUs.

Alibaba's Qwen: An Open-Source AI Model that Surpasses DeepSeek?


The long-awaited NVIDIA GeForce RTX 5090 is here – but good luck getting your hands on one. From shipping delays to hardware defects, this GPU launch has been far from smooth.

Controversy aside, it's exciting to finally have confirmed performance details for this high-end graphics card. You can [try it now](https://cloud.vast.ai/?gpu_option=RTX%205090) for yourself on Vast.ai. Here's everything you need to know about the RTX 5090...

### RTX 5090: The Performance King?

As NVIDIA's new flagship GPU, the [RTX 5090](https://vast.ai/pricing/gpu/RTX-5090) is undoubtedly an impressive machine. Is it worth the $1999 price tag, though? That depends on more than just raw power – let's take a look.

#### Official Specs

For starters, here are some of the official specs for the RTX 5090:

-   **Architecture:** Powered by NVIDIA's cutting-edge Blackwell architecture.
-   **Gaming Chip:** Featuring the next-gen GB202 gaming chip.
-   **Memory:** 32GB of GDDR7 memory with a blazing-fast 1,792GB/sec bandwidth, supported by a super-wide 512-bit memory interface for even faster data transfer.
-   **CUDA Cores:** 21,760 cores, ideal for high-performance graphics and AI workloads.
-   **L2 Cache:** Expanded to 98MB, up from 73MB on the RTX 4090, improving memory efficiency and reducing latency.
-   **Display Support:** PCIe Gen 5 with DisplayPort 2.1b connectors supporting up to 8K at 165Hz.
-   **Power Requirements:** 575W total graphics power (TGP), with a 1000W power supply recommended for optimal performance.

Another bonus? The RTX 5090 has a redesigned **Founders Edition** with a two-slot design and a dual flow-through fan configuration – so it's compact enough for small form factor PCs, a remarkable shift from its predecessors.

For reference, here's a quick overview of the RTX 5090 specs:

| Spec                           | RTX 5090                       |
| :----------------------------- | :----------------------------- |
| **CUDA Cores**                 | 21,760                         |
| **Ray Tracing Cores**          | 170 (4th-gen)                  |
| **Tensor Cores**               | 680 (5th-gen)                  |
| **Base Clock**                 | 2,017 MHz                      |
| **Boost Clock**                | 2,407 MHz                      |
| **L2 Cache**                   | 98,304KB                       |
| **ROPs**                       | 176                            |
| **VRAM**                       | 32GB 28Gbps GDDR7              |
| **VRAM Interface**             | 512-bit                        |
| **Memory Bandwidth**           | 1,792 GB/s                     |
| **Interface**                  | 16x PCIe 5.0                   |
| **Power Connectors**           | 1 x 16-pin / 4 x 8-pin adapter |
| **Total Graphics Power (TGP)** | 575W                           |

(For more specifications – and to see how the RTX 5090 stacks up against its younger siblings in the RTX 50-series lineup as well as the RTX 4090 – check out the chart at the bottom of this page.)

#### AI and Professional Workloads

Beyond gaming, the RTX 5090 is well suited for high-performance AI and professional workloads. Its **680 Tensor cores** and **32GB of GDDR7** memory efficiently accelerate deep learning, 3D rendering, and other compute-intensive tasks. With **98MB of L2 cache** and a **512-bit memory interface**, the RTX 5090 delivers faster data access and reduced latency – crucial for AI training, scientific computing, and content creation.

The **5th-gen Tensor cores** boost AI inference efficiency in frameworks like TensorFlow and PyTorch, while **4th-gen RT cores** enhance real-time ray tracing for animation, visual effects, CAD work, and more.

For demanding AI and video workloads, the RTX 5090 more than meets the challenge. In Procyon's AI XL (FP16) test, the RTX 5090 was **40% faster** than the RTX 4090, and in PugetBench's DaVinci Resolve video processing test, it was **12% faster**.

However, cutting-edge performance doesn't necessarily mean a flawless launch\! Early adopters are already encountering serious issues in some cases.

#### Next-Gen Gaming Features

The biggest leap in technology for this GPU comes from DLSS 4's new **Multi Frame Generation**, an AI-driven frame interpolation technique exclusive to the RTX 50 series. It uses AI to generate up to three additional frames between traditionally rendered ones, significantly boosting frame rates with minimal visual artifacts.

Further, the RTX 5090 introduces a range of AI-driven enhancements in what's called **neural rendering**, designed to push in-game realism to new levels. **RTX Neural Shaders** improve texture compression and deliver film-quality lighting and shading in real time, while **RTX Neural Faces** uses generative AI to create lifelike skin, hair, and facial details.

Building on the power of DLSS 4, these neural rendering techniques integrate directly into the graphics pipeline, leveraging NVIDIA’s Tensor cores to enhance visual fidelity. Though game support is still in development, early demos – such as _Half-Life 2 RTX_ – [showcase](https://www.pcgamesn.com/nvidia/geforce-rtx-5090-review) the potential for deeper shadows, realistic translucency in materials, and an overall richer environment.

Another breakthrough is **Mega Geometry**, a feature that uses RT cores to significantly boost the number of triangles in ray-traced scenes. This allows game engines to maintain full geometric detail without sacrificing performance, resulting in more realistic object depth, shadows, and fine details.

The above features position the RTX 5090 as a powerhouse for next-generation gaming visuals... but there's a catch.

##### Caveat: Performance Claims vs. Real-World Results

The way that NVIDIA presents its performance numbers can be misleading. The company leans heavily on Multi Frame Generation-boosted FPS (frames per second) to compare these new GPUs against older ones.

Frame Generation and Multi Frame Generation (MFG) are powerful tools, but they aren't universal fixes. Their impact varies by game – some titles see massive FPS gains with little visual trade-off, while others may introduce artifacts or latency.

In addition, **not all games will support these features**, as DLSS adoption depends on individual developers. A GPU's raw performance still matters. MFG-inflated benchmarks can make performance gains seem more significant, but Frame Generation still needs a strong base frame rate to be effective. In CPU-limited games, or titles that don't support DLSS 4, the real-world performance gap may be far smaller.

Ultimately, while the RTX 5090 excels at AI-driven rendering, **real-world results will depend on game support and how well these features are implemented.**

### Early Defects and Controversies

#### Melting Power Connectors

Reports of **melted power connectors** on the RTX 5090 Founders Edition have [surfaced](https://www.theverge.com/news/609207/nvidia-rtx-5090-power-connector-melting-burning-issues). A number of users have experienced burnt plastic at both the GPU and PSU ends – with user error and third-party cables evidently not to blame.

NVIDIA introduced the updated 12V-2x6 power connector for the RTX 50 series following melting and overheating issues with the RTX 40-series GPUs. The new connector features shorter sensing pins and longer conductor terminals to improve connection reliability. Users can still use existing 12VHPWR cables with the RTX 50 series; however, the RTX 5090 can draw up to 575 watts of power – which is dangerously close to the cable's 600-watt rating.

PSU manufacturers have taken precautions, such as adding visual indicators to ensure a secure connection, but it remains unclear whether this fully resolves the risk of overheating. So far, NVIDIA has declined to comment on the reports.

#### Chip-Level Fault

One of the more unexpected issues plaguing the RTX 5090 launch is what's now being called the **"missing ROPs" defect.** NVIDIA has officially [confirmed](https://www.techradar.com/computing/gpu/some-lucky-rtx-5080-buyers-may-not-be-so-lucky-after-all-as-nvidia-confirms-rare-chip-fault-also-applies-to-this-gpu-as-well-as-the-rtx-5090-and-5070-ti) that a small number of RTX 5090, RTX 5090D, and RTX 5070 Ti cards were shipped with fewer ROPs (Render Output Units) than they should have. These ROPs are basically rendering pipelines for 3D graphics. Having fewer of them slows down PC gaming performance.

While this defect reportedly only impacts an estimated 0.5% of units (or around 1 in 200), affected cards have an average 4% drop in graphical performance. Since it's a hardware issue, it can't be fixed via BIOS or driver updates. However, it has no impact on AI and compute tasks, and NVIDIA has corrected the production anomaly while offering free replacements for affected cards.

Tools like GPU-Z can help verify whether your card has the correct number of ROPs; if fewer than 176, it should probably be replaced.

### Shipping Delays and Stock Shortages

Due to a combination of these production issues as well as high demand and supply chain challenges, the RTX 5090 is currently experiencing **significant shipping delays.** Availability is low and wait times are high. UK retailer Overclockers has been particularly transparent about lead times, [revealing](https://www.pcgamesn.com/nvidia/overclockers-stock-announcement) that it could take anywhere from 3-16 weeks for its customers to receive an RTX 5090 that they've already pre-ordered.

Most other retailers are facing similar challenges, with limited stock and long waitlists becoming prevalent. Speculation about imminent price hikes is already growing.

### Price Increase Rumors

It's no surprise that high demand and limited supply often lead to rising prices. With the RTX 5090, though, there may be another factor at play. DeepSeek's rapid rise in popularity has sparked a huge [surge](https://www.theinformation.com/articles/deepseek-spurs-black-market-demand-for-nvidia-gaming-chips-in-china) in demand for NVIDIA's gaming GPUs in China, as these cards can be repurposed to run DeepSeek's AI models – and provide a workaround for U.S. export restrictions on advanced GPUs.

(For more info on DeepSeek and how to run their LLM on our GPU cloud, see our previous post [here](https://vast.ai/article/deepseek-r1-open-source-disruptor-or-overhyped-upstart).)

With both gamers and AI developers vying for the same hardware, the strain on supply could push prices even higher in the near future. Some forecasts predict RTX 5090 prices could skyrocket to $5,000 or more.

### Final Thoughts: The Power of the RTX 5090 – Without the Hassle

Despite its hurdles, the RTX 5090 represents a major step forward in AI-driven graphics and cutting-edge performance, cementing its place as the most powerful consumer-grade GPU available today.

At the same time, it may only be worth purchasing if you really want the best of the best and you're willing to shell out for it. With a $1,999 price tag and relatively modest raw performance gains over the RTX 4090, the RTX 5090's true potential hinges on the wider adoption of Multi Frame Generation and neural rendering – features that, if fully embraced, could eventually make this flagship GPU a must-have for those willing to invest.

In the meantime, why wait? Skip the delays and high costs – [rent an RTX 5090](https://cloud.vast.ai/?gpu_option=RTX%205090) on Vast.ai for instant access to top-tier performance\! Given its limited availability on store shelves and its 575-watt power draw, our affordable rental options are the most practical way to access the RTX 5090's full potential without the upfront investment.

[Try it out](https://cloud.vast.ai/?gpu_option=RTX%205090) for yourself today\!

\*\*\*\*\*\*\*\*\*\*

#### Additional Specifications:

| Spec                                |   RTX 5090    |   RTX 5080    |  RTX 5070 Ti  |   RTX 5070   |    RTX 4090    |
| :---------------------------------- | :-----------: | :-----------: | :-----------: | :----------: | :------------: |
| **Architecture**                    |     GB202     |     GB203     |     GB203     |    GB205     |     AD102      |
| **VRAM**                            |  32GB GDDR7   |  16GB GDDR7   |  16GB GDDR7   |  12GB GDDR7  |  24GB GDDR6X   |
| **VRAM Speed**                      |    28Gbps     |    30Gbps     |    28Gbps     |    28Gbps    |     21Gbps     |
| **VRAM Bus Width**                  |      512      |      256      |      256      |     192      |      384       |
| **Bandwidth**                       |   1792GB/s    |    960GB/s    |    896GB/s    |   672GB/s    |    1.01TB/s    |
| **Streaming Multiprocessors (SMs)** |      170      |      84       |      70       |      48      |                |
| **Tensor Cores**                    |      680      |      336      |      280      |     192      |      512       |
| **Ray Tracing Cores**               |      170      |      84       |      70       |      48      |      128       |
| **Boost Clock**                     |   2407 MHz    |   2617 MHz    |   2452 MHz    |   2512 MHz   |    2520 MHz    |
| **Total Graphics Power (TGP)**      |     575W      |     360W      |     300W      |     250W     |      450W      |
| **Recommended Power Supply**        |     1000W     |     850W      |     750W      |     650W     |      850W      |
| **Launch Date**                     | Jan. 30, 2025 | Jan. 30, 2025 | Feb. 20, 2025 | Mar. 5, 2025 | Sept. 20, 2022 |
| **Launch Price**                    |    $1,999     |     $999      |     $749      |     $549     |     $1,599     |


NVIDIA’s RTX 5090 delivers cutting-edge AI and gaming performance, but launch issues, high costs, and stock shortages make it hard to get. Try it now on Vast.ai!

NVIDIA GeForce RTX 5090 Specs: Everything You Need to Know


The growth of AI is outpacing compute. Today, there's an unprecedented demand for computing power as interest in AI surges worldwide – and servers just can't keep up.

More advanced AI models require increasingly powerful hardware. **Without enough GPUs, these large-scale AI systems struggle to run, and also cannot improve.**

Even OpenAI's CEO, Sam Altman, has [admitted](https://techcrunch.com/2024/10/31/openai-ceo-sam-altman-says-lack-of-compute-is-delaying-the-companys-products/) that computing limitations are slowing down the company's ability to release new products. Microsoft's CFO, Amy Woods, [noted](https://www.microsoft.com/en-us/Investor/events/FY-2025/earnings-fy-2025-q1) last year that demand for AI "continues to be higher" than their "available capacity."

If industry giants like OpenAI and Microsoft are feeling the strain, smaller organizations face even steeper challenges. Without deep pockets or dedicated data centers, they often have no choice but to explore other options.

With GPUs in short supply and demand skyrocketing, many companies are turning to a more flexible, scalable solution: **GPU-as-a-service, or GPUaaS**.

## What Is GPU-as-a-Service?

Simply put, **GPUaaS is a cloud-based solution that provides on-demand access to high-performance GPUs.** It fills a gap in the AI industry by enabling customers to rent the compute power they need, scaling up or down as their workloads demand.

Big cloud providers like AWS, Google Cloud, and Azure offer GPU instances, but GPUaaS goes further. Platforms like our own here at Vast.ai tap into a decentralized network of idle GPUs from data centers, enterprises, and even individuals, making high-performance compute more affordable and widely available.

As a result, GPUaaS addresses a critical need: namely, that **AI infrastructure must be scalable, cost-efficient, and accessible to everyone.**

## Why GPUaaS Is the Key to Scalable AI Infrastructure

AI adoption shows no sign of slowing down – and with it comes fluctuating and unpredictable demand for computing power at any given time. Traditional centralized infrastructure isn't well suited to this type of volatility.

Relying on centralized GPU resources often leads to wasted potential and frustrating roadblocks, such as:

-   **High upfront costs** for purchasing and maintaining GPUs.
-   **Underutilized hardware** during slow periods, resulting in inefficiency and wasted costs.
-   **Bottlenecks** when demand spikes, slowing down innovation and delaying results.
-   **Limited access** to GPUs due to supply chain slowdowns and shortages, impacting AI scalability.

This is where GPU-as-a-Service offers a smarter solution. With GPUaaS, you can:

-   **Quickly scale AI workloads** to match demand without over-provisioning.
-   **Reduce costs** with a flexible pay-as-you-go model while eliminating the risks of heavy upfront investment.
-   **Access worldwide GPU resources**, minimizing latency and enhancing performance.
-   **Decrease energy consumption** by leveraging existing, idle GPUs instead of powering additional servers.

## GPUaaS: The Future of AI Compute is Flexible

As AI continues to advance, organizations that can efficiently scale their compute power will have a clear advantage. GPU-as-a-Service reduces the barriers of cost, accessibility, and scalability, offering an alternative to traditional infrastructure. With GPUaaS, you can tap into a global network of GPUs and ensure you always have the resources you need, when you need them.

So why wait? In a world where AI innovation moves fast, compute should never be the bottleneck. **Vast.ai makes enterprise-grade GPUs accessible and affordable through our decentralized marketplace, offering savings of 5-6X on GPU compute.** Whether you're training cutting-edge models or running large-scale inference, we provide the flexibility and efficiency to keep your AI projects moving forward.

Get started and harness the power of GPUaaS with Vast.ai today!

GPU shortages are slowing AI growth. GPU-as-a-Service (GPUaaS) offers a scalable, cost-effective solution for on-demand AI compute power.

AI

Running Google's Gemma 3 on Vast.ai

SEC Filing Analysis Using Mistral Small 3.1's Long Context Window

Alibaba's Qwen: An Open-Source AI Model that Surpasses DeepSeek?

NVIDIA GeForce RTX 5090 Specs: Everything You Need to Know

GPU-as-a-Service: The Scalable Solution to AI's Compute Crisis