
In the rapidly evolving world of cloud computing, Vast.ai remains steadfast in our commitment to being the leader in cost-effective cloud GPU rental. Today, we are thrilled to announce our latest feature – Reserved Instance Discounts – aimed at helping our users get even better pricing for longer term reservations.

## On-Demand vs Reserved: Striking the Right Balance

In the on-demand world of cloud computing, users only pay for what they use and have the flexibility to end a contract at any time. While this offers unrivaled flexibility, the occupancy rate – or the fraction of time GPUs are rented – often clocks in at less than 100%, sometimes even as low as 50%. The resulting impact on provider revenue is noticeable as their hardware sits idle.

Clients willing to commit upfront to a longer rental term can now leverage our Reserved Instance Discounts feature which solves this idle problem. With this, clients can commit to an extended rental contract by pre-paying in advance. While they give up the on-demand liquidity, they receive a hefty discount in return. The provider controls the discount percentage, so each offer will vary. 

## How Reserved Instance Discounts Work

Here's the breakdown:

1. Users can deposit credits into a specific instance using either the [CLI](https://cloud.vast.ai/cli/) or website GUI (look for the **green tag**). The CLI gives users a detailed control over deposit amounts, while the GUI provides a user-friendly interface. The clickable green tag in the GUI is shown for participating machines in the bottom right of the instance card.

![green tag](/uploads/prepaidicon.png)

2. The discount schedule is a function of the time duration of the balance. Larger deposits unlock greater discounts, albeit with diminishing returns. The default discount schedule offers a 20% discount for a 1-month commitment, 30% for a 3-month commitment, and 40% for a 6-month commitment.

![](/uploads/reservedmodal.png)

3. To give an example, if an instance usually costs $1/hr on-demand, a 1-month prepaid rental with the default discount schedule would cost $576 instead of $720. That's a savings of 20%!

4. When additional funds are deposited into an existing reserved instance, everything is recalculated according to the new total balance, leading to a new discount rate.

The specific discount schedule can vary from provider to provider, with 40% at 6 months being the default setting.

## Link to the Documentation

For the complete description of how these works, read our documentation for [Reserved Instance Discounts](https://vast.ai/docs/instance-rental/prepaid-plans).

## Seize the Savings

Our new Reserved Instance Discounts feature is here to change the way you use cloud computing, providing an avenue to commit to your long-term compute needs while reaping the benefits of considerable discounts. So whether you are a start-up scaling your operations or an established business with predictable computing demands, let Vast.ai help you make the most of your cloud computing budget. Your savings start today with Reserved Instance Discounts. Take control of your cloud computing journey and optimize your budget like never before.





Our on-demand rentals provide incredible value. For hosts that enable this feature, you can get further discounts by reserving an instance for months.

Stretching Your Cloud Computing Budget: Introducing Reserved Instance Discounts


# Latest AI Model Releases: September and October

The world of Open Source AI has gotten many updates in the last month or so. There are now many new models with great quality:speed ratios and models that challenge the frontier of closed source models. This makes it even easier to build applications and automate workflows with open source models, which you can deploy on [Vast.ai](https://vast.ai). Meta, Mistral, and Nvidia have made the biggest waves with their recent releases.

## Meta and Llama 3.2 Models

On September 25th, Meta introduced several exciting [updates](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/) to their AI model lineup, focusing on lightweight models and vision capabilities.

### Lightweight Models

First, Meta released impressive 1B and 3B parameter models that are best in class for their size. These models allow for near-instantaneous interactions on edge devices and can be used for processing large amounts of tokens with simple processing tasks.

The development process for these models involved pruning from Llama 3.1-8B, followed by knowledge distillation using the 8B and 70B models as teachers. The instruction-tuned versions underwent additional training with Supervised Fine-tuning (SFT), Direct Policy Optimization (DPO), and Rejection Sampling, incorporating synthetic data from Llama 3.1-405B.

These small models are easy to download and get playing with locally, but can be leveraged for large scale data processing tasks as they are much less expensive to run and still have great capabilities.

### Vision Models

Meta's new vision models come in 11B and 90B parameter sizes. Interestingly, these are built as adapters for the 8B and 70B text models, respectively, maintaining the same performance on text tasks while adding visual capabilities. This approach saved compute during training and allows for flexible deployment options.

The vision models were created by pretraining adapters and image encoders with noisy image and text data, followed by post-training similar to the text-only models. This has two advantages: For the same compute budget, Meta could train the models on more data, and the models are deployable on the same GPU's at the same time as their text-only models and develoeprs don't have to worry about regressions in their text-only tasks.

### Llama Guard Models

Meta also released new Llama Guard models for safety and moderation tasks, based on the 11B vision model and the 1B and 3B text models. These models are designed to help with safety and moderation tasks at the edge.

## Mistral's AI Innovations

Mistral has been on a similar research path as Meta, releasing models that are focused on similar efforts and qualities, small efficient models for the edge, and multimodal models as drop-in replacements for existing text-only models.

### Pixtral Model

Released on September 17th, [Pixtral](https://mistral.ai/news/pixtral-12b/) is a 12B parameter multimodal model. Similar to Meta's Llama 3.2 Vision approach, Pixtral was created as a drop-in replacement for Mistral-Nemo-11B, a text-only model. It features a 400M image encoder as an adapter, maintaining deployment flexibility. It performs well on a variety of multi-modal tasks.

Pixtral is available under the Apache 2.0 license and can be downloaded for the base model [here](https://huggingface.co/mistralai/Pixtral-12B-Base-2409) and for the instruct model behind a gated release [here](https://huggingface.co/mistralai/Pixtral-12B-2409).

### Mistral-Small v24.09

Also released on September 17th, 2024, [Mistral-Small v24.09](https://mistral.ai/news/september-24-release/) is a 22B parameter text model that bridges the gap between Mistral Nemo-11B and Mistral Large v2. This model offers substantial improvements in reasoning, code generation, and function calling compared to its predecessor. It's available under the Mistral Research License, and can be downloaded [here](https://huggingface.co/mistralai/Mistral-Small-Instruct-2409) behind the research gate.

### Ministral Models

On September 25th, 2024, Mistral introduced two new models: [Ministral 3B and Ministral 8B](https://mistral.ai/news/ministraux/).

Ministral 3B is a compact powerhouse, offering capabilities roughly equivalent to the original Mistral 7B but in just 3B parameters. Its instruct-tuned version outperforms Mistral-7B instruct models and shows superior performance to Llama 3.2-3B. However, it is not availabel on HuggingFace, and requires an Enterprise License from Mistral to access.

Ministral 8B represents a significant upgrade over the Mistral 7B series, offering a substantial leap in capabilities. According to benchmarks, it outperforms Llama 3.1-8B across of series of tasks. You can download the model [here] (https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) behind the gated access and run it on Vast to evaluate it for your workloads.

## Nvidia's Contribution to AI Advancement

Nvidia has also focused on 70B models to compete with frontier models like those from OpenAI and Anthropic.

### Llama 3.1-Nemotron-70B Models

Nvidia has [introduced](https://developer.nvidia.com/blog/new-reward-model-helps-improve-llm-alignment-with-human-preferences/) two notable models based on the Llama 3.1 architecture:

1. **Llama 3.1-Nemotron-70B-Reward**: This model was trained via Reinforcement Learning from Human Feedback (RLHF) to serve as a reward model. It currently holds the top position on the [RewardBench leaderboard](https://huggingface.co/spaces/allenai/reward-bench), showcasing its effectiveness in evaluating AI responses. It was also used to train the Instruct model, showing how well this model can be used to train other models synthetically.

2. **Llama 3.1-Nemotron-70B-Instruct**: Leveraging the reward model mentioned above, this instruct-tuned version has achieved impressive results. It currently leads the [ArenaHard Leaaderboard](https://huggingface.co/spaces/lmarena-ai/chatbot-arena-leaderboard/commit/2bba63b79d685a2d46696c2da0576a86a3cfc7c6), demonstrating its superior performance in complex instruction-following tasks.

The Llama 3.1-Nemotron-70B-Reward model was trained on the [HelpSteer-v2 dataset](https://arxiv.org/abs/2406.08673), which likely contributed to its strong performance as a reward model.

What's particularly noteworthy is that the Llama 3.1-Nemotron-70B-Instruct model has shown outstanding results in automatic benchmarks, outperforming even some of the most advanced models in the field, including Claude 3.5 Sonnet and GPT-4.

These developments from Nvidia show that methods to move beyond the GPT-4 wall of capabilities are not just being developed, but open sourced and available to developers to use wherever they use compute.

## Conclusion:

It's a great time to be a developer using AI. In the past few months, there have been major capability improvements for LLM's and Multimodal models, enabling tasks that were not possible before with open source models. And the release of many smaller language models that are still as capable as previous generation models means that the cost curve is coming down for deploying to production.

At Vast, we're very excited to see these advancements, and you can look out for more updates and building on top of these types of models for specific workflows in the coming months.


Open Source AI has evolved rapidly, with new models offering top quality and speed, challenging closed-source standards.

Latest AI Model Releases: September and October


# NVIDIA GeForce RTX 5090: Rumors and Speculations

As the year winds down, rumors are intensifying around NVIDIA's highly anticipated **GeForce RTX 5090** GPU. Industry insiders are divided on the release date, with some sources suggesting a launch just in time for Christmas, while other reports point to a formal announcement at **CES 2025** in the new year.

**NVIDIA's CEO Jensen Huang** will be hosting the keynote at CES on **January 6th, 2025** – which could be the perfect opportunity to launch the full trio of GPUs in the 50-series lineup: the **RTX 5090**, **5080**, and **5070**. But it's also possible that the **5090** itself will debut in time for last-minute holiday shopping.

The exact timing of the RTX 5090 release may be uncertain, but one thing is clear: NVIDIA's next flagship GPU is poised to redefine performance. Let's take a quick look at the speculation around this next-gen graphics powerhouse and its companions in the new lineup!

## NVIDIA RTX 5090: Power, Redefined

As we explored in a [previous blog post](https://vast.ai/article/nvidia-rtx-5090-leaks-rumors-gpu-performance), the flagship consumer-grade **RTX 5090** is set to dethrone its predecessor, the RTX 4090, with some truly remarkable upgrades. Here's what the various leaks and rumors suggest:

- Built on the cutting-edge **Blackwell architecture**, leaving Ada Lovelace behind
- **32GB** of next-gen **GDDR7 memory**, setting a new standard
- **512-bit** memory bus (compared to the 4090's 384-bit)
- **21,760 CUDA cores**, providing incredible horsepower for heavy workloads
- Base clock speed of ~**2900 MHz**, with potential boost clock speeds exceeding **3 GHz**
- Features the new **GB202 gaming chip**
- Powered by a single **12VHPWR connector**
- **600W** power consumption

The RTX 5090's little siblings in the 50 series are expected to pack a punch as well, balancing performance and cost for those who may want to upgrade without breaking the bank.

## NVIDIA RTX 5080 & 5070: Cost-effective Alternatives

The **RTX 5080** will likely be an excellent option for anyone needing a high-performance GPU without going all-in on the 5090. In fact, it reportedly could be even more powerful than initially expected. Rumors suggest it will feature a **32Gbps** memory speed – instead of the previously reported 28Gbps – and a maximum bandwidth of **1024GB/s** rather than **896GB/s**.

If these specs turn out to be true, then the RTX 5080 could outperform the RTX 4090 while also being more power-efficient, making it an exciting contender for high-performance users. To round out the rumored details, the **5080** will reportedly feature a **256-bit memory bus** with **16GB of GDDR7 VRAM**, around **10,000 CUDA cores**, and **400W** power consumption.

Finally, the **RTX 5070** will serve as a mid-range option without the extreme capabilities of the 5090 or 5080. It's rumored to offer a **192-bit memory bus** with **12GB of GDDR7 VRAM** (to the consternation of some who feel 12GB is surprisingly low), around **6,400 CUDA cores**, plus **250W** power consumption.

With all three cards – the **RTX 5090**, **5080**, and **5070** – likely being powered by a single 12VHPWR connector, built on the latest Blackwell architecture, and delivering everything from outstanding 4K resolution to potentially next-gen DLSS 4, these GPUs are expected to deliver the kind of leap in performance that will cater to a wide range of needs.

## Comparing the Rumored Specs: RTX 5090, 5080, and 5070

For easy reference, here's a side-by-side comparison of the rumored specifications for the entire 50-series lineup.

| Spec                  | RTX 5090 (Rumored)  | RTX 5080 (Rumored)  | RTX 5070 (Rumored) |
| --------------------- | ------------------- | ------------------- | ------------------ |
| **Architecture**      | Blackwell GB202-300 | Blackwell GB203-400 | Blackwell GB205    |
| **Memory**            | GDDR7               | GDDR7               | GDDR7              |
| **VRAM**              | 32GB                | 16GB                | 12GB               |
| **CUDA Cores**        | 21,760              | 10,752              | 6,400              |
| **Boost Clock**       | 3.0 GHz             | TBD                 | TBD                |
| **Memory Bus**        | 512-bit             | 256-bit             | 192-bit            |
| **Memory Speed**      | 28 Gbps             | 32 Gbps             | 28 Gbps            |
| **Bandwidth**         | 1792 GB/s           | 1024 GB/s           | 672 GB/s           |
| **Total Board Power** | 600W                | 400W                | 250W               |

> **Note:** All specifications mentioned are based on current rumors and reports. Final specifications may vary upon release.

## Can't Wait? We've Got You Covered

While we all wait for the official release (whether it's Christmas or CES), in the meantime you don't have to put your projects on hold. **Vast.ai** offers immediate access to high-performance GPUs through our [cloud GPU rental platform](https://vast.ai/).

Ready to supercharge your compute power today? Check out the flexible GPU rental options on Vast and get started in minutes – no waiting required!


GPU prices continue to increase, but there exists a happy and unexpected middle ground for AI applications.

NVIDIA RTX 5090: Out by Christmas? A Look at the Latest Rumors


# Serving Online Inference with SGLang on Vast.ai

## Background

SGLang is an open-source framework for serving language models, focusing on throughput for serving and batch workloads. This makes it ideal for building apps that serve multiple users and need to scale. SGLang is used by many high-tech companies for serving their Language Models to the public, most notably [X AI and their Grok Model.](https://venturebeat.com/ai/grok-2-gets-a-speed-bump-after-developers-rewrite-code-in-three-days/)

SGLang provides an OpenAI-compatible server, allowing you to easily integrate it into chatbots and other applications.

As companies develop their AI products, they often face challenges like rate limits and high costs when using these models. With SGLang on Vast, you can run your own models in the form factor you need, at a much more affordable price point. As inference demand grows with agents and complex workflows, SGLang on Vast excels in performance and affordability where it matters most.

This guide will show you how to set up SGLang to serve a language model on Vast. We reference a notebook that you can use [here](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/XEyMo8/72acf45925f8230b75195b7da9fff1884fa4052d/files/serving_vllm_on_vast.json)

## Setup and Querying

First, set up your environment and Vast API key:

```bash
pip install --upgrade vastai
```

Once you create your account, go [here](https://cloud.vast.ai/cli/) to find your API key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving a language model, we're looking for a machine with a static IP address, available ports to host on, and a single modern GPU with decent RAM, CUDA graphs take a lot of space and can affect deployment options for certain models. SGLang also requires CUDA version 12.2 or higher, so we'll filter for that as well. We will query the Vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 40 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.2'
```

## Deploying the Image

The easiest way to deploy this instance is to use the command line. Copy and paste a specific instance ID you choose from the list above into `instance-id` below. We'll also need to input the Huggingface API Token into `<Huggingface-Token>`.

```bash
vastai create instance <instance-id> --image lmsysorg/sglang:latest --env '-p 8000:8000 -e HF_TOKEN=<Huggingface-Token>' --disk 60 --args python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --port 8000
```

## Connecting and Testing

To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view_sglang.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view_sglang.png)

We will copy over the IP address and the port into the cell below.

```bash
# This request assumes you haven't changed the model. If you did, fill it in the "model" value in the payload json below
curl -X POST http://<IP-Address>:<Port>/v1/completions -H "Content-Type: application/json"  -d '{"model" : "meta-llama/Meta-Llama-3.1-8B-Instruct", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

In the [notebook](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/XEyMo8/72acf45925f8230b75195b7da9fff1884fa4052d/files/serving_vllm_on_vast.json), we include ways to call this model with requests or OpenAI

### Other things to look out for with other configurations

If you are downloading a model that needs authentication from the Hugging Face Hub, passing `-e HF_token=<Your-Read-Only-Token>` within Vast's `--env` variable string should help.

Sometimes the full context of a model can't be used given the space allocated for SGLang on the GPU + the model's size. In those cases, you might want to increase `--gpu-memory-utilization`, or decrease the `max-model-len`. Increasing `--gpu-memory-utilization` does come with CUDA OutOfMemory issues that can be hard to predict ahead of time.

We won't need either of these for this specific model and GPU configuration.

### Testing

Copy the IP address from your instance once it is ready, and then we can use the following code to call it. Note that while your server might have ports ready, the model might not have downloaded yet as it is much larger this time. You can check the status of this via the logs to see when it has started serving.

```bash
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'meta-llama/Meta-Llama-3.1-8B-Instruct',
    'prompt': 'Hello, how are you?',
    'max_tokens': 50,
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/completions', headers=headers, json=json_data)
print(response.content)
```

Or use OpenAI:

```bash
pip install openai
```

```python
from openai import OpenAI

# Modify OpenAI's API key and API base to use SGLang's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<Instance-IP-Address>:<Port>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
completion = client.completions.create(model="meta-llama/Meta-Llama-3.1-8B-Instruct",
                                      prompt="Hello, how are you?",
                                      max_tokens=50)
print("Completion result:", completion)
```

## Conclusions

Model inference is expensive, and leveraging more affordable compute/models makes a huge difference for engineering teams in terms of margins and shipping velocity.

Using SGLang on Vast is perfect for this, pairing Vast's access to affordable compute with the simplicity and state-of-the-art throughput of the SGLang backend.

SGLang is a great beginning to building Generative AI Apps. We will continue to explore using this tool more with Vast in future posts.

Llama-3 is already ready to go on Vast to start experimenting and building!


SGLang is an open-source framework for serving language models, focusing on throughput for serving and batch workloads.

Blog

Stretching Your Cloud Computing Budget: Introducing Reserved Instance Discounts

On-Demand vs Reserved: Striking the Right Balance

How Reserved Instance Discounts Work

Link to the Documentation

Seize the Savings

Continue Reading: