
Vast.ai provides a platform where developers can access a wide selection of GPUs from various hosts and data centers, offering competitive pricing to fit any budget. The cost and availability of GPUs are critical to the success and speed of generative AI projects. We’re excited to announce that [dstack](https://dstack.ai/) now includes integration with our [GPU marketplace](https://vast.ai/). dstack is an open-source cloud platform for training and deploying gen AI models. This integration enables our users to access an extensive selection of affordable GPUs, breaking down the barriers to entry for developers and ensuring that high costs do not stifle innovation.

# Diverse GPU Options at Your Fingertips

Vast.ai’s marketplace stands out by offering a wide selection of GPUs, ensuring that developers can select the best fit for their projects without cost constraints. The platform recommends GPU instances based on a comprehensive quality score but also gives you the flexibility to set a maximum price, ensuring that you stay within your budget.

# Empowering Developers with dstack and Vast.ai

With dstack 0.12.3, accessing Vast.ai's marketplace becomes a seamless part of the development workflow. Now, dstack’s intuitive CLI and API are equipped to tap into Vast.ai's resources, allowing you to train and deploy generative AI models with greater ease and less financial strain. This integration streamlines your development process, enabling you to utilize the cost benefits of Vast.ai without the hassle of complex cloud infrastructure management.

![](/uploads/dstack-top.png)

# Effortless Setup and Deployment

Using Vast.ai with dstack is easy and straightforward:

1. Log into your Vast.ai account and navigate to the 'Account' section to find your API Key.
2. With your API Key, configure the dstack backend according to the provided steps.

![](/uploads/dstack-bottom.png)

3. Once configured, begin using the robust features of dstack’s CLI and API.

This integration not only facilitates [fine-tuning](https://dstack.ai/docs/guides/fine-tuning/) and [deployment](https://dstack.ai/docs/guides/text-generation/) of models but also the creation of [development environments](https://dstack.ai/docs/guides/dev-environments/) that support interactive coding, making it a perfect fit for developers who prefer to work within their favorite IDEs.

# Seamless Switching Between Cloud Providers

The flexibility that dstack offers is unparalleled. You can now effortlessly switch to Vast.ai when it suits your project needs while retaining the option to use other cloud providers as required. This ensures that your workflow is as efficient and cost-effective as possible.

# Streamline Your AI Projects with dstack and Vast.ai

If you're looking for a simple and cost-effective way to train and deploy generative AI models, the combination of dstack and Vast.ai is ready for you. It's designed to save you time and money. With dstack’s new preview versions of the [fine-tuning](https://dstack.ai/docs/guides/fine-tuning/) and [text generation](https://dstack.ai/docs/guides/text-generation/) APIs, your development process just got even better.

Need a hand? Our [community](https://discord.gg/hSuEbSQ4X8) as well as dstack’s [community on Discord](https://discord.gg/u8SmfwPpMd) is available to offer support and answer your questions. We're committed to helping you make the most of what Vast.ai and dstack have to offer.


We’re excited to announce that dstack now includes integration with our GPU marketplace.

Accessing the GPU marketplace with Vast.ai and dstack


# Latest AI Model Releases: September and October

The world of Open Source AI has gotten many updates in the last month or so. There are now many new models with great quality:speed ratios and models that challenge the frontier of closed source models. This makes it even easier to build applications and automate workflows with open source models, which you can deploy on [Vast.ai](https://vast.ai). Meta, Mistral, and Nvidia have made the biggest waves with their recent releases.

## Meta and Llama 3.2 Models

On September 25th, Meta introduced several exciting [updates](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/) to their AI model lineup, focusing on lightweight models and vision capabilities.

### Lightweight Models

First, Meta released impressive 1B and 3B parameter models that are best in class for their size. These models allow for near-instantaneous interactions on edge devices and can be used for processing large amounts of tokens with simple processing tasks.

The development process for these models involved pruning from Llama 3.1-8B, followed by knowledge distillation using the 8B and 70B models as teachers. The instruction-tuned versions underwent additional training with Supervised Fine-tuning (SFT), Direct Policy Optimization (DPO), and Rejection Sampling, incorporating synthetic data from Llama 3.1-405B.

These small models are easy to download and get playing with locally, but can be leveraged for large scale data processing tasks as they are much less expensive to run and still have great capabilities.

### Vision Models

Meta's new vision models come in 11B and 90B parameter sizes. Interestingly, these are built as adapters for the 8B and 70B text models, respectively, maintaining the same performance on text tasks while adding visual capabilities. This approach saved compute during training and allows for flexible deployment options.

The vision models were created by pretraining adapters and image encoders with noisy image and text data, followed by post-training similar to the text-only models. This has two advantages: For the same compute budget, Meta could train the models on more data, and the models are deployable on the same GPU's at the same time as their text-only models and develoeprs don't have to worry about regressions in their text-only tasks.

### Llama Guard Models

Meta also released new Llama Guard models for safety and moderation tasks, based on the 11B vision model and the 1B and 3B text models. These models are designed to help with safety and moderation tasks at the edge.

## Mistral's AI Innovations

Mistral has been on a similar research path as Meta, releasing models that are focused on similar efforts and qualities, small efficient models for the edge, and multimodal models as drop-in replacements for existing text-only models.

### Pixtral Model

Released on September 17th, [Pixtral](https://mistral.ai/news/pixtral-12b/) is a 12B parameter multimodal model. Similar to Meta's Llama 3.2 Vision approach, Pixtral was created as a drop-in replacement for Mistral-Nemo-11B, a text-only model. It features a 400M image encoder as an adapter, maintaining deployment flexibility. It performs well on a variety of multi-modal tasks.

Pixtral is available under the Apache 2.0 license and can be downloaded for the base model [here](https://huggingface.co/mistralai/Pixtral-12B-Base-2409) and for the instruct model behind a gated release [here](https://huggingface.co/mistralai/Pixtral-12B-2409).

### Mistral-Small v24.09

Also released on September 17th, 2024, [Mistral-Small v24.09](https://mistral.ai/news/september-24-release/) is a 22B parameter text model that bridges the gap between Mistral Nemo-11B and Mistral Large v2. This model offers substantial improvements in reasoning, code generation, and function calling compared to its predecessor. It's available under the Mistral Research License, and can be downloaded [here](https://huggingface.co/mistralai/Mistral-Small-Instruct-2409) behind the research gate.

### Ministral Models

On September 25th, 2024, Mistral introduced two new models: [Ministral 3B and Ministral 8B](https://mistral.ai/news/ministraux/).

Ministral 3B is a compact powerhouse, offering capabilities roughly equivalent to the original Mistral 7B but in just 3B parameters. Its instruct-tuned version outperforms Mistral-7B instruct models and shows superior performance to Llama 3.2-3B. However, it is not availabel on HuggingFace, and requires an Enterprise License from Mistral to access.

Ministral 8B represents a significant upgrade over the Mistral 7B series, offering a substantial leap in capabilities. According to benchmarks, it outperforms Llama 3.1-8B across of series of tasks. You can download the model [here] (https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) behind the gated access and run it on Vast to evaluate it for your workloads.

## Nvidia's Contribution to AI Advancement

Nvidia has also focused on 70B models to compete with frontier models like those from OpenAI and Anthropic.

### Llama 3.1-Nemotron-70B Models

Nvidia has [introduced](https://developer.nvidia.com/blog/new-reward-model-helps-improve-llm-alignment-with-human-preferences/) two notable models based on the Llama 3.1 architecture:

1. **Llama 3.1-Nemotron-70B-Reward**: This model was trained via Reinforcement Learning from Human Feedback (RLHF) to serve as a reward model. It currently holds the top position on the [RewardBench leaderboard](https://huggingface.co/spaces/allenai/reward-bench), showcasing its effectiveness in evaluating AI responses. It was also used to train the Instruct model, showing how well this model can be used to train other models synthetically.

2. **Llama 3.1-Nemotron-70B-Instruct**: Leveraging the reward model mentioned above, this instruct-tuned version has achieved impressive results. It currently leads the [ArenaHard Leaaderboard](https://huggingface.co/spaces/lmarena-ai/chatbot-arena-leaderboard/commit/2bba63b79d685a2d46696c2da0576a86a3cfc7c6), demonstrating its superior performance in complex instruction-following tasks.

The Llama 3.1-Nemotron-70B-Reward model was trained on the [HelpSteer-v2 dataset](https://arxiv.org/abs/2406.08673), which likely contributed to its strong performance as a reward model.

What's particularly noteworthy is that the Llama 3.1-Nemotron-70B-Instruct model has shown outstanding results in automatic benchmarks, outperforming even some of the most advanced models in the field, including Claude 3.5 Sonnet and GPT-4.

These developments from Nvidia show that methods to move beyond the GPT-4 wall of capabilities are not just being developed, but open sourced and available to developers to use wherever they use compute.

## Conclusion:

It's a great time to be a developer using AI. In the past few months, there have been major capability improvements for LLM's and Multimodal models, enabling tasks that were not possible before with open source models. And the release of many smaller language models that are still as capable as previous generation models means that the cost curve is coming down for deploying to production.

At Vast, we're very excited to see these advancements, and you can look out for more updates and building on top of these types of models for specific workflows in the coming months.


Open Source AI has evolved rapidly, with new models offering top quality and speed, challenging closed-source standards.

Latest AI Model Releases: September and October


# Could the End Be Near for the NVIDIA GeForce RTX 4090?

Rumors are circulating that the high-end consumer GPU is on the verge of being discontinued. With production potentially stopping, the 4090 could soon be disappearing off the shelves to make way for NVIDIA's new flagship GPU: the [RTX 5090](https://vast.ai/article/nvidia-rtx-5090-out-by-christmas-rumors).

As with any unconfirmed reports, it's best to take this news with a grain of salt – but the possibility has already sparked plenty of buzz among GPU enthusiasts.

## What's Driving the Rumors?

Reports about the potential discontinuation of the RTX 4090 appear to have [originated](https://wccftech.com/nvidia-geforce-rtx-4090-4090d-discontinued-next-month-in-preparation-for-next-gen-rtx-5090-5090d-gpus/) on China’s Board Channels, a frequent source of supply chain rumors. According to these sources, NVIDIA may stop producing the RTX 4090, along with its China-specific RTX 4090D variant, as early as October, with inventory expected to dwindle through the rest of the year.

But given that it's already late October without any official announcements from NVIDIA, the RTX 4090's availability seems secure in the near term. Still, we can't rule out any sudden shifts as NVIDIA prepares for the release of the next-gen RTX 5000 series.

According to rumors, the launch date of the new [flagship Blackwell RTX 5090 GPU](https://vast.ai/article/nvidia-rtx-5090-out-by-christmas-rumors) will be either in late 2024 or in early January at CES 2025. That timing does line up with NVIDIA potentially pulling the plug on the RTX 4090 sooner rather than later. Plus, the move would allow add-in board (AIB) partners to clear their stock of the RTX 4090 ahead of the RTX 5000-series release.

## Potential Impact on Pricing and Availability

Ending production of the RTX 4090 now (or very soon) would give NVIDIA's partners the holiday season to move existing inventory. This could result in shortages during the lead-up to Christmas as supplies run low, causing prices to spike as stock dwindles.

That said, with the highly anticipated RTX 5090 on the horizon, many buyers may hold off and wait for the new flagship release before making any purchase decisions. There's a chance this could put downward pressure on RTX 4090 prices, making it a more attractive option in the meantime.

It's hard to predict how things will unfold. Without official confirmation from NVIDIA, everything is merely speculation – and for now, it's a waiting game.

## Exploring Your GPU Options

While the future of the RTX 4090 remains uncertain, and the next-gen RTX 5090 will likely come with a steep price tag, one thing is clear: our cloud GPU rental platform allows you to access powerful GPUs at the most competitive rates available.

You don't need to make any huge purchases or commit to a single machine. With Vast.ai, you have the flexibility to choose from a range of GPUs that best fit your project, budget, and performance requirements at any given moment. Whether you're fine-tuning deep learning models, running complex simulations, or pushing your gaming rig to new heights, we're here to connect you with the GPU compute power that meets your exact needs.

[Discover the possibilities](https://vast.ai/) with Vast.ai today.


Rumors are circulating that the high-end consumer GPU is on the verge of being discontinued.

NVIDIA RTX 4090 to Be Discontinued Sooner Rather Than Later?


# Security and Compliance at Vast.ai

At Vast.ai, your security is our priority. We're proud of our track record of excellence over the past six years serving clients worldwide while keeping up the highest standards of regulatory compliance.

In fact, here at Vast.ai, we're currently in the process of completing our SOC 2 Type 1 certification – further solidifying our commitment to data security and regulatory compliance.

Our [Compliance Policy](https://vast.ai/compliance) is designed to protect your data every step of the way. Here's how:

## Strengthening Security with Datacenter Partners

On our GPU cloud platform, our top-tier datacenter providers set the standard. These aren't your average hosts. We partner exclusively with vetted datacenter providers who maintain rigorous third-party compliance certifications. They hold a minimum of ISO 27001 certification – and many go further, maintaining compliance with industry-specific regulations in areas like healthcare, finance, and government.

You can trust that any datacenter provider you work with through Vast has undergone extensive due diligence. We audit their compliance regularly, ensuring that everything from physical access controls to continuous monitoring systems are in place. If you need documentation, we can provide the relevant certifications directly to your compliance team.

> To filter for these providers only on Vast, be sure to select the **"Secure Cloud (Only Trusted Datacenters)"** checkbox.

## Key Security Features of Our Datacenter Partners

To ensure the highest level of security for your data, our trusted datacenter partners implement a comprehensive set of protective measures. Here are the key features that set them apart:

### Third-Party Certifications

All of our datacenter partners are ISO 27001-certified, ensuring they meet globally recognized information security standards. They may also hold additional certifications, such as HIPAA, NIST, PCI, and HITRUST, as well as GDPR compliance.

### Physical and Environmental Security

Our partners manage their equipment in secure facilities with restricted physical access, regular access reviews and recertifications, continuous video monitoring with 90+ day retention, and stringent asset tracking and removal approval processes. Environmental controls, including fire suppression systems and power backups, are tested regularly to prevent disruptions.

### Enhanced Auditing

We conduct thorough audits of our datacenter partners, verifying their ownership structure, source of funds, and identity, while ensuring they adhere to best practices and maintain equipment in line with their certifications.

### Extended Legal Agreements

All datacenter hosts sign expanded hosting agreements that include a separate Data Processing Agreement (DPA) for added protection.

Other compliance measures include robust incident response protocols and regular security and compliance training for all employees.

## Securing Your Peace of Mind

Compliance can be complex and daunting – but we're here to help!

Whether you're working in healthcare, finance, or other sectors with stringent data protection requirements, we offer a level of security that you can trust for your most sensitive workloads. With a six-year track record of no major incidents, Vast.ai remains a trusted choice for Fortune 500 companies and individuals alike.

If you have questions about how Vast.ai's system fits within your compliance policy framework, or if you'd like more information about our vetted datacenter providers, we’d love to talk. Feel free to schedule a consultation using [this Calendly link](https://calendly.com/travis-_da/compliance?month=2024-10), or reach out to us at [compliance@vast.ai](mailto:compliance@vast.ai). Our team is ready to help you navigate your security and compliance needs with confidence.


At Vast.ai, your security is our priority. We're proud of our track record of excellence over the past six years serving clients worldwide while keeping up the highest standards of...

Blog

Accessing the GPU marketplace with Vast.ai and dstack

Diverse GPU Options at Your Fingertips

Empowering Developers with dstack and Vast.ai

Effortless Setup and Deployment

Seamless Switching Between Cloud Providers

Streamline Your AI Projects with dstack and Vast.ai

Continue Reading: