
Welcome to another update from the team at Vast.ai! We remain dedicated to continually refining our GPU rental platform, ensuring it not only meets but exceeds the expectations of our valued users.

To that end, we're currently still beta testing AMD support. If you'd like to test an AMD machine, simply select a ROCm template from the recommended templates. If you see no machines, select 'unverified' and you'll find some AMD servers online. Enjoy!

Back to regularly scheduled maintenance… As with any technology-driven service, occasional bugs come with the territory. Addressing these promptly remains a top priority. Here are some of the latest bug fixes we’ve implemented since our last update:

- Fixed our cluster rental form resizing issue on mobile
- Safeguards are now in place to prevent users from using instance copy with no SSH keys
- Expired instances now show as expired and not scheduling
- Fixed issue with instance price detail showing wrong pre-discount price
- Reserved instances now do not stop when primary balance goes negative as long as instance balance is positive
- Template editor now filters out CUDA tags that are known not to work

### Cloud GPU Rental Platform Updates

Here at Vast.ai, we also continuously strive to enhance the quality and functionality of our platform. This past month, we've introduced several key improvements that boost the flexibility and performance of our GPU rental marketplace. Below are some highlights of the latest changes:

- [Machine Hosting Setup guide](https://cloud.vast.ai/host/setup/) has been updated to include AMD support
- Added AMD ROCm templates for base ROCm and ROCm Pytorch
- AMD GPUs that are listed on Vast have been added to search filters
- New automated port testing for verified machines to check port mappings/settings
- Updated Docs with [Windows Powershell SSH](https://vast.ai/docs/use-cases/windows-ssh-scp-guide) directions
- Added sort options on host machine page along with sort by hostname
- Added display if instance is static/dynamic

Check out the full change log for our April 2024 product updates at the bottom of this page. We're confident these updates will enhance your experience with Vast!

If you have any questions about our services or need assistance maximizing your use of our platform, don't hesitate to get in touch. Whether you're a longtime user or just starting out, we're here to support all your GPU rental and hosting needs. Feel free to chat with us directly through the chatbox at the bottom right, email us at [support@vast.ai](mailto:support@vast.ai), or join our vibrant Discord community for immediate advice and resources.

### Change Log

- We are still beta testing AMD support. If you want to test an AMD machine, select a ROCm template from the recommended templates. If you see no machines, include unverified. You will see some AMD servers online.
- Updated host setup guide to include AMD support - [https://cloud.vast.ai/host/setup/](https://cloud.vast.ai/host/setup/)
- Added AMD ROCm templates for base ROCm and ROCm Pytorch
- Added a pop-up after clicking the instance-to-instance copy button for clarity
- Bugfix: resized SSH key fields on instance card
- Bugfix: fixed our cluster rental form resizing issue on mobile
- New automated port testing for verified machines to check port mappings/settings
- Negative credits are being shown to users now
- Cloud dialog can be properly scrolled
- AMD GPUs that are listed on Vast have been added to search filters
- Updated Docs with Windows Powershell SSH directions (easier than Putty) - [https://vast.ai/docs/use-cases/windows-ssh-scp-guide](https://vast.ai/docs/use-cases/windows-ssh-scp-guide)
- Bugfix: safeguards in place to prevent users from using instance copy with no SSH keys
- Bugfix: clarified cloud copy 'directory not found' and related errors
- Bugfix: fixed bug with ./vast.py create subaccount --explain
- Schedule maint command now supports human-readable time
- Added sort options on host machine page along with sort by hostname
- Added notification re: why credit card can't be removed for negative balance accounts
- Moved maintenance info icon under report icon on instance card
- Added amortized bandwidth cost estimate in dlp/$ and auto search metrics based on duration and disk size
- Improved Rclone cloud copy with better options. API & CLI now support most of the Rclone options
- Added display if instance is static/dynamic
- Bugfix: expired instances show as expired and not scheduling
- Bugfix: fixed scroll jump after closing and canceling cloud copy dialogue
- Bugfix: fixed issue with instance price detail showing wrong pre-discount price
- Bugfix: fixed issue with listing results becoming out of sync/delayed
- Bugfix: reserved instances now do not stop when primary balance goes negative as long as instance balance is positive
- Bugfix: fixed wrapping issue in template UI
- Bugfix: template editor now filters out CUDA tags that are known not to work


Ongoing product updates are an important part of what we offer at Vast.ai...

Vast Product Update | April 2024


NVIDIA's highly anticipated next-generation GH200 Grace Hopper Superchip platform is finally available. Perfectly suited for the era of accelerated computing and generative AI, it's essentially the world's most powerful GPU paired with the most versatile CPU. 

  

With its scalable design, the NVIDIA GH200 can handle the most complex generative AI workloads, from large language models (LLMs) and recommender systems to vector databases, graph neural networks (GNNs), and beyond. 

  

"To meet surging demand for generative AI, data centers require accelerated computing platforms with specialized needs," [said](https://nvidianews.nvidia.com/news/gh200-grace-hopper-superchip-with-hbm3e-memory) NVIDIA founder and CEO Jensen Huang. This is precisely the gap that the GH200 aims to fill. 

  

Huang added, "[It] delivers this with exceptional memory technology and bandwidth to improve throughput, the ability to connect GPUs to aggregate performance without compromise, and a server design that can be easily deployed across the entire data center."

  

### Groundbreaking Memory and Bandwidth

  

The GH200's architecture redefines high-performance computing (HPC). It integrates the cutting-edge Hopper GPU and the versatile Grace CPU into a single superchip – bridged by the lightning-fast, memory-coherent NVIDIA NVLink Chip-2-Chip (C2C) interconnect. 

  

The NVLink-C2C interconnect is the heart of the GH200 Grace Hopper Superchip. The power of its coherent memory is pretty incredible; the interconnect delivers 900GB/s of bidirectional CPU-GPU bandwidth. That's seven times the performance compared to the PCIe Gen5 connections found in conventional accelerated systems! Plus, interconnect power consumption is slashed by more than 5X.

  

With NVLink-C2C, applications can oversubscribe the GPU's memory, harnessing the high-bandwidth memory of the Grace CPU directly. The GH200 offers up to 480GB of LPDDR5X CPU memory, meaning that – combined with 96GB HBM3 or 144GB HBM3e – the Hopper GPU can directly access up to 624GB of fast memory. 




![](/uploads/wxoZpDKUOFRqBvNJkCLco6gy3-gBofvjPwirVSYuHDdHVBbFVT-JDjSOQHIYWTSV6VTnVj4U5kFJ9lhfUeHalOA5N0QRdDAV0TxByVawrhTyJID0yActvuTghOYzziRHm9xDCI0xIwY3ubuD5nRSEas.png)


(Image credit: NVIDIA)

  

Key features of the NVIDIA Grace CPU include:

  

- 2X the performance per watt of conventional x86-64 platforms
    
- 72 Neoverse V2 Armv9 cores with up to 480GB of server-class LPDDR5X memory with error-correction code (ECC)
    
- Up to 53% more bandwidth at one-eighth the power per GB per second, compared to an eight-channel DDR5 design
    

  

Then there's the H100 Tensor Core GPU, based on the new Hopper GPU architecture, which showcases several innovative highlights:

  

- Unprecedentedly fast matrix computations by the new fourth-generation Tensor Cores, enabling a wider range of AI and HPC tasks
    
- Up to 9X faster AI training and up to 30X faster AI inference with its new Transformer Engine compared to the prior-generation NVIDIA A100
    
- Maximized quality of service for smaller workloads thanks to secure Multi-Instance GPU (MIG) partitioning the GPU into isolated, right-size instances
    

  

In a nutshell, if you need raw power, you've got it with the GH200. Unfortunately, it's so new that there hasn't really been much comprehensive benchmarking. But let's take a look anyway. 

### Benchmarks: Pitting the GH200 Against its Main Rivals

  

On the Linux benchmark site Phoronix, author Michael Larabel [published](https://www.phoronix.com/review/nvidia-gh200-gptshop-benchmark) some initial HPC benchmarks run on a GH200 workstation sourced from GPTshop.ai. Testing was done on the Grace CPU performance only.

  

The GH200 system in question was tested with 72 cores, a Quanta S74G motherboard, 480GB of RAM, and 960GB + 1920GB SAMSUNG SSD drives. (A complete description of specs and environment can be found on the Phoronix site at the above link.) Again, this benchmarking was merely preliminary, as it focused on CPU performance, and no power consumption numbers are available – but there were some interesting results nonetheless. 

  

According to the standard [HPCG](https://www.hpcg-benchmark.org/) memory bandwidth benchmark, the GH200 Grace CPU came in at a solid 41.7 GFLOPS. 

  

  ![](/uploads/OatdrZqcQK_Bp41PV_DBcUFPeSRnvYSJpF4O1uc1Q6D_dLyr4ESSPbeS5HzrYaK92ybZSAVi3ma0rKQ0FmlpXcgDBrhGdY06dP5C980mThA-hIlAexWMgn0yiXt3GXod1eAklyI8p1HdjCGyRpoEw-U.png)

Results for NVIDIA GH200 running HPCG benchmark. (Image credit: Phoronix)

5vFBmCRUcbfgb9PR7NpFoXpSeRSFqr47lr9V1YtOG-k2Aus1jPY1T217XD_EnBKKZKnn8MTFUs-Jx2lMYVR_8Ek4Q9eQcM-kml5l0C2B4wfzwdytunAxtEcGZNfi_pp-GQKy8jTPCrEpqRUtlrk8OXw

Another notable result came from the [NWChem](https://www.nwchem-sw.org/) benchmark, with the GH200 coming in second place at 1403.5 seconds.

  

![](/uploads/5vFBmCRUcbfgb9PR7NpFoXpSeRSFqr47lr9V1YtOG-k2Aus1jPY1T217XD_EnBKKZKnn8MTFUs-Jx2lMYVR_8Ek4Q9eQcM-kml5l0C2B4wfzwdytunAxtEcGZNfi_pp-GQKy8jTPCrEpqRUtlrk8OXw.png)

Results for NVIDIA GH200 running NWChem benchmark. (Image credit: Phoronix)

  

On a geometric mean basis across all the benchmarks, the GH200 Grace CPU performance was perfectly respectable overall. 

  
![](/uploads/O_VgErmTsszfyrnSV20nbY-BdZb4QvnVJe7LOp55l33JKKI3JFoQSRpWU2lia3g_QlqKTwi4_1VXjUQ2gE1iHKNYy_A4JkgtMsPnn5uzc7TBf9JjVzH1rifiNFA_n17d4TI_t7MB6bsjIMCMYGMcrXo.png)

(Image credit: Phoronix)

  

A set of GPU benchmarks run by researcher Simon Butcher [compared](https://blog.hpc.qmul.ac.uk/benchmarking-grace-hopper-nodes/) performance of the PyTorch ResNet50 [training recipes](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnet50v1.5) published by NVIDIA. The ImageNet 2012 dataset (150GB) was used, and training was run for 90 epochs, or about an hour. The GH200 showed excellent performance here.

  

![](/uploads/qlwGIbeeRNXXquUTSoF6EDJmoss443fwqWUEQg9PtgAVPwIHTNTbi0U5BsEFx3Wx2dX10b0gfTdlgMfQxU_kHulwTU5QaOGlz7Oy9U8GBpdkvj4pCE7wjW7PJzfU_lTK-mdd3VUlbSjUaXEx2RVtloA.png)

(Image credit: Simon Butcher, Queen Mary University of London)

  

NVIDIA also [published](https://resources.nvidia.com/en-us-dgx-gh200/nvidia-grace-hopper-superchip-datasheet) some performance comparisons that may be of interest.

  

![](/uploads/dXjLeW3nCbOSpjMxABHSZ6LP84CLuzszoNLvi2m9CD1Dxf9GgTYk17wr2YlxWxqNM5xF3U6GXV-E6JFILwTqmj4zEmdd9Rv5MNAUNK9xaa6YeTfM9krmcLeMRaUQ4eF8qt_G0-xW81JoRiaEz4157xQ.png)

(Image credit: NVIDIA)

  

### Final Thoughts

  

The NVIDIA GH200 Grace Hopper Superchip delivers the performance required for truly giant-scale AI and HPC applications running terabytes of data. Whether you're a scientist, an engineer, or a massive data center, you'll find it up to the task. 

As for what the future holds? Nvidia [recently announced](https://www.tomshardware.com/pc-components/gpus/nvidias-next-gen-ai-gpu-revealed-blackwell-b200-gpu-delivers-up-to-20-petaflops-of-compute-and-massive-improvements-over-hopper-h100) the GH200's successor: Grace Blackwell B200, the next-generation data center and AI GPU.

  

Here at Vast.ai, we're proud to say that we recently added ARM64 and GH200 support to our lineup. On our GPU rental platform, just about anyone can get access to top-tier GPUs at affordable rates for a variety of computational needs. You can skip the cost-prohibitive investment in hardware and get right to work today – we'd love to help you discover the possibilities!

NVIDIA's highly anticipated next-generation GH200 Grace Hopper Superchip platform is finally available. Perfectly suited for the era of accelerated computing and generative AI, it's essentially the world's most powerful GPU paired with the most versatile CPU.

Examining the NVIDIA GH200


It's time for another product update! Here at Vast.ai, we strive to keep improving our product for the benefit of new and old customers alike, so we can continue to serve as your reliable gateway to affordable and powerful cloud GPU resources.

One of our big updates this month is that we've added beta hosting support for AMD GPUs! For AMD-specific directions, please see our updated hosting guide. If you have Radeon or Radeon Instinct GPUs, you can now run our Daemon software. (Please reach out via the chatbox at the bottom right if you experience any issues while setting up AMD GPUs.)

As always, we address bugs as soon as we're aware of them, in order to maintain the highest standards of performance and reliability on Vast. The following are some of our latest bug fixes:

- SHA256 Docker images now work
- Search results by geolocation issue fixed
- Fixed the VRAM cost-per-hour filter and sort setting in GUI
- Instance duration bug fixed so 3-day default is now accurate
- Fixed issue where logging in with OAuth + 2FA fails
- Issue with copy progress bar being lost on page refresh now fixed

### NVIDIA Cloud GPU Updates

With each update, we're committed to enhancing performance, expanding capabilities, and streamlining user experience. Our goal is to continually evolve to meet our customers' needs. Below are some highlights of the latest enhancements we've made to our website and platform:

- Reorganized our Docs and added a more detailed walkthrough of all pages inside the console
- Updated hosting documentation with requirements for datacenter status
- Added machine listing support for "geforce rtx 4080 super", "geforce rtx 4070 ti super", "quadro k620", "quadro k2200" to gpu list
- Added the ability to stop/start offline instances
- Autoscaler now supports templates
- Added cpu_ghz filter
- Cloud copy settings now remembered from prior use

For a full list of all improvements and bug fixes since our last update, check out the March 2024 change log at the bottom of this page.

If you're new to Vast, we highly recommend joining our active Discord community – whether for helpful resources and support or just for fun. As of this month, we even have a new Discord bot named @Vasti that has been trained on our Docs and is always available for different support requests. Simply tag it in a public channel or send it a direct message!

For any other questions or assistance getting started with GPU rental or hosting on Vast, feel free to reach out at your convenience – we would be delighted to help you in any way we can. Drop us a line via the chatbox at the bottom right or email us at support@vast.ai any time!

### Change Log

- Added a reactive 2FA button
- Updated hosting documentation with requirements for datacenter status: https://vast.ai/docs/hosting/datacenter
- Added machine listing support for "geforce rtx 4080 super", "geforce rtx 4070 ti super", "quadro k620", "quadro k2200" to gpu list
- Added invoice info section to account and earnings page
- Added improvements to invoicing: refresh button, calendar range, re-trigger to get last invoice
- UI now shows recent failed billing attempts and explanation for when auto-billing is disabled
- Bugfix: fixed copy between two machines on the same network with the same IP
- Bugfix: fixed an issue in the templates with Extra_Filters not parsing non-floats properly
- Added the ability to stop/start offline instances
- Added CLI command 'create template' that can create a template, use --help for info
- Autoscaler now supports templates
- Added a 'machine_id' filter for hosts to find their own machine
- Added a button to show more results (so hosts can find their offers in relative rankings)
- Improved the add credit UI widget
- Improved max instance duration and price and other filters/sliders
- Added cpu_ghz filter
- Bugfix: SHA256 Docker images now work
- Bugfix: search results by geolocation issue fixed
- Bugfix: fixed issue with instance create scheduling
- Bugfix: fixed the VRAM cost-per-hour filter and sort setting in GUI
- Bugfix: added proper spacing support for GPU names in CLI searching
- Bugfix: fixed instance duration bug so 3-day default is accurate
- Fairly major reorganization of our docs, including a more detailed walkthrough of all pages inside the console - https://vast.ai/docs/console/introduction
- Added CLI function cancel_maint for hosts to be able to cancel a maintenance window
- Improved periodic auto cleanup of Docker storage for machines
- Bugfix: fixed 403 issue when creating new template
- Bugfix: host payout PDFs had total earnings and total payouts mislabeled, now fixed
- Bugfix: reserved pricing was displaying a NaN value, now it is showing correct values
- Bugfix: issues with instance→instance copy UI state
- Bugfix: fixed issue where logging in with OAuth + 2FA fails
- Bugfix: fixed billing PDF "include payments" issue where only auto billings were showing up
- Bugfix: crypto payments now properly exported in PDF
- Our new Discord bot named @Vasti has been trained on our Docs and can be handy for different support requests – give it a try by tagging it in a public channel or sending it a DM on Discord
- Added day to PDF invoices
- Bugfix: fixed issue with tag menu in template editor
- Bugfix: fixed invalid queries being sent from the webUI
- Added beta hosting support for AMD GPUs! (Our hosting guide is updated with AMD-specific directions. If you have Radeon or Radeon Instinct GPUs you can now run our Daemon software. Please reach out on our website chat if you experience any issues while setting up AMD GPUs.)
- Added an SSH key manager for each instance UI; click on the key icon to see what SSH keys are active for that instance
- Added a reminder to refresh the UI when there are deployed changes to the UI available
- Added UI help when using the instance copy command
- Cloud copy settings now remembered from prior use
- Bugfix: issue with copy progress bar being lost on page refresh now fixed


Vast Product Update | March 2024


We're pleased to reveal the latest round of updates here at Vast.ai! Our mission is to provide our customers with the best possible GPU rental platform, and we believe that every update is an opportunity to make our product shine that much more. For instance, we've recently added ARM64 and GH200 support – so we're now able to support ARM64 platforms including the Grace Hopper superchip!

But before we delve into some of our bigger updates below, first up on the agenda are bug fixes. We strive to address every minor hiccup and bug as quickly as possible, so that you can enjoy a seamless, uninterrupted experience on Vast. Here are a few of our latest bug fixes:

- Linking to docs now aligns correctly
- Random logouts from the webUI have been fixed
- Users don't see the host setup command unless they are in a hosting account
- Fixed an off-by-1 error with the CLI command ssh-url
- Everywhere a template name is mentioned, it is now linkable with a hash ID
- An issue with number of referrals not displaying has been corrected
- Fixed issue with template editor in search interface

![](/uploads/feb24Image.png)

### NVIDIA Cloud GPU Updates

Over the past few weeks, we've made some important enhancements to our platform. It's our hope that these updates will help save you time and effort, as well as maximize your goals and results on Vast. The following are some highlights of the latest changes:

- As mentioned, we've added ARM64 and GH200 support; we now support ARM64 platforms including the Grace Hopper superchip
- Added a template for NVIDIA RAPIDS - [https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593](https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593)
- Changed Autoscaler CLI commands to work with template linking
- Added a search feature for templates and hover help (tooltip) to the template page
- Implemented flexible date format for CLI so it isn't just Unix epoch time
- Improved search API for performance
- Added a DLPerf filter

Please check out the full change log for our February 2024 product updates at the bottom of this page. Getting to know the latest capabilities will help you make the most of your time on Vast!

If you're looking for affordable, on-demand cloud GPU rental, you've come to the right place – and if you need some assistance getting started, we have the experience and know-how to guide you through whatever your project requires. Join our thriving Discord community for support or reach out to our customer service team to get all your questions answered! We would love to help you explore the possibilities that Vast holds for you today. Please feel free to contact us through the chatbox at the bottom right or drop us a line at [support@vast.ai](mailto:support@vast.ai) whenever you'd like!

### Change Log

- Added ARM64 and GH200 support; we now support ARM64 platforms including the Grace Hopper superchip
- Added a CLI parameter on create instance for a template hash; i.e. vastai create instance --template_hash HASH
- WebGUI uses cancel_unavail by default (instances are not created stopped)
- Added a template for NVIDIA RAPIDS - [https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593](https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593)
- Upgraded our SD guides with ai-dock template and info on setting a strong password
- Improved Autoscaler to use template info more completely
- Changed Autoscaler CLI commands to work with template linking
- Bugfix: args launch now doesn't accept an onstart script
- Bugfix: linking to docs now aligns correctly
- Bugfix: random logouts from the webUI fixed
- Bugfix: users don't see host setup command unless they are in a hosting account
- Bugfix: fixed an issue with number of referrals not displaying
- Bugfix: fixed host link to Stripe connect earnings dashboard
- Bugfix: on instance to instance copy, copy to /root is fixed
- Bugfix: fixed issue with template editor in search interface
- Added a search feature for templates
- Added a "Create new template" feature on templates
- Improved search API for performance
- Added the team name and member name for the API/team contexts on the account page
- Added hover help to the template page
- Added DLPerf filter
- Added flexible date format for CLI so it isn't just Unix epoch time
- Bugfix: everywhere a template name is mentioned, it is now linkable with a hash ID
- Bugfix: loading bar was showing before transfer commands, it is now fixed
- Bugfix: template tag dropdown selector is fixed, as it wasn't updating correctly
- Bugfix: fixed an off-by-1 error with the CLI command ssh-url


February 2024 Product Update


The rise of GPU-centric tasks in Python environments means a balanced computing setup is as important as ever. While a powerful GPU handles parallel processing tasks efficiently, a good CPU is necessary to prevent bottlenecks and ensure smooth performance. A common misconception is that a strong GPU can make up for a weak CPU, but this is not the case. A weak CPU can stall, creating delays that lower performance, regardless of the GPU's strength.

The real issue lies in the CPU's role in managing a program’s workflow, including memory management, I/O operations, and providing instructions to the GPU. When the CPU lags, it can't feed instructions to the GPU quickly enough, causing bottlenecks. This problem is especially true in Python environments, whose language limitations can worsen the bottleneck.

# Why Does a Weak CPU Lead to Bottleneck?

In computing, the CPU and GPU work together. The CPU directs operations and sends instructions to the GPU for processing. However, when a CPU is weak or slow, it can't keep up with the GPU, causing a bottleneck. This means the GPU has to wait for the CPU to send instructions, which slows down the entire process.

Python environments can exacerbate this issue due to certain limitations. For instance, Python's Global Interpreter Lock (GIL) can further slow down the CPU, making the bottleneck issue more noticeable. The efficiency of running Python tasks largely depends on the CPU's capabilities. A stronger CPU can send instructions to the GPU faster, keeping the process running smoothly and ensuring better performance.

# Introducing AMD’s Threadripper: A More Powerful CPU

The [AMD Ryzen Threadripper 7000 Series and PRO 7000 WX-Series](https://www.amd.com/en/newsroom/press-releases/2023-10-19-amd-introduces-new-amd-ryzen-threadripper-7000-ser.html), built on the advanced "Zen 4" architecture, offer high core and thread counts. This design significantly alleviates CPU bottlenecks, especially in Python environments where the CPU often stalls due to the language's inefficiencies.

Threadripper's superior multi-core performance allows for more efficient instruction management, reducing the wait time for the GPU. This results in significantly better Python execution times and smoother operations across various computational scenarios.

# Bridging the Gap with Vast.ai

Many individuals lacking access to high-performance CPUs like Threadripper may experience bottlenecks in their GPU-accelerated tasks. The inefficiencies of a weaker CPU can significantly slow down processing, making certain computational tasks a slog.

[Vast.ai](https://vast.ai/) presents a viable solution to this problem. It provides a platform where users can rent high-performance GPU resources, which can be paired with superior CPU capabilities. By offering a gateway to better hardware resources, Vast.ai helps in overcoming the limitations imposed by less capable CPUs. Users can thus experience smoother, more efficient processing, enabling them to execute their tasks more effectively and in a timely manner. This way, Vast.ai plays a crucial role in alleviating the hardware constraints that many individuals face, bridging the gap between their current setup and the ideal computational environment.

# Final Thoughts

A robust CPU is crucial for maximizing GPU utilization, particularly in Python environments where a weaker CPU can cause bottlenecks. [The AMD Ryzen Threadripper](https://www.amd.com/en/products/processors/workstations/ryzen-threadripper.html) significantly alleviates this issue with its high core and thread counts. For those without access to such high-performance CPUs, [platforms like Vast.ai](https://vast.ai/) offer the opportunity to rent superior GPU and CPU resources, providing a practical solution to hardware limitations. Exploring these options can lead to better performance in GPU-centric tasks and a smoother computing experience overall.


Introducing AMD’s Threadripper: A More Powerful CPU

Threadripper: Balancing CPU and GPU for Optimized Performance in Python


## Introduction to Oobabooga

Oobabooga is a tool designed for interacting with Large Language Models (LLMs), capable of understanding and generating text. With its user-friendly web interface, it's perfect for non-technical users interested in harnessing the power of LLMs. [Oobabooga supports a wide range of models](https://vast.ai/article/deploying-oobabooga-for-running-large-language-models), excelling in various use-cases like prompting, loading, and managing models.

### Why Run Oobabooga in API Mode vs Standard Mode?

Choosing API Mode for Oobabooga on Vast.ai offers several advantages, including enhanced scalability for multitasking, seamless tool integration via user-friendly endpoints, and streamlined workflow automation. Given these benefits, let's dive into how to actually run Oobabooga on Vast.ai.

### Why Choose Vast.ai for Oobabooga Deployment?

Running Oobabooga can be GPU-intensive, posing challenges for those without high-end GPUs or the financial resources to acquire them. That's where Vast.ai, a cloud-based GPU rental platform, comes in. By deploying Oobabooga on Vast.ai, you gain access to powerful, on-demand GPU resources and pay only for what you use, enabling complex text-generation tasks without hardware limitations.

### Launching Oobabooga on Vast.ai: A Step-by-Step Guide

#### Logging In to Vast.ai:

1. [Log into your Vast.ai account](https://vast.ai/).
2. Ensure you have enough credits for the task.

#### Selecting Oobabooga Template:

1. Click into the console.
2. Click ‘EDIT IMAGE & CONFIG.’

![](/uploads/Instance-Configuration.png)

3. Select the recommended template for Oobabooga.

![](/uploads/kO9lpdJwnxVSVjFnuprmaBQ3IJ6K2ALmORoSIyyWt3fG2gdPRIqrrFAVT8O6uCVqQwdTgPm6tRvIgbJFHmeaS8G3hIZxsKPBqC18fklDy62VIw8cf7OmhXiVkUcA.png)

4. Click on ‘EDIT’

![](/uploads/of-these-templates.png)

5. Scroll down to ‘On-start Script’ and type in ‘--api’ before the final semicolon

![](/uploads/vast.ai.png)

6. Click on ‘Select And Save’

![](/uploads/SELECT-AND-SAVE.png)

#### Allocating Disk Storage:

1. Adjust disk storage to a minimum of 80GB.

![](/uploads/EDIT-IMAGE-&-CONFIG.png)

#### Choosing a Suitable GPU:

1. [Review the Large Language Model (LLM) leaderboard on Hugging Face](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) to understand the GPU RAM requirements of your chosen model.
2. Filter for the total GPU RAM on the machine. Oobabooga can combine the GPU RAM on multi-GPU instances to load the model.

![](/uploads/GPU-Resources.png)

#### Launching Oobabooga Instance:

1. Click ‘Rent’.

![](/uploads/Auto-Sort.png)

2. Go to ‘Instances’

![](/uploads/cgNf2u4ui_EUkbfjH86kj1qltiHu7-CP32MKL-z5YTFjvc5fYPAPDnC06HXeuzG8R4YczJhTNVKkPB1LBqmxpt7g9nw_l9IpfYPFUaahDz3fYLMA3kQLw199j3Qt.png)

3. Click ‘Open’ on your newly created instance.

![](/uploads/entation.png)

## Final Thoughts

Oobabooga's user-friendly interface and versatile support for various LLMs make it an invaluable tool for text generation tasks. When considering deployment options, Vast.ai emerges as a cost-effective solution, providing access to powerful GPU resources on-demand. Following this guide, users can easily set up Oobabooga on Vast.ai, bypassing any hardware hurdles, and making using Oobabooga straightforward and simple.


Guide to deploying Oobabooga in API Mode on Vast.ai

Posts about: Vast

Vast Product Update | April 2024

Examining the NVIDIA GH200

Vast Product Update | March 2024

February 2024 Product Update

Threadripper: Balancing CPU and GPU for Optimized Performance in Python

Running Oobabooga in API Mode on Vast.ai

Posts about: Vast

Vast Product Update | April 2024

Examining the NVIDIA GH200

Vast Product Update | March 2024

February 2024 Product Update

Threadripper: Balancing CPU and GPU for Optimized Performance in Python

Running Oobabooga in API Mode on Vast.ai

Subscribe for our product updates.