
## Introduction to Oobabooga

Oobabooga is a tool designed for interacting with Large Language Models (LLMs), capable of understanding and generating text. With its user-friendly web interface, it's perfect for non-technical users interested in harnessing the power of LLMs. [Oobabooga supports a wide range of models](https://vast.ai/article/deploying-oobabooga-for-running-large-language-models), excelling in various use-cases like prompting, loading, and managing models.

### Why Run Oobabooga in API Mode vs Standard Mode?

Choosing API Mode for Oobabooga on Vast.ai offers several advantages, including enhanced scalability for multitasking, seamless tool integration via user-friendly endpoints, and streamlined workflow automation. Given these benefits, let's dive into how to actually run Oobabooga on Vast.ai.

### Why Choose Vast.ai for Oobabooga Deployment?

Running Oobabooga can be GPU-intensive, posing challenges for those without high-end GPUs or the financial resources to acquire them. That's where Vast.ai, a cloud-based GPU rental platform, comes in. By deploying Oobabooga on Vast.ai, you gain access to powerful, on-demand GPU resources and pay only for what you use, enabling complex text-generation tasks without hardware limitations.

### Launching Oobabooga on Vast.ai: A Step-by-Step Guide

#### Logging In to Vast.ai:

1. [Log into your Vast.ai account](https://vast.ai/).
2. Ensure you have enough credits for the task.

#### Selecting Oobabooga Template:

1. Click into the console.
2. Click ‘EDIT IMAGE & CONFIG.’

![](/uploads/Instance-Configuration.png)

3. Select the recommended template for Oobabooga.

![](/uploads/kO9lpdJwnxVSVjFnuprmaBQ3IJ6K2ALmORoSIyyWt3fG2gdPRIqrrFAVT8O6uCVqQwdTgPm6tRvIgbJFHmeaS8G3hIZxsKPBqC18fklDy62VIw8cf7OmhXiVkUcA.png)

4. Click on ‘EDIT’

![](/uploads/of-these-templates.png)

5. Scroll down to ‘On-start Script’ and type in ‘--api’ before the final semicolon

![](/uploads/vast.ai.png)

6. Click on ‘Select And Save’

![](/uploads/SELECT-AND-SAVE.png)

#### Allocating Disk Storage:

1. Adjust disk storage to a minimum of 80GB.

![](/uploads/EDIT-IMAGE-&-CONFIG.png)

#### Choosing a Suitable GPU:

1. [Review the Large Language Model (LLM) leaderboard on Hugging Face](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) to understand the GPU RAM requirements of your chosen model.
2. Filter for the total GPU RAM on the machine. Oobabooga can combine the GPU RAM on multi-GPU instances to load the model.

![](/uploads/GPU-Resources.png)

#### Launching Oobabooga Instance:

1. Click ‘Rent’.

![](/uploads/Auto-Sort.png)

2. Go to ‘Instances’

![](/uploads/cgNf2u4ui_EUkbfjH86kj1qltiHu7-CP32MKL-z5YTFjvc5fYPAPDnC06HXeuzG8R4YczJhTNVKkPB1LBqmxpt7g9nw_l9IpfYPFUaahDz3fYLMA3kQLw199j3Qt.png)

3. Click ‘Open’ on your newly created instance.

![](/uploads/entation.png)

## Final Thoughts

Oobabooga's user-friendly interface and versatile support for various LLMs make it an invaluable tool for text generation tasks. When considering deployment options, Vast.ai emerges as a cost-effective solution, providing access to powerful GPU resources on-demand. Following this guide, users can easily set up Oobabooga on Vast.ai, bypassing any hardware hurdles, and making using Oobabooga straightforward and simple.


Guide to deploying Oobabooga in API Mode on Vast.ai

Running Oobabooga in API Mode on Vast.ai


NVIDIA's highly anticipated next-generation GH200 Grace Hopper Superchip platform is finally available. Perfectly suited for the era of accelerated computing and generative AI, it's essentially the world's most powerful GPU paired with the most versatile CPU. 

  

With its scalable design, the NVIDIA GH200 can handle the most complex generative AI workloads, from large language models (LLMs) and recommender systems to vector databases, graph neural networks (GNNs), and beyond. 

  

"To meet surging demand for generative AI, data centers require accelerated computing platforms with specialized needs," [said](https://nvidianews.nvidia.com/news/gh200-grace-hopper-superchip-with-hbm3e-memory) NVIDIA founder and CEO Jensen Huang. This is precisely the gap that the GH200 aims to fill. 

  

Huang added, "[It] delivers this with exceptional memory technology and bandwidth to improve throughput, the ability to connect GPUs to aggregate performance without compromise, and a server design that can be easily deployed across the entire data center."

  

### Groundbreaking Memory and Bandwidth

  

The GH200's architecture redefines high-performance computing (HPC). It integrates the cutting-edge Hopper GPU and the versatile Grace CPU into a single superchip – bridged by the lightning-fast, memory-coherent NVIDIA NVLink Chip-2-Chip (C2C) interconnect. 

  

The NVLink-C2C interconnect is the heart of the GH200 Grace Hopper Superchip. The power of its coherent memory is pretty incredible; the interconnect delivers 900GB/s of bidirectional CPU-GPU bandwidth. That's seven times the performance compared to the PCIe Gen5 connections found in conventional accelerated systems! Plus, interconnect power consumption is slashed by more than 5X.

  

With NVLink-C2C, applications can oversubscribe the GPU's memory, harnessing the high-bandwidth memory of the Grace CPU directly. The GH200 offers up to 480GB of LPDDR5X CPU memory, meaning that – combined with 96GB HBM3 or 144GB HBM3e – the Hopper GPU can directly access up to 624GB of fast memory. 




![](/uploads/wxoZpDKUOFRqBvNJkCLco6gy3-gBofvjPwirVSYuHDdHVBbFVT-JDjSOQHIYWTSV6VTnVj4U5kFJ9lhfUeHalOA5N0QRdDAV0TxByVawrhTyJID0yActvuTghOYzziRHm9xDCI0xIwY3ubuD5nRSEas.png)


(Image credit: NVIDIA)

  

Key features of the NVIDIA Grace CPU include:

  

- 2X the performance per watt of conventional x86-64 platforms
    
- 72 Neoverse V2 Armv9 cores with up to 480GB of server-class LPDDR5X memory with error-correction code (ECC)
    
- Up to 53% more bandwidth at one-eighth the power per GB per second, compared to an eight-channel DDR5 design
    

  

Then there's the H100 Tensor Core GPU, based on the new Hopper GPU architecture, which showcases several innovative highlights:

  

- Unprecedentedly fast matrix computations by the new fourth-generation Tensor Cores, enabling a wider range of AI and HPC tasks
    
- Up to 9X faster AI training and up to 30X faster AI inference with its new Transformer Engine compared to the prior-generation NVIDIA A100
    
- Maximized quality of service for smaller workloads thanks to secure Multi-Instance GPU (MIG) partitioning the GPU into isolated, right-size instances
    

  

In a nutshell, if you need raw power, you've got it with the GH200. Unfortunately, it's so new that there hasn't really been much comprehensive benchmarking. But let's take a look anyway. 

### Benchmarks: Pitting the GH200 Against its Main Rivals

  

On the Linux benchmark site Phoronix, author Michael Larabel [published](https://www.phoronix.com/review/nvidia-gh200-gptshop-benchmark) some initial HPC benchmarks run on a GH200 workstation sourced from GPTshop.ai. Testing was done on the Grace CPU performance only.

  

The GH200 system in question was tested with 72 cores, a Quanta S74G motherboard, 480GB of RAM, and 960GB + 1920GB SAMSUNG SSD drives. (A complete description of specs and environment can be found on the Phoronix site at the above link.) Again, this benchmarking was merely preliminary, as it focused on CPU performance, and no power consumption numbers are available – but there were some interesting results nonetheless. 

  

According to the standard [HPCG](https://www.hpcg-benchmark.org/) memory bandwidth benchmark, the GH200 Grace CPU came in at a solid 41.7 GFLOPS. 

  

  ![](/uploads/OatdrZqcQK_Bp41PV_DBcUFPeSRnvYSJpF4O1uc1Q6D_dLyr4ESSPbeS5HzrYaK92ybZSAVi3ma0rKQ0FmlpXcgDBrhGdY06dP5C980mThA-hIlAexWMgn0yiXt3GXod1eAklyI8p1HdjCGyRpoEw-U.png)

Results for NVIDIA GH200 running HPCG benchmark. (Image credit: Phoronix)

5vFBmCRUcbfgb9PR7NpFoXpSeRSFqr47lr9V1YtOG-k2Aus1jPY1T217XD_EnBKKZKnn8MTFUs-Jx2lMYVR_8Ek4Q9eQcM-kml5l0C2B4wfzwdytunAxtEcGZNfi_pp-GQKy8jTPCrEpqRUtlrk8OXw

Another notable result came from the [NWChem](https://www.nwchem-sw.org/) benchmark, with the GH200 coming in second place at 1403.5 seconds.

  

![](/uploads/5vFBmCRUcbfgb9PR7NpFoXpSeRSFqr47lr9V1YtOG-k2Aus1jPY1T217XD_EnBKKZKnn8MTFUs-Jx2lMYVR_8Ek4Q9eQcM-kml5l0C2B4wfzwdytunAxtEcGZNfi_pp-GQKy8jTPCrEpqRUtlrk8OXw.png)

Results for NVIDIA GH200 running NWChem benchmark. (Image credit: Phoronix)

  

On a geometric mean basis across all the benchmarks, the GH200 Grace CPU performance was perfectly respectable overall. 

  
![](/uploads/O_VgErmTsszfyrnSV20nbY-BdZb4QvnVJe7LOp55l33JKKI3JFoQSRpWU2lia3g_QlqKTwi4_1VXjUQ2gE1iHKNYy_A4JkgtMsPnn5uzc7TBf9JjVzH1rifiNFA_n17d4TI_t7MB6bsjIMCMYGMcrXo.png)

(Image credit: Phoronix)

  

A set of GPU benchmarks run by researcher Simon Butcher [compared](https://blog.hpc.qmul.ac.uk/benchmarking-grace-hopper-nodes/) performance of the PyTorch ResNet50 [training recipes](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnet50v1.5) published by NVIDIA. The ImageNet 2012 dataset (150GB) was used, and training was run for 90 epochs, or about an hour. The GH200 showed excellent performance here.

  

![](/uploads/qlwGIbeeRNXXquUTSoF6EDJmoss443fwqWUEQg9PtgAVPwIHTNTbi0U5BsEFx3Wx2dX10b0gfTdlgMfQxU_kHulwTU5QaOGlz7Oy9U8GBpdkvj4pCE7wjW7PJzfU_lTK-mdd3VUlbSjUaXEx2RVtloA.png)

(Image credit: Simon Butcher, Queen Mary University of London)

  

NVIDIA also [published](https://resources.nvidia.com/en-us-dgx-gh200/nvidia-grace-hopper-superchip-datasheet) some performance comparisons that may be of interest.

  

![](/uploads/dXjLeW3nCbOSpjMxABHSZ6LP84CLuzszoNLvi2m9CD1Dxf9GgTYk17wr2YlxWxqNM5xF3U6GXV-E6JFILwTqmj4zEmdd9Rv5MNAUNK9xaa6YeTfM9krmcLeMRaUQ4eF8qt_G0-xW81JoRiaEz4157xQ.png)

(Image credit: NVIDIA)

  

### Final Thoughts

  

The NVIDIA GH200 Grace Hopper Superchip delivers the performance required for truly giant-scale AI and HPC applications running terabytes of data. Whether you're a scientist, an engineer, or a massive data center, you'll find it up to the task. 

As for what the future holds? Nvidia [recently announced](https://www.tomshardware.com/pc-components/gpus/nvidias-next-gen-ai-gpu-revealed-blackwell-b200-gpu-delivers-up-to-20-petaflops-of-compute-and-massive-improvements-over-hopper-h100) the GH200's successor: Grace Blackwell B200, the next-generation data center and AI GPU.

  

Here at Vast.ai, we're proud to say that we recently added ARM64 and GH200 support to our lineup. On our GPU rental platform, just about anyone can get access to top-tier GPUs at affordable rates for a variety of computational needs. You can skip the cost-prohibitive investment in hardware and get right to work today – we'd love to help you discover the possibilities!

NVIDIA's highly anticipated next-generation GH200 Grace Hopper Superchip platform is finally available. Perfectly suited for the era of accelerated computing and generative AI, it's essentially the world's most powerful GPU paired with the most versatile CPU.

Examining the NVIDIA GH200


We're pleased to reveal the latest round of updates here at Vast.ai! Our mission is to provide our customers with the best possible GPU rental platform, and we believe that every update is an opportunity to make our product shine that much more. For instance, we've recently added ARM64 and GH200 support – so we're now able to support ARM64 platforms including the Grace Hopper superchip!

But before we delve into some of our bigger updates below, first up on the agenda are bug fixes. We strive to address every minor hiccup and bug as quickly as possible, so that you can enjoy a seamless, uninterrupted experience on Vast. Here are a few of our latest bug fixes:

- Linking to docs now aligns correctly
- Random logouts from the webUI have been fixed
- Users don't see the host setup command unless they are in a hosting account
- Fixed an off-by-1 error with the CLI command ssh-url
- Everywhere a template name is mentioned, it is now linkable with a hash ID
- An issue with number of referrals not displaying has been corrected
- Fixed issue with template editor in search interface

![](/uploads/feb24Image.png)

### NVIDIA Cloud GPU Updates

Over the past few weeks, we've made some important enhancements to our platform. It's our hope that these updates will help save you time and effort, as well as maximize your goals and results on Vast. The following are some highlights of the latest changes:

- As mentioned, we've added ARM64 and GH200 support; we now support ARM64 platforms including the Grace Hopper superchip
- Added a template for NVIDIA RAPIDS - [https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593](https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593)
- Changed Autoscaler CLI commands to work with template linking
- Added a search feature for templates and hover help (tooltip) to the template page
- Implemented flexible date format for CLI so it isn't just Unix epoch time
- Improved search API for performance
- Added a DLPerf filter

Please check out the full change log for our February 2024 product updates at the bottom of this page. Getting to know the latest capabilities will help you make the most of your time on Vast!

If you're looking for affordable, on-demand cloud GPU rental, you've come to the right place – and if you need some assistance getting started, we have the experience and know-how to guide you through whatever your project requires. Join our thriving Discord community for support or reach out to our customer service team to get all your questions answered! We would love to help you explore the possibilities that Vast holds for you today. Please feel free to contact us through the chatbox at the bottom right or drop us a line at [support@vast.ai](mailto:support@vast.ai) whenever you'd like!

### Change Log

- Added ARM64 and GH200 support; we now support ARM64 platforms including the Grace Hopper superchip
- Added a CLI parameter on create instance for a template hash; i.e. vastai create instance --template_hash HASH
- WebGUI uses cancel_unavail by default (instances are not created stopped)
- Added a template for NVIDIA RAPIDS - [https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593](https://cloud.vast.ai/?template_id=ce0f0b38b72857851f17c71492870593)
- Upgraded our SD guides with ai-dock template and info on setting a strong password
- Improved Autoscaler to use template info more completely
- Changed Autoscaler CLI commands to work with template linking
- Bugfix: args launch now doesn't accept an onstart script
- Bugfix: linking to docs now aligns correctly
- Bugfix: random logouts from the webUI fixed
- Bugfix: users don't see host setup command unless they are in a hosting account
- Bugfix: fixed an issue with number of referrals not displaying
- Bugfix: fixed host link to Stripe connect earnings dashboard
- Bugfix: on instance to instance copy, copy to /root is fixed
- Bugfix: fixed issue with template editor in search interface
- Added a search feature for templates
- Added a "Create new template" feature on templates
- Improved search API for performance
- Added the team name and member name for the API/team contexts on the account page
- Added hover help to the template page
- Added DLPerf filter
- Added flexible date format for CLI so it isn't just Unix epoch time
- Bugfix: everywhere a template name is mentioned, it is now linkable with a hash ID
- Bugfix: loading bar was showing before transfer commands, it is now fixed
- Bugfix: template tag dropdown selector is fixed, as it wasn't updating correctly
- Bugfix: fixed an off-by-1 error with the CLI command ssh-url


February 2024 Product Update


The rise of GPU-centric tasks in Python environments means a balanced computing setup is as important as ever. While a powerful GPU handles parallel processing tasks efficiently, a good CPU is necessary to prevent bottlenecks and ensure smooth performance. A common misconception is that a strong GPU can make up for a weak CPU, but this is not the case. A weak CPU can stall, creating delays that lower performance, regardless of the GPU's strength.

The real issue lies in the CPU's role in managing a program’s workflow, including memory management, I/O operations, and providing instructions to the GPU. When the CPU lags, it can't feed instructions to the GPU quickly enough, causing bottlenecks. This problem is especially true in Python environments, whose language limitations can worsen the bottleneck.

# Why Does a Weak CPU Lead to Bottleneck?

In computing, the CPU and GPU work together. The CPU directs operations and sends instructions to the GPU for processing. However, when a CPU is weak or slow, it can't keep up with the GPU, causing a bottleneck. This means the GPU has to wait for the CPU to send instructions, which slows down the entire process.

Python environments can exacerbate this issue due to certain limitations. For instance, Python's Global Interpreter Lock (GIL) can further slow down the CPU, making the bottleneck issue more noticeable. The efficiency of running Python tasks largely depends on the CPU's capabilities. A stronger CPU can send instructions to the GPU faster, keeping the process running smoothly and ensuring better performance.

# Introducing AMD’s Threadripper: A More Powerful CPU

The [AMD Ryzen Threadripper 7000 Series and PRO 7000 WX-Series](https://www.amd.com/en/newsroom/press-releases/2023-10-19-amd-introduces-new-amd-ryzen-threadripper-7000-ser.html), built on the advanced "Zen 4" architecture, offer high core and thread counts. This design significantly alleviates CPU bottlenecks, especially in Python environments where the CPU often stalls due to the language's inefficiencies.

Threadripper's superior multi-core performance allows for more efficient instruction management, reducing the wait time for the GPU. This results in significantly better Python execution times and smoother operations across various computational scenarios.

# Bridging the Gap with Vast.ai

Many individuals lacking access to high-performance CPUs like Threadripper may experience bottlenecks in their GPU-accelerated tasks. The inefficiencies of a weaker CPU can significantly slow down processing, making certain computational tasks a slog.

[Vast.ai](https://vast.ai/) presents a viable solution to this problem. It provides a platform where users can rent high-performance GPU resources, which can be paired with superior CPU capabilities. By offering a gateway to better hardware resources, Vast.ai helps in overcoming the limitations imposed by less capable CPUs. Users can thus experience smoother, more efficient processing, enabling them to execute their tasks more effectively and in a timely manner. This way, Vast.ai plays a crucial role in alleviating the hardware constraints that many individuals face, bridging the gap between their current setup and the ideal computational environment.

# Final Thoughts

A robust CPU is crucial for maximizing GPU utilization, particularly in Python environments where a weaker CPU can cause bottlenecks. [The AMD Ryzen Threadripper](https://www.amd.com/en/processors/ryzen-threadripper-pro) significantly alleviates this issue with its high core and thread counts. For those without access to such high-performance CPUs, [platforms like Vast.ai](https://vast.ai/) offer the opportunity to rent superior GPU and CPU resources, providing a practical solution to hardware limitations. Exploring these options can lead to better performance in GPU-centric tasks and a smoother computing experience overall.


Introducing AMD’s Threadripper: A More Powerful CPU

Blog

Running Oobabooga in API Mode on Vast.ai

Introduction to Oobabooga

Why Run Oobabooga in API Mode vs Standard Mode?

Why Choose Vast.ai for Oobabooga Deployment?

Launching Oobabooga on Vast.ai: A Step-by-Step Guide

Logging In to Vast.ai:

Selecting Oobabooga Template:

Allocating Disk Storage:

Choosing a Suitable GPU:

Launching Oobabooga Instance:

Final Thoughts

Continue Reading: