
We're pleased to unveil the latest updates here at Vast.ai, geared toward achieving the best possible product experience for new and existing customers alike! Our aim is to provide you with a GPU rental platform that offers best-in-class service at a fraction of the price.

To that end, we work tirelessly to stay on top of the game when it comes to delivering improvements and enhancements. Bug fixes are always first on the agenda so we can ensure our customers enjoy a seamless experience at all times. Here are some of the latest fixes we've implemented:

- Refreshing now no longer deselects the selected template
- Fixed issue with setting internet pricing on host machine page
- Billing page now correctly displays no charges for offline instances
- GPU filters now working for all GPU types; added L40 and A6000ada
- Fixed incorrect invoice charges summation
- Scroll issues on instance page fixed
- Client earnings page now shows payouts for referrals

#### NVIDIA Cloud GPU Updates

Updating our product isn't just about bug fixes – we also extend our focus to implementing meaningful enhancements in a variety of ways. We want all of our customers to be able to make the most of their time on Vast! Below are some highlights of the latest changes:

- Documents and guides for Autoscaler (in beta) are now available – https://vast.ai/docs/autoscaler/introduction

- Contact us (on website chat or via email at contact@vast.ai) if you want to try out the Autoscaler for automating the provisioning of GPU workers for dynamic workloads.

- Added support for Oobabooga and Stable Diffusion for the Autoscaler beta
- Added warning to user for contract expiration
- Added example CLI autoscale script
- Added in PDF and CSV billing reports for clients with a date range picker
- Added a clean modal for hosts to manage machines in the GUI with a beta flag

We've also recently updated our Terms of Service, which can be read [here](https://vast.ai/terms). The full change log for our November 2023 product updates is included at the bottom of this page for your convenience – there are plenty of additional improvements we've made in recent weeks that you can check out below.

If you're new to Vast and looking for a cost-effective cloud GPU rental solution for any number of potential use cases (a few of which we outlined in a [recent blog post](https://vast.ai/article/GPU-Rental-Use-Cases)) we invite you to discover what our platform can do for you! Please reach out at any time via the chatbox at the bottom right or email us at [support@vast.ai](mailto:support@vast.ai) with any questions – we're happy to help!

#### Change Log

- Templates can now specify recommended disk size
- Added docs for Autoscaler (in beta) - https://vast.ai/docs/autoscaler/introduction

- Contact us (on website chat or email at contact@vast.ai) if you want to try out the Autoscaler for automating the provisioning of GPU workers for dynamic workloads.

- Added links to the txn for Crypto payments in billing
- Added confirmation on reboot
- Copy modal now has a pop-up with indication of success or failure
- For reserved instances, a pop-up now appears if there's not enough credit
- Updated our hosting landing page with info on how to become a datacenter partner ([https://vast.ai/hosting](https://vast.ai/hosting))
- Bugfix: issues with de-verification email to hosts
- Bugfix: issue with setting internet pricing on host machine page
- Bugfix: refreshing now no longer deselects the selected template
- Bugfix: payment information on billing page fixed
- Bugfix: fixes for templates with filters, such as GLX desktop
- Bugfix: SSH proxy port was wrong, now fixed
- Added in equivalent CLI command in each template
- /Host/setup is now the only place an account can be converted to a host account
- Added better code snippet text in docs
- Reworked the email notification system for when balance is zero
- Added warning to user for contract expiration
- Added example CLI autoscale script
- Bugfix: args for templates now filter machines properly
- Bugfix: autoscroll issues fixed
- Added in PDF and CSV billing reports for clients with a date range picker
- Added in multi-region support in the geo filter
- Added more clear arrows for the cloud copy UI to make it clear you can swap directions
- Added a warning pop-up for when you set your autobilling threshold above 0 (which could result in an immediate charge)
- Fixed the client referral earnings page to be more clear
- Instance deposits for reserved discounts now show instance ID in billing page
- Add or change invoices CLI/API endpoint to get charge history for a set of instances with contract-based filters (like instance tag for autoscaling groups)
- Bugfix: billing page now correctly displays no charges for offline instances
- Bugfix: fixed overflow/spacing issues on mobile browsers on the homepage
- Bugfix: GPU filters now working for all GPU types; added L40 and A6000ada
- Bugfix: fixed issue with payouts appearing on client billing pages
- Bugfix: invoices now use start date and end date properly
- Added support for Oobabooga and Stable Diffusion for the Autoscaler beta
- Added additional documentation for the Autoscaler
- Added in an easy switch link to the login & register modal
- Added feedback notifications and spinners on PDF exports for invoices
- Bugfix: fixed incorrect invoice charges summation
- Bugfix: fixed scroll issues on instance page
- Bugfix: date range is now properly set for hosts on the invoice/earnings page
- Bugfix: full invoice address now in the PDF invoices
- We have updated our Terms of Service ([https://vast.ai/terms](https://vast.ai/terms))
- Added a clean modal for hosts to manage machines in the GUI with a beta flag
- Added a South America filter
- Added a verification email re-sent button on account page
- Added additional documentation and guides for the Autoscaler beta
- Template name and image are now displayed when available in the UI
- You can now override the entrypoint for args launch instances using the option --onstart-cmd (or aliased as --entrypoint). For example: vastai create instance 7277200 --image pytorch/pytorch --disk 40 --onstart-cmd 'bash' --args -c 'echo hello; sleep infinity;'
- Bugfix: client earnings page now shows payouts for referrals
- Bugfix: frontend GUI now enforcing 8GB minimum disk size
- Bugfix: added additional detections and deletions for instances that failed to load


November product updates


Vast.ai provides a platform where developers can access a wide selection of GPUs from various hosts and data centers, offering competitive pricing to fit any budget. The cost and availability of GPUs are critical to the success and speed of generative AI projects. We’re excited to announce that [dstack](https://dstack.ai/) now includes integration with our [GPU marketplace](https://vast.ai/). dstack is an open-source cloud platform for training and deploying gen AI models. This integration enables our users to access an extensive selection of affordable GPUs, breaking down the barriers to entry for developers and ensuring that high costs do not stifle innovation.

# Diverse GPU Options at Your Fingertips

Vast.ai’s marketplace stands out by offering a wide selection of GPUs, ensuring that developers can select the best fit for their projects without cost constraints. The platform recommends GPU instances based on a comprehensive quality score but also gives you the flexibility to set a maximum price, ensuring that you stay within your budget.

# Empowering Developers with dstack and Vast.ai

With dstack 0.12.3, accessing Vast.ai's marketplace becomes a seamless part of the development workflow. Now, dstack’s intuitive CLI and API are equipped to tap into Vast.ai's resources, allowing you to train and deploy generative AI models with greater ease and less financial strain. This integration streamlines your development process, enabling you to utilize the cost benefits of Vast.ai without the hassle of complex cloud infrastructure management.

![](/uploads/dstack-top.png)

# Effortless Setup and Deployment

Using Vast.ai with dstack is easy and straightforward:

1. Log into your Vast.ai account and navigate to the 'Account' section to find your API Key.
2. With your API Key, configure the dstack backend according to the provided steps.

![](/uploads/dstack-bottom.png)

3. Once configured, begin using the robust features of dstack’s CLI and API.

This integration not only facilitates [fine-tuning](https://dstack.ai/docs/guides/fine-tuning/) and [deployment](https://dstack.ai/docs/guides/text-generation/) of models but also the creation of [development environments](https://dstack.ai/docs/guides/dev-environments/) that support interactive coding, making it a perfect fit for developers who prefer to work within their favorite IDEs.

# Seamless Switching Between Cloud Providers

The flexibility that dstack offers is unparalleled. You can now effortlessly switch to Vast.ai when it suits your project needs while retaining the option to use other cloud providers as required. This ensures that your workflow is as efficient and cost-effective as possible.

# Streamline Your AI Projects with dstack and Vast.ai

If you're looking for a simple and cost-effective way to train and deploy generative AI models, the combination of dstack and Vast.ai is ready for you. It's designed to save you time and money. With dstack’s new preview versions of the [fine-tuning](https://dstack.ai/docs/guides/fine-tuning/) and [text generation](https://dstack.ai/docs/guides/text-generation/) APIs, your development process just got even better.

Need a hand? Our [community](https://discord.com/invite/5j3ZDpHeBB) as well as dstack’s [community on Discord](https://discord.gg/u8SmfwPpMd) is available to offer support and answer your questions. We're committed to helping you make the most of what Vast.ai and dstack have to offer.


We’re excited to announce that dstack now includes integration with our GPU marketplace.

Accessing the GPU marketplace with Vast.ai and dstack


# Understanding the GPU Supply Crunch

Graphics Processing Units (GPUs) have evolved from their original use in video game graphics to become essential in fields like artificial intelligence, data processing, and advanced computing. Their ability to efficiently handle complex calculations has made them a key component in today's technology landscape. Despite their importance, there's a significant problem: a shortage of GPUs, known as the GPU supply crunch. This shortage is affecting many industries, leading to delays in projects and rising costs. In this article, we'll explore the reasons behind this shortage, which include problems in global supply chains, production challenges, and a sudden increase in demand for GPUs. We'll also look at its impact and how [services like Vast.ai](https://vast.ai/), a GPU rental platform, are providing solutions during this challenging time.

# What Caused The GPU Supply Crunch?

The cause of the GPU supply crunch can be attributed to several key factors. A major driver is the increased demand for GPUs, especially due to the rise of generative AI technologies in 2023. As AI becomes more integral to sectors like healthcare, automotive, finance, and entertainment, the competition for these powerful GPUs has intensified. Additionally, the COVID-19 pandemic caused significant disruptions in manufacturing and supply chains, leading to factory closures and production slowdowns. This situation was worsened by [shortages of crucial components like CoWoS packaging](https://asia.nikkei.com/Business/Tech/Semiconductors/TSMC-sees-AI-chip-output-constraints-lasting-1.5-years), which is essential for integrating memory and chip components.

Furthermore, geopolitical factors, such as the [U.S. restrictions on Nvidia's GPU shipments to China](https://www.trendforce.com/presscenter/news/20231019-11891.html), have impacted manufacturing capacities and market dynamics. In response, companies like Nvidia are adapting by increasing the availability of GPUs through cloud-based rental services, marking a shift in how these resources are distributed and used.

# Consequences of the Crunch

The GPU supply crunch has significantly affected various sectors, with the professional computing and AI research areas facing notable challenges. The growing demand for GPUs, particularly for AI applications like training models such as ChatGPT, has been remarkable. For instance, the [ChatGPT model was initially trained on 10,000 NVIDIA GPUs, but current demands have risen to about 25,000 NVIDIA GPUs](https://www.techgenyz.com/2023/02/13/nvidias-gpus-to-face-shortage-supply/). This increase has put a strain on GPU supplies, impacting not only AI research but also professional computing sectors. As generative AI continues to evolve, there's a possibility that companies like Nvidia might focus more on AI applications, potentially affecting GPU supply and pricing across other sectors.

The crunch has resulted in increased GPU prices and limited availability, affecting both consumers and businesses. They face challenges in affording and acquiring the needed hardware. Although the situation is slowly improving after the COVID-19 pandemic, the market is still not back to its pre-pandemic levels, and prices are higher than previously

# Industry Responses

The industry has been actively addressing the GPU supply crunch with several initiatives. Major players, such as Nvidia, are significantly increasing their production capacities. For example, [Nvidia plans to double its CoWoS packaging capacity by the end of 2024 with a $2.9 billion investment in Taiwan](https://www.irrgame.com/news/nvidia-gpu-shortage/), a step aimed at easing the GPU shortage and potentially lowering prices. Other companies like TSMC, Intel, and Samsung are also focusing on improving chip packaging, a key bottleneck in production. Intel, for instance, plans to quadruple its capacity by 2025. These measures demonstrate the industry's dedication to tackling the current production issues and meeting the growing demand, especially in AI and professional computing sectors.

# Government and Regulatory Responses 

On the government and regulatory side, there's been a notable effort to boost domestic chip manufacturing. The U.S. Commerce Secretary, Gina Raimondo, [has been vocal in urging Congress to act and provide funding for this initiative](https://www.industryweek.com/supply-chain/planning-forecasting/article/21243009/global-chip-shortage-likely-to-last-through-2023-us-official). Key legislative initiatives include the CHIPS Act and the America COMPETES Act, both proposing an investment of $52 billion in domestic chip research and manufacturing. While the final agreements on these bills are still pending, discussions among top chipmakers and government officials suggest that substantial relief from the shortage may not be seen until 2024. This highlights the necessity for global collaboration and long-term planning in the semiconductor industry.

# Vast.ai: Addressing the GPU Market Needs

During the current GPU supply crunch, [Vast.ai has become a key player in the GPU rental market](https://vast.ai/). This platform connects those who need GPUs for tasks such as AI training, data analysis, or graphics rendering with those who have spare computational power to rent. This model provides an affordable alternative to buying expensive hardware, particularly useful when prices are high and availability is low.

Vast.ai's system enables GPU owners to offer their processing power for rent, making high-end computing resources more accessible and affordable. This is particularly beneficial during the supply crunch, as it eases the burden on companies and individuals who are struggling to find available GPUs or who cannot afford their high costs.

The advantages of Vast.ai's rental model are evident in the current situation. It's a cost-effective option for those requiring GPUs for short-term projects or for those who can't meet the high initial costs of purchasing. The platform also offers flexibility, allowing users to adjust their computing resources according to their needs without the need for a long-term commitment or large investment. Vast.ai won’t just be useful during the supply crunch, though; it's a forward-thinking approach to managing the allocation and accessibility of resources in the increasingly demanding world of advanced computing.

# Wrapping Up: Navigating the GPU Supply Crunch

The GPU supply crunch presents a multifaceted challenge, but it's met with dynamic solutions and strategies. From the surge in demand for GPUs driven by the rise of generative AI to the disruptions caused by the COVID-19 pandemic and geopolitical factors, this shortage has far-reaching implications. However, the tech industry's response, highlighted by significant investments from companies like Nvidia, TSMC, and Intel, shows a strong commitment to overcoming these obstacles. Similarly, government initiatives like the CHIPS Act indicate a strategic approach to revitalizing domestic chip manufacturing.

[Vast.ai's emergence as a resourceful player in the GPU rental space](https://vast.ai/) illustrates an innovative response to the crunch, offering accessible and flexible computing power to those in need. This model not only addresses the immediate challenge but also sets a precedent for future resource distribution in the world of advanced computing.

As we move forward, these collective efforts from industry leaders, governments, and innovative platforms like Vast.ai will be instrumental in navigating the current crisis. They pave the way for a more resilient and efficient tech industry, capable of adapting to the growing demands and challenges of the future.


Despite their importance, there's a significant problem: a shortage of GPUs, known as the GPU supply crunch.

Quieting The Chip Crunch: Understanding the GPU Supply Shortage


# PyTorch vs TensorFlow: Which One Is Right For You? 

[PyTorch](https://pytorch.org/) and [TensorFlow](https://www.tensorflow.org/) are two of the most widely used deep learning libraries in the field of artificial intelligence. Both are supported on Vast.ai with easy to use templates. PyTorch, initially developed by Meta, offers an intuitive approach to building neural networks and is favored for its flexibility and ease of use in research. On the other hand, TensorFlow, developed by Google Brain, is designed for large-scale and complex machine learning models, with strong support for production environments. This article will compare PyTorch and TensorFlow, helping you understand which library may be more suitable for your project's needs and computational requirements.

# PyTorch Overview

PyTorch, initially developed by Meta, has quickly become a top choice for deep learning enthusiasts and professionals thanks to its dynamic computational graphing and deep integration with the Python language. It's specifically designed to facilitate rapid development and iteration of deep learning models, resonating with the workflow of many developers.

The true power of PyTorch comes when combined with NVIDIA's CUDA, a parallel computing platform that allows dramatic increases in computing performance by harnessing the power of GPUs. With CUDA, PyTorch can perform data-intensive computations at a fraction of the time it would take on traditional CPUs. This capability is very important when training complex models or dealing with massive datasets, a common task in AI research and development.

PyTorch also supports production needs with tools like TorchScript for model exporting and TorchServe for deployment. Its ecosystem is full of specialized libraries, fostering a collaborative environment where developers share tools and pre-built models, enhancing the PyTorch experience for both research and industry use.

# TensorFlow Overview

TensorFlow, developed by the Google Brain team, has also become one of the most widely used frameworks in the industry due to its scalable and flexible architecture. Its architecture centers around a static computational graph that simplifies and accelerates data processing. This feature, combined with its capacity for distributed computing, makes TensorFlow ideal for large-scale projects. In addition, transitioning models from the lab to the field is seamless with TensorFlow, thanks to tools like TensorFlow Serving for servers and TensorFlow Lite for mobile devices.

Furthermore, TensorFlow's integration with TensorFlow Extended (TFX) eases the creation of end-to-end machine learning workflows, making it more suitable for production use. The platform also includes comprehensive tools such as TensorBoard, which simplifies model analysis and debugging with its visualization capabilities.

The framework has fostered a massive community ranging from academic researchers to industry professionals. This diverse user base benefits from a large source of educational materials, add-ons from third parties, and forums for troubleshooting. Its widespread use in the corporate sector attests to TensorFlow's dependability and advanced capabilities. Ongoing input from this community contributes to TensorFlow's growth, keeping it at the forefront of AI application development.

# Comparing PyTorch and TensorFlow Metrics

### Performance Comparison

Both PyTorch and TensorFlow offer fast performance, but they do come with their own set of advantages and disadvantages. [A benchmark comparison revealed](https://viso.ai/deep-learning/pytorch-vs-tensorflow/) that PyTorch had a better performance compared to TensorFlow, particularly when offloading most of the computation to the cuDNN and cuBLAS libraries, which are essential components for GPU-accelerated computing​. However, performance can vary depending on the specific use case and hardware, with both frameworks capable of achieving high performance when properly optimized​​.

### Training Time and Memory Usage

[In a direct comparison](https://viso.ai/deep-learning/pytorch-vs-tensorflow/) utilizing CUDA, PyTorch outperforms TensorFlow in training speed, completing tasks in an average of 7.67 seconds against TensorFlow's 11.19 seconds. However, TensorFlow is more memory-efficient, using 1.7 GB of RAM during training compared to PyTorch’s 3.5 GB. Therefore, for quicker training, PyTorch is favorable, but for lower memory usage, TensorFlow is the better choice​.

### Ease of Use

PyTorch is often favored for its intuitive, Pythonic interface, making it easier for rapid prototyping, especially when utilizing CUDA for GPU acceleration. On the other hand, TensorFlow, although having a steeper learning curve, offers a more structured environment suitable for large-scale or commercial projects. Over time, TensorFlow has improved its usability, making it more approachable, but PyTorch is still considered more usable. In the end, PyTorch is simpler for research and development, while TensorFlow is structured for larger, production-oriented projects. However, regardless of which framework fits your project’s requirements, the computational requirements remain a substantial factor that can influence your choice.

# Addressing Computational Demands with Vast.ai

The computational intensity of working with PyTorch or TensorFlow can present challenges, especially when processing large datasets or training complex models. Both frameworks perform much better with high-performance GPUs, but they are not always readily accessible or economically feasible for every developer. [Vast.ai](https://vast.ai/) steps in to fill this gap by offering affordable and scalable GPU resources. With Vast.ai, developers and researchers can expedite training times and scale computing resources in tandem with their project demands, effectively leveraging the strengths of either framework without hardware limitations.

# Conclusion

In the end, your choice between PyTorch and TensorFlow should align with your project requirements: PyTorch for its user-friendly nature in research and development, and TensorFlow for its robustness in large-scale, production-level projects. Both frameworks have their merits and can be significantly enhanced by overcoming hardware limitations, a service that [Vast.ai](http://vast.ai/pricing) provides effectively. With the right tools and resources, you can ensure that your machine learning projects are not only innovative but also computationally feasible.


PyTorch vs TensorFlow: Which One Is Right For You? 

PyTorch vs TensorFlow: Which One Is Right For You


## GPU Rental: A Closer Look at Use Cases

Over the past few years, cloud GPU rental has emerged as a game-changer for innovators everywhere who want to power next-generation capabilities for their projects in various industries – without the hassle and big price tag associated with maintaining their own hardware.

The use cases for GPU rental are so diverse and wide-ranging that many people aren't aware of just how much of a difference it can make in both their personal and professional lives. With Vast.ai, anyone can access high-end computing power, on demand, while transforming their project's performance, scalability, and affordability.

Let's take a look at some specific use cases to explore the benefits and advantages of renting GPU resources on the cloud.

#### In-Depth Look: Podcast Transcription

A great example is **podcast transcription**. GPU rental through a platform like Vast.ai can power large language models that are able to perform transcription services with incredible speed and accuracy. Trained on massive datasets, these models can handle different accents, complex terminology, and industry-specific jargon, all while sifting through background noise.

[Whisper ASR Webservice](https://github.com/ahmetoner/whisper-asr-webservice/) is a simple to use transcription service that can harness the power of Vast GPUs. Simply select the template for Whisper by clicking this [link](https://cloud.vast.ai/?ref_id=43484&template_id=e795f6239ba0236393d61d892c3f4152), select a GPU and launch the service. It will then be accessible from your web browser and utilize the GPU on Vast for video/audio transcription to multiple languages.

Not only that, but GPU-accelerated transcription services can also provide near-instantaneous speech-to-text conversion for live broadcasts, enabling real-time closed captions or transcriptions for viewers.

Renting GPUs only when needed optimizes expenses and ensures you pay for computing resources only during active transcription tasks. And if you need to transcribe a high volume of podcasts at once (e.g. if you're working through a backlog or you're providing transcription as a service), GPU rental allows you to scale your operations as needed. You can flexibly increase or decrease GPU resources at any time.

Any podcaster, content creator, business owner, or even college student needing transcription assistance can take advantage of the speed, accuracy, real-time capability, cost-efficiency, and scalability that GPU rental offers, whenever and wherever they need it.

But there's so much more.

#### Other GPU Rental Use Cases

Renting cloud GPUs gives you an incredibly powerful workstation without the overhead and maintenance that normally go along with it. This frees you up to focus on accelerating diverse workloads, from the mundane to the most mission-critical.

Below are just a few of the possibilities.

**Translation services**: Similar to transcription, GPU-accelerated translation rapidly processes text and even speech via deep learning and neural networks, delivering speed and accuracy for high-quality multilingual communication. Real-time translation is also possible for applications like live broadcasts and video conferencing.

**Video rendering and editing**: Content creators can quickly render high-definition videos and easily edit multimedia projects using the immense processing power of a rented GPU server. The same is true for **3D rendering and animation**, speeding up a process that might otherwise take many hours.

**Writing and storytelling**: Seasoned authors and novice writers alike can make use of AI models to spark creativity in their own writing or to help craft novels, articles, text-based adventure games, and dynamic and immersive storytelling experiences. Right here on Vast.ai, we offer access to [KoboldAI](https://vast.ai/article/exploring-ai-based-writing) – an AI-powered writing assistant – via a ready-to-use template suitable for even the most non-tech-savvy user.

**Automated customer service**: From text-based chatbots and virtual assistants to voice-driven customer service through automated phone systems, GPU rental can provide the computational power needed for cutting-edge natural language processing. Images and videos attached to customer inquiries can also be analyzed and identified, improving the ability to provide relevant responses.

**Virtual desktop infrastructure**: Businesses that are embracing remote work can employ GPU rental to provide employees with high-performance virtual workstations, enabling dynamic collaboration from anywhere in the world.

**AI-generated artwork**: Whatever your artistic needs may be, deep learning AI tools like Stable Diffusion – based on latent diffusion processes – bring image generation to your fingertips with the press of a button. Frameworks like [ComfyUI](https://vast.ai/article/getting-started-with-comfy-UI), available as a template right here on Vast, allow for unprecedented flexibility and speed in AI-powered art, whether through text-to-image generation, image-to-image translation, or image interpolation.

#### The Bottom Line

The above-mentioned use cases are just the beginning. From **high-performance computing** and **AI model training** to **scientific simulations** and **advanced medical imaging**, the solution to a wide range of complex computing and data-intensive tasks is just a few clicks away! Renting the GPU resources you need, with built-in scalability, minimizes OpEx and maximizes ROI.

As the market leader in low-cost cloud GPU rental, we invite you to explore what Vast.ai can offer you and your organization today. We would be pleased to help you discover the powerful and affordable GPU rental solution you've been waiting for.


use cases to explore the benefits and advantages of renting GPU resources on the cloud

GPU Rental: A Closer Look at Use Cases


## Introduction to Oobabooga

Oobabooga is a tool designed for interacting with Large Language Models (LLMs), capable of understanding and generating text. With its user-friendly web interface, it's perfect for non-technical users interested in harnessing the power of LLMs. [Oobabooga supports a wide range of models](https://vast.ai/article/deploying-oobabooga-for-running-large-language-models), excelling in various use-cases like prompting, loading, and managing models.

### Why Run Oobabooga in API Mode vs Standard Mode?

Choosing API Mode for Oobabooga on Vast.ai offers several advantages, including enhanced scalability for multitasking, seamless tool integration via user-friendly endpoints, and streamlined workflow automation. Given these benefits, let's dive into how to actually run Oobabooga on Vast.ai.

### Why Choose Vast.ai for Oobabooga Deployment?

Running Oobabooga can be GPU-intensive, posing challenges for those without high-end GPUs or the financial resources to acquire them. That's where Vast.ai, a cloud-based GPU rental platform, comes in. By deploying Oobabooga on Vast.ai, you gain access to powerful, on-demand GPU resources and pay only for what you use, enabling complex text-generation tasks without hardware limitations.

### Launching Oobabooga on Vast.ai: A Step-by-Step Guide

#### Logging In to Vast.ai:

1. [Log into your Vast.ai account](https://vast.ai/).
2. Ensure you have enough credits for the task.

#### Selecting Oobabooga Template:

1. Click into the console.
2. Click ‘EDIT IMAGE & CONFIG.’

![](/uploads/Instance-Configuration.png)

3. Select the recommended template for Oobabooga.

![](/uploads/kO9lpdJwnxVSVjFnuprmaBQ3IJ6K2ALmORoSIyyWt3fG2gdPRIqrrFAVT8O6uCVqQwdTgPm6tRvIgbJFHmeaS8G3hIZxsKPBqC18fklDy62VIw8cf7OmhXiVkUcA.png)

4. Click on ‘EDIT’

![](/uploads/of-these-templates.png)

5. Scroll down to ‘On-start Script’ and type in ‘--api’ before the final semicolon

![](/uploads/vast.ai.png)

6. Click on ‘Select And Save’

![](/uploads/SELECT-AND-SAVE.png)

#### Allocating Disk Storage:

1. Adjust disk storage to a minimum of 80GB.

![](/uploads/EDIT-IMAGE-&-CONFIG.png)

#### Choosing a Suitable GPU:

1. [Review the Large Language Model (LLM) leaderboard on Hugging Face](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) to understand the GPU RAM requirements of your chosen model.
2. Filter for the total GPU RAM on the machine. Oobabooga can combine the GPU RAM on multi-GPU instances to load the model.

![](/uploads/GPU-Resources.png)

#### Launching Oobabooga Instance:

1. Click ‘Rent’.

![](/uploads/Auto-Sort.png)

2. Go to ‘Instances’

![](/uploads/cgNf2u4ui_EUkbfjH86kj1qltiHu7-CP32MKL-z5YTFjvc5fYPAPDnC06HXeuzG8R4YczJhTNVKkPB1LBqmxpt7g9nw_l9IpfYPFUaahDz3fYLMA3kQLw199j3Qt.png)

3. Click ‘Open’ on your newly created instance.

![](/uploads/entation.png)

## Final Thoughts

Oobabooga's user-friendly interface and versatile support for various LLMs make it an invaluable tool for text generation tasks. When considering deployment options, Vast.ai emerges as a cost-effective solution, providing access to powerful GPU resources on-demand. Following this guide, users can easily set up Oobabooga on Vast.ai, bypassing any hardware hurdles, and making using Oobabooga straightforward and simple.


Guide to deploying Oobabooga in API Mode on Vast.ai, leveraging powerful GPU resources and simplifying LLM interactions.

Running Oobabooga in API Mode on Vast.ai


It's that time again! We're pleased to announce the latest round of updates at Vast.ai. Our goal is to continually identify ways to improve, in order to elevate our service to new heights and bring you a product experience that is second to none.

An important aspect of our ongoing commitment to excellence is to fix bugs as quickly as we can. This ensures that our customers enjoy the best possible performance at all times. Some of our latest bug fixes include the following:

- Google Drive verification is complete and now works for all users!
- Reliability score has been added back to host machine page
- Made fixes to host machine page for setting price and date
- Reserved instances cannot be reserved for longer than they exist
- Earnings dashboards no longer display internal transfers
- Instance prepay items now show correctly in billing tab
- Interruptible bid pricing now includes storage cost and keeps the minimum in the make bid modal

### NVIDIA Cloud GPU Updates

At Vast.ai, we are committed to enhancing the user experience. Whether you're a current or future customer, you can expect continual improvements as we make adjustments that boost usability and performance. Below are some of the latest changes to help you make the most of our product:

- Templates now have a permalink and a version link
- Added NCCL test to host installer
  - For any hosts with AMD EPYC systems, that hardware has compatibility issues/problems with the core NVIDIA NCCL library that distributed PyTorch (and other frameworks) use, which may make some machines unusable. You can read more and find workarounds if you Google (or join our Discord server and search the history for EPYC), but here are a few links:
    - [PyTorch Discussion](https://discuss.pytorch.org/t/torch-distributed-init-process-group-hangs-with-4-gpus-with-backend-nccl-but-not-gloo/149061?u=ptrblck)
    - [NVIDIA NCCL Troubleshooting](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs)
- Added filters for max CUDA and driver version
- Added a "rented" option in the CLI so you can filter offers with GPUs overlapping your current stopped instances on the same machine
- Added filter for total system GPU RAM
- Added display of args for args launch instances
- Improved docs for copying and moving data

For full details, please check out the change log for our October 2023 product updates below. We've implemented a host of additional improvements in recent weeks that we believe our valued customers will appreciate.

If you'd like to learn more about the services we offer or explore how we can help you with your GPU rental needs, you are welcome to contact us at any time! Feel free to reach out via the chatbox at the bottom right, email us at [support@vast.ai](mailto:support@vast.ai), or join our Discord server today to enjoy a helpful community and a library of resources available at all times.

## Change Log

- Reformatted the stop instance modal for clarity
- Templates now have a permalink and a version link
- Complete status msg now displayed in expanded instance display
- Bugfix: fixed scroll jump after stop instance button pushed
- Bugfix: WebGUI Docker login validation now matches Daemon
- Bugfix: earnings dashboards no longer display internal transfers
- Bugfix: reserved instances cannot be reserved for longer than they exist
- Bugfix: host dashboard client end date no longer displays anything when there are no active contracts
- Added NCCL test to host installer
  - For any hosts with AMD EPYC systems, that hardware has compatibility issues/problems with the core NVIDIA NCCL library that distributed PyTorch (and other frameworks) use, which may make some machines unusable. You can read more and find workarounds if you Google (or join our Discord server and search the history for EPYC), but here are a few links:
    - [PyTorch Discussion](https://discuss.pytorch.org/t/torch-distributed-init-process-group-hangs-with-4-gpus-with-backend-nccl-but-not-gloo/149061?u=ptrblck)
    - [NVIDIA NCCL Troubleshooting](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs)
- Added filters for max CUDA and driver version
- Autobilling threshold now sets the low balance email threshold
- Updated email warning for all billing attempts
- All accounts are client accounts unless changed in the hosting directions
- Added a "rented" option in the CLI so you can filter offers with GPUs overlapping your current stopped instances on the same machine
- Bugfix: remove the Docker login/password after instance is created
- Bugfix: Coinbase payment improvements
- Bugfix: web UI direct port filter fixed
- Bugfix: added back reliability score to host machine page
- Bugfix: Jupyter button appears if JUPYTER_TOKEN or JUPYTER_PORT is defined
- Bugfix: client end date now handles no/null end date contracts
- Added filter for total system GPU RAM
- Added react code blocks in documentation
- SSH/Jupyter launch mode does not show entrypoint args
- Added feedback for password reset
- Added display of args for args launch instances
- Improved docs for copying and moving data
- Kaalia – upgraded disk IO testing to use Fio
- Bugfix: host machine page fixes for setting price and date
- Bugfix: interruptible bid pricing now includes storage cost and keeps the minimum in the make bid modal
- Bugfix: image CUDA version detection now properly detects the right CUDA version
- Bugfix: logging in refreshes templates
- Bugfix: Oobabooga template now opens the webUI upon clicking open
- Bugfix: instance prepay items now show correctly in billing tab
- Bugfix: email notifications for hosts fixes
- Bugfix: Firefox issue no longer blocking setting name in invoice
- Bugfix: pending crypto transactions no longer appear on pdf billing statements
- Google Drive verification is complete and now works for all users!


Discover Vast.ai's October updates: enhanced GPU features, vital bug fixes, and platform improvements for an optimal user experience.

October Product Update


Writing a book is a daunting task.

Coming up with just the right words, and keeping track of the ideas that you’re weaving together - this is difficult work. The difficulty can even make it seem that there might be better ways to be spending your time.

A new generation of tools seeks to solve this challenge though - to allow you to bring your powerful intuition and vision to life, while leaving the management of the details to the machines.

This may sound too good to be true - but it’s not. Try the tools out for yourself and see. We’ll help you get started.

Setting some context is important, though - so we’ll go quickly over some of the basics, and then show you step-by-step how to get started.

KoboldAI is a tool that provides functionality specifically related to writing. KoboldAI in turn can leverage any number of models while producing its assistance. The one that we will use here is called Pygmalion.

And finally the magic dust that brings all of this technical wizardry together is Vast.ai. We will install the KoboldAI tooling with the Pygmalion model on the powerful Vast.ai GPUs - which we can rent for pennies-per-hour. This not only gives us a cost-effective way to put these tools to use - it also allows us to only pay for what we need.

## Step 1: Create a Vast.ai Account

If you don't already have a Vast.ai account, you'll need to create one. Visit [Vast.ai](Vast.ai), sign up, and follow the on-screen instructions to create your account.

## Step 2: Add Credits

Before renting a virtual machine on Vast.ai, you'll need to add credits to your account. This is how you pay for the GPU resources you use. Simply click on your account balance and follow the prompts to add funds securely. Vast accepts credit card payments through Stripe and crypto through Crypto.com and Coinbase.

## Step 3: Running the Template

1. Browse Available Templates: Within the Vast.ai platform, take advantage of their pre-configured templates, which include the Pygmalion model. Visit the [Vast.ai Templates](https://cloud.vast.ai/templates/) section to explore these ready-to-use configurations.

2. Select the [KoboldAI + Pygmalion Template](https://cloud.vast.ai/?ref_id=43484&template_id=0930ce52c987ce85e981371fadacd7b1): Click that link, or locate and select the Pygmalion template from the list of available templates. These templates come pre-packaged with the necessary settings, eliminating the need for manual setup.
   [![ScreenShot](/uploads/1DCF197C-F94C-4EE0-9A60-DBD3ED8AB42B.jpeg)](/uploads/1DCF197C-F94C-4EE0-9A60-DBD3ED8AB42B.jpeg)

3. Customize as Needed: While the templates streamline the process, you still have the flexibility to customize certain parameters to suit your specific requirements, such as GPU type or resource allocation. Select your preferred storage size to find a machine for your needs.

4. Rent the Instance: Clicking the “Rent” button will start the instance.
   [![ScreenShot2](/uploads/B87F9218-AEA0-450C-BEFF-D5468A6C4212_4_5005_c.jpeg)](/uploads/B87F9218-AEA0-450C-BEFF-D5468A6C4212_4_5005_c.jpeg)

## Step 4: Write Your Next Great Novel!

Once your instance has started (this will take a few minutes), the final entries in the logs will show the URLs that can be used to access the system. They are labeled according to the type of the interface. Take one of these and plug it into your browser, and you can begin your work.

You will notice that one of these URLs is for use as an API - so if you are looking to programmatically include the capabilities of this tool into your own application, this is how you would do it.

### The Sky Is The Limit

This is just a small taste of the capability that might be provided by this and by similar tools. The flexible compute power offered by Vast.ai offers a cost-effective and straightforward way to get started - either with experimenting with workflows or with putting them to real use.


Blog