
# Vast.ai Product Update | September 2023

At Vast.ai, we understand how crucial it is to provide our customers with the top-notch performance that they’ve come to expect from our platform. To that end, we are constantly looking for ways to improve and bring you a service that continues to innovate.

When bugs pop up on occasion, we work quickly to address and rectify them, ensuring seamless performance moving forward. The following are some of the latest bug fixes we’ve made since our [previous product update](https://vast.ai/article/august-product-update):

- Hosts can no longer set a price below 0
- Template editor now updates correctly
- Broken template link has been fixed
- Issue with Backblaze data sync is fixed
- Backblaze copy now has an “operation complete” message

## NVIDIA Cloud GPU Updates

In our product updates, we also aim to deliver enhancements that make the user experience as smooth and uncomplicated as possible. Below are some of the latest adjustments we’ve made to boost usability and performance:

- Replaced the modal errors on the search page with snackbars (messages that temporarily show on the screen)
- Added a snackbar that says “copied” when someone copies a shareable link
- The instance extended view now shows actual status and intended status instead of current state and next state
- Template editor version tag will now auto-populate the docker image version
- Machine page client end date is no longer an editable box

You can check out the full change log for our September 2023 product updates below. It is our hope that these latest changes will continue to advance our goal of making Vast.ai as user-friendly as possible for our current and future customers.

If you’re interested in learning more about how we can assist you with your GPU computing needs, or if you need some help with deployment on our cost-effective infrastructure, please feel free to contact us anytime. There is a chatbox at the bottom right for your convenience, and you can also reach us via email at [support@vast.ai](mailto:support@vast.ai), or join our Discord server for instant access to a wealth of information and resources.

## Change Log

- Replaced the modal errors on search page with snackbars
- Added a snackbar that says "copied" when someone copies a shareable link
- Report machine icon is now a wrench and the warning (!) icon is used for scheduled maintenance
- The instance extended view shows intended status and actual status vs. current state and next state
- Template editor version tag will auto-populate the docker image version
- Machine page client end date is now not an editable box
- Bugfix: hosts now cannot set a price below 0
- Bugfix: Backblaze copy now has an "operation complete" message
- Bugfix: template editor now updates correctly
- Bugfix: issue with Backblaze data sync fixed
- Bugfix: search page inline image editor clicks/button pushes only do 1 clear action
- Bugfix: broken template link is fixed

- CLI driver_version field now supports >, >= queries
- Reserve credit balance calculation now based on GPU+storage
- New cluster rental page for cluster orders - https://vast.ai/clusters
- Added more visible warning that Coinbase does not support refunds
- Problem button stays orange after problem is reported on that machine
- Selecting template with no account active now opens login flow
- Cloud copy dialogue closes once operation starts
- Cloud copy remembers last arguments
- Earning dashboard shows payment service type
- GPU price for a stopped instance now displays in gray strikethrough when stopped
- Template editor renamed "Dockerhub location" to "Docker Options"
- Bugfix: fixed GPU flops display
- Bugfix: expired contracts now charging correctly
- Bugfix: template page template links fixed
- Bugfix: cloud sync now reporting errors correctly
- Bugfix: "add card" message in upper right now displays correctly
- Bugfix: bandwidth pricing for hosts now displays correctly in machine interface
- Bugfix: client end date now ignores host jobs
- Bugfix: docker tag selector fixed for the inline template editor
- Bugfix: env variables not highlighted red
- Bugfix: template readme editor working again
- Bugfix: open button logic fixed


We are constantly looking for ways to improve and bring you a service that continues to innovate.

September Product Update


# Getting Started with ComfyUI for Stable Diffusion: A Beginner's Handbook to Running on Vast.ai

Artwork created using artificial intelligence (AI) has made a big splash in recent years. Simple text descriptions using natural language can generate a vast array of images in different styles. From watercolor, oil, and acrylic-style digital paintings to cartoons and photorealism, fashion photography, and nature scenes, the possibilities are limited only by one’s imagination.

But how do you go from text to image? Deep learning AI tools like Stable Diffusion – based on latent diffusion processes – make it possible to create complex images with just a few words typed into a text box. Frameworks like ComfyUI make that tool accessible to just about anyone.

Released in January 2023, [ComfyUI](https://www.futuretools.io/tools/comfyui) is an open-source modular Stable Diffusion workflow engine with a node-based interface. It can create and execute advanced Stable Diffusion pipelines for use cases like text-to-image generation, image-to-image translation, and image interpolation – aka inpainting and outpainting, or filling in / extending the missing areas of an image. But ComfyUI goes even further than other AI art interfaces, providing cutting-edge configurability.

With ComfyUI, the user builds a specific workflow of their entire process. This approach is more technically challenging but also allows for unprecedented flexibility. For instance, you can preview images at any point in the generation process, or compare sampling methods by running multiple generations simultaneously.

The concept of node-based interfaces is most commonly associated with industries like 3D design and visual effects (VFX). Each node typically carries out specific code-based functions.

In the case of ComfyUI, every node is a representation of distinct facets within the Stable Diffusion process. Through the creation and interconnection of these nodes you gain the ability to seamlessly run Stable Diffusion experiments.

Speed is a major advantage, as well. Due to the granularity of its node-based interface, ComfyUI is able to re-execute only the components that have changed between executions, optimizing performance and speed. Many users have reported that ComfyUI image generation is two to three times faster than other interfaces.

Some other useful features of ComfyUI include the following:

- Workflows can be saved as JSON files, allowing you to easily reload an entire layout of nodes for consistent and reproducible results across multiple images – and share and revisit projects later.
- The interface supports loading checkpoints and safetensors models, as well as different upscaling models.
- Full workflows are saved in image metadata and can be loaded from the resulting PNG files.

With these innovations and more, ComfyUI ushers in a flexible, infinitely configurable approach to AI-powered image generation. The easy-to-use interface – with no programming knowledge required – means even beginners can quickly achieve great results. Here’s how you can get started with Vast.ai.

## Running ComfyUI on Vast

You can get ComfyUI up and running in just a few clicks.

1. Head to our [Templates page](https://cloud.vast.ai/templates/) and select ComfyUI. Yep, it’s that simple. (Already signed in? Click [here](https://cloud.vast.ai/?ref_id=43484&template_id=b349721c098bec541f600cd96e50b00f) for our ComfyUI template directly.)
   _Note: A template contains a Linux docker image, related settings and launch mode(s) for connecting to the machine. Whenever you edit a template, a new version is created and stored in your recent folder._

![](/uploads/comfyBlogScreenshot.png)

2. Filter and select the machine (GPU) for your project.
3. Select enough storage for your project.
   _Note: You can't change this after instance creation, so make sure to size correctly at start! Use the slider and filters on the left-hand side of the console to find the best fit for you._
4. Start your Instance by clicking the blue RENT button. The instance will appear in the Instances tab. After loading, your instance will start up, and you can click the Open button to launch ComfyUI.

Now you’re running ComfyUI! **Remember:** destroy your instance once you’re done to stop your GPU and avoid incurring storage charges.

It’s that easy!

For more Vast.ai templates, check out our [console](https://cloud.vast.ai/templates/).
For in-depth guides on all things Vast.ai, [explore our docs](https://vast.ai/docs/overview/introduction).

For curious minds, here’s how you install ComfyUI on your own machine. Fair warning, it’s a bit more complicated than running directly on Vast.ai.

### Installing ComfyUI Locally

For **Windows** users, the easiest method to install ComfyUI is by using the standalone installer available on the releases page:

1. Visit the official releases page: [ComfyUI Releases](https://github.com/comfyanonymous/ComfyUI/releases).
2. Download the latest comfy-windows.zip file from the list of releases.
3. Extract the downloaded zip file using a tool like 7-Zip or WinRAR.
4. After extraction, you'll have a portable ComfyUI folder that's fully configured.
5. Place your Stable Diffusion models in the "models/checkpoints" directory within the ComfyUI folder.
6. To launch ComfyUI, simply run "comfyui.exe" from the extracted folder.

## Installing ComfyUI on Linux

You can easily install ComfyUI on various Linux distributions, including Ubuntu, Debian, and Arch, by following these requirements and steps:

**Requirements:**

- NVIDIA GPU with CUDA support
- Python 3.7 or higher
- Pip package manager

**Installation Steps:**

- Clone the ComfyUI repository :://github.com/comfyanonymous/ComfyUI.git
- Install PyTorch with CUDA support
- Install dependencies
- Download Stable Diffusion models and place them in models/checkpoints
- Download a VAE like Latent Diffusion and place it in models/vae
- Launch ComfyUI


Getting Started with ComfyUI for Stable Diffusion: A Beginner's Handbook to Running on Vast.ai.

Getting Started With ComfyUI


As part of our commitment to outstanding customer service at Vast.ai, we are continually working to provide our users with the best possible product experience. An important aspect of that is making adjustments to functionality as soon as we become aware of opportunities for improvement. This means that Vast.ai customers get to enjoy a service that only grows better with time.

The occasional bug here and there is inevitable, so our first priority is to address any issues that have popped up since our last product update. Some of our latest fixes include the following:

- Delete instance no longer auto-scrolls to the top
- Stripe now works with SEPTA cards
- Earnings page correctly displays transfers
- New instance creation is prevented on occupied GPU
- Reserved instances now show as reserved correctly
- Interruptible instances display proper status
- Host billing charts show correct amounts
- Cloud sync rclone updating to show 100% complete upon success

## NVIDIA Cloud GPU Updates

Another goal we strive toward in our product updates is delivering meaningful improvements to the platform itself. Below are some highlights of the latest capabilities that have been added to improve flexibility and overall performance:

- New section for referral program to docs
- Cancel button added for rclone
- Template share link icon added to templates and confirmation dialog pop-up
- Updated [docs](https://vast.ai/docs/) with API info, search, and better navigation
- Added button to report a machine problem
- Improved ranking system for popular templates
- Refactored the host dashboard and search page, and restyled billing page
- Added filter for Internet bandwidth pricing

The full change log for our August 2023 product updates is available below for your perusal. Please check it out to see what new capabilities are possible, to help you make the most of Vast.ai!

For more information about the many features and services we offer, you can talk with us right here on our site via the chatbox at the bottom right. You can also join our Discord server to learn more and stay informed of developments, or drop us a line at [support@vast.ai](support@vast.ai).

## Change Log

- Added section for referral program to docs
- Added a CLI and GUI filter for machines with static IP
- Added cancel button for rclone
- Added H100 SMX and PCIE into the GPU filter
- Added RTX 3090 Ti into the GPU filter
- Added template share link icon to templates and confirmation dialog pop-up

- Bugfix: delete instance no longer auto-scrolls to top
- Bugfix: Stripe now works with SEPTA cards
- Bugfix: in the host install script, Ubuntu 22 NVIDIA-docker bug detection now works
- Bugfix: fixed issues with multi-GPU contract ordering
- Bugfix: earnings page displays transfers correctly
- Bugfix: open TCP/UDP ports slider now works with GPU filter
- Bugfix: fixed issue with Fios test speeds
- Bugfix: fixed issue with pricing hover
- Bugfix: fixed min CUDA filter
- Bugfix: fixed OPEN_BUTTON_PORT env

- Updated [docs](https://vast.ai/docs/) with API info, search, and better navigation
- Added button to report a machine problem
- Added better ranking system for popular templates
- Added a smart retry option for the CLI to handle rate limits
- Added warning and confirmation for instance copy from / to /
- Refactored the host dashboard
- Restyled billing page
- Added hover text to SSH info
- Refactored search page
- Added template edit button in search interface
- Minor update to privacy policy for Google Drive integration

- Bugfix: prevent new instance creation on occupied GPU
- Bugfix: force re-approve in Dropbox

- Added filter for Internet bandwidth pricing
- Copy buttons now prevent users from copying / to /
- Added a template link for referrals that always loads the latest template version
- CLI show instances command returns results in human-readable format
- Replaced warning icon with wrench to report instance issues
- Added option to sort by VRAM or VRAM/$
- Added a billing history CSV export
- Added confirmation messages after API-key, SSH key, and email account changes

- Bugfix: reserved instances now show as reserved correctly
- Bugfix: show proper status for interruptible instance
- Bugfix: host billing charts now show correct amounts
- Bugfix: cloud sync rclone updating now to show 100% complete upon success
- Bugfix: open button connecting→open now test correct ports
- Bugfix: fixed issues with host interface setting end date


As part of our commitment to outstanding customer service at Vast.ai, we are continually working to provide our users with the best possible product experience.

Vast Product Update


# Apple GPT: What We Know So Far

Apple has been secretly developing a generative AI tool called "Apple GPT." The tool's foundation is the Ajax framework, which Apple developed last year to streamline its machine learning development.

In simple terms, Apple GPT is a chatbot that can generate text, translate languages, create content, and answer your inquiries. Apple has not yet provided all the information on its intriguing project, but based on what is known so far, it will compete with even the most powerful AI chatbots, like ChatGPT, and might be made available as early as 2024.

## What Should We Expect From Apple GPT?

- According to Mark Gurman of Bloomberg, Apple uses AI technology for internal chatbot access for Apple workers and product prototyping.
- There is no easy way to incorporate the Apple GPT output into customer-facing new product features. Apple GPT's "Ajax" framework, created especially for huge language models like ChatGPT, Microsoft Bing, and Google Bard, is its beating heart. Ajax is based on Google's Jax machine learning framework, and it runs on Google Cloud.
- Historically, Apple has always taken a conservative, privacy-over-function stance when developing artificial intelligence and has frequently come under fire for having inferior skills to rivals.
- Apple CEO Tim Cook stressed the need for a careful and systematic approach to the development of AI while acknowledging that there are still problems in the field.

## Potential Features of Apple GPT

- Apple GPT might be used to enhance the precision and fluidity of Siri as well as other Apple services that rely on NLP (Natural Language Processing).
- Apple may employ Apple GPT to develop new goods and services, such as a chatbot to assist you at work or a writing helper to support your creative endeavors.
- Apple GPT might provide individualized replies to your queries or offer content to improve the user experience on its devices.

## How Effective Will Apple GPT Be?

It isn't easy to compare Apple GPT's performance to its rivals because it is currently under development and not yet offered to the marketplace.

According to a Bloomberg report, the chatbot remains relatively constant from ChatGPT, Bard, or other generative AI tools, according to multiple staff members who have tried it.
But as Apple is known for delaying the launch of new products until they are confident they offer a superior experience, we may infer that when Apple GPT is ultimately released, it will be a powerful competitor to the significant AI chatbots and have a distinct edge.

## Apple finally joins the AI revolution

It isn't the first time this year that Apple's possible AI projects have come to light; in April, it was reported that the company was developing an AI-powered health app with emotional analysis capabilities. Of course, Apple already uses AI in various ways across all of its devices.

For instance, Siri, the company's voice-activated personal assistant in all iPhones, is an example of artificial intelligence.

Apple CEO Tim Cook, who had earlier predicted that AI would be "huge," has also called attention to several privacy issues that, in his view, need to be resolved right now.

## When Will Apple GPT Be Available?

The million-dollar question people are clamoring to find out…

Insiders and rumor sites claim that Apple will make a "significant AI-related announcement" in 2024, which many believe would be regarding Apple GPT and the company's plans to make the platform available to consumers.

## Takeaway

Apple is aggressively investing in AI technology to stay competitive and provide cutting-edge solutions to its user base. However, details about the public release of Apple GPT and its possible capabilities are still unknown. Apple's strategic leap into the AI chatbot market opens the door for new developments in the near future as the realm of AI continues to develop.


Apple GPT: What We Know So Far


# Harness the Power of Cloud GPUs with Vast.AI: Running the 70B LLama2 GPTQ

The rapid advancements in the AI domain have given rise to the need for powerful computational resources. If you're involved in data science or AI research, you're already aware of the immense processing capabilities required to train complex models like the 70B LLama2 GPTQ. Fortunately, with cloud GPU rental services like [Vast.AI](https://vast.ai/), accessing these resources has become easier and more efficient than ever.

## Quick Guide to Launch Oobabooga webUI on Vast.AI

For those interested in leveraging the groundbreaking [70B LLama2 GPTQ](https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ), TheBloke made this possible.

[We’ve created a template](https://cloud.vast.ai/?ref_id=43484&template_id=47c1a7f2cd646ecbb295e4491cd48134) to auto-launch the Oobabooga webUI. Notably, this template grants users the versatility of direct SSH and Jupyter (notebook) integration.

Users should note this particular model demands a staggering 40 GB of VRAM. To accommodate this, Vast.AI's interface has a **VRAM slider** to guide you in selecting suitable machines like the A6000, A40, and A100.

For the 70B Model, we recommend the A6000 (_currently $0.50/hr_) or A40 (_currently $0.40/hr_). To run the 13B model, we recommend either a 3090 (_currently $0.20/hr_) or 4090 (_currently $0.48/hr_). To see all current pricing, refer to our dynamic [pricing page](https://vast.ai/pricing), or head to our [search console](https://cloud.vast.ai/).

## Steps to Download and Execute the Model in text-generation-webui:

1. Navigate to the Model tab.
2. In the section labeled Download custom model or LoRA, input TheBloke/Llama-2-70B-chat-GPTQ.
3. For downloading from specific branches, for instance, TheBloke/Llama-2-70B-chat-GPTQ:gptq-4bit-32g-actorder_True, refer to the Provided Files section for a comprehensive list of branches for each choice.
4. Click Download. Wait for the model to download, and once it's completed, a "Done" message will appear.
5. Set the Loader to ExLlama if you're planning to utilize a 4-bit file. Alternatively, opt for AutoGPTQ or GPTQ-for-LLaMA.
6. Using AutoGPTQ? Ensure that the "No inject fused attention" option is selected.
7. On the top left, click the refresh icon adjacent to the Model label.
8. From the Model dropdown, select the freshly downloaded model: **TheBloke/Llama-2-70B-chat-GPTQ**.
9. Once the model automatically loads, it's set and ready for deployment!
10. Secure your settings by clicking Save settings for this model. Then, click Reload the Model in the top right to ensure your configurations are retained.
11. Now, head over to the Text Generation tab, input your desired prompt, and watch the magic unfold!

For those considering running LLama2 on GPUs like the 4090s and 3090s, **TheBloke/Llama-2-13B-GPTQ** is the model you'd want. Vast.AI's platform is diverse, offering a plethora of options tailored to meet your project's requirements.

For additional details and to delve deeper, please visit the official [github page for ooba](https://github.com/oobabooga/text-generation-webui).

In today's competitive digital environment, it's crucial to have scalable and reliable computational resources at your fingertips. Vast.AI is revolutionizing the way researchers and developers access and utilize GPU power, making AI model training seamless and efficient. Whether you're an AI novice or an established researcher, Vast.AI is your go-to solution for all your cloud GPU rental needs.


Harness the Power of Cloud GPUs with Vast.AI: Running the 70B LLama2 GPTQ.

Harness the Power of Cloud GPUs with Vast.AI: Running the 70B LLama2 GPTQ


## Introduction

Meta has recently launched Llama 2, the latest edition of the widely recognized Llama models, which has been trained with a 40% increase in data. The announcement of this new model is quite thrilling, considering the meteoric rise in popularity of open-source large language models (LLMs) like Llama. The Llama LLM bares some similarities to other large language models, but it’s commercially available for free, which levels the playing field. Llama 2 was pre-trained on an enormous dataset of publicly available online text and code. The fine-tuned model, Llama-2-chat, was trained on this dataset as well as over 1 million human annotations. These models have given rise to popular offspring such as Vicuna and Falcon.

Concurrently, the open-source community has launched an abundance of utilities designed to fine-tune and deploy these language models. Tools such as Peft, Bitsandbytes, and TRL allow for fine-tuning of LLMs on machines that don't have the capacity to hold the full precision model in their GPU RAM.

The aim of this blog post is to guide you on how to fine-tune Llama 2 models on the Vast platform. We have benchmarked this on an RTX 3090, RTX 4090, and A100 SMX4 80GB. See the [latest pricing](https://vast.ai/pricing) on Vast for up to the minute on-demand rental prices.

## Benchmarks

The RTX 4090 demonstrates an impressive 1.5 8-bit samples/sec with a batch size of 8. This is almost twice as fast as running this on an A100 when accounting for batch size! Considering that the RTX 4090 is $0.50/hr, the price for performance is about 6X when compared to an A100 for $1.50/hr.

<table className="llamaTable mx-auto table-fixed" style={{color: "white"}}>
<tr className="llamaTableRow" style={{ padding: '10px', textAlign:'left',     borderBottom: "1px solid lightGray"}}>
<th style={{  width: '4.5rem' }}>Model</th>
<th style={{  width: '10rem' }}>GPU</th>
<th style={{  width: '7rem' }}>Batch Size</th>
<th style={{ width: '12rem'}}>4bit samples/sec[1]</th>
<th style={{ width: '11rem'}}>8bit samples/sec[1]</th></tr>

<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>RTX 4090</th>
  <th>8</th>
  <th>1</th>
  <th>1.5</th>
</tr>
<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>A100 SMX4 80GB</th>
  <th>32</th>
  <th>0.5</th>
  <th>0.8</th>
</tr>
<tr
  className="llamaTableRow"
  style={{
    textAlign: "left",
    color: "lightGray",
    borderBottom: "1px solid lightGray",
  }}
>
  <th>7B</th>
  <th>A100 RTX 3090</th>
  <th>8</th>
  <th>0.5</th>
  <th>0.9</th>
</tr>

</table>

[1] Here samples/sec is calculated by multiplying batch size by the inverse of s/iter that the sft_trainer script reports. All training runs had gradient accumulation steps equal to 1.

## How to run this yourself on Vast

Follow the rest of this post for a guide on how to fine-tune LLaMA 2 on Vast using TRL.

## Sign up with Meta and Hugging Face for access:

1- Request access from Meta here: [https://ai.meta.com/resources/models-and-libraries/llama-downloads/](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)

2- Request access from Hugging Face on any of the model pages: [https://huggingface.co/meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b)

3 - Set up an auth token with Hugging Face here: [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)

You will need to do both 1 and 2 in order to get access to LLaMA 2.

## Rent a powerful GPU on Vast.ai

Vast has RTX 3090s, RTX 4090s and A100s for on-demand rentals. Our pricing is typically the best you can find online.

To run LLaMA 2 fine tuning, you will want to use a Pytorch image on the machine you pick. To do that, click on [this Vast console link](https://cloud.vast.ai/?ref_id=43484&template_id=2d307d4c1ba166b8e4bafcd40d3033e2), which will select our recommended Pytorch template with SSH enabled and other settings enabled. If you don’t have an account set up, no problem! You will need to first quickly set up a Vast account by registering your email, verifying your email, and then purchasing credits.

To rent a machine, pick an RTX 3090 or RTX 4090 by selecting that filter in the upper right. Move the storage slider over on the left of the interface to ~30GB so your instance will have enough storage to download the model weights.

Hit the rent button to start the instance. Once it is done loading, hit the blue <\_ button to get the SSH details and SSH into the instance by copying and pasting that SSH command to your command line. You will now be logged into your Vast instance via SSH. If you have trouble setting up SSH, [read our SSH docs](https://vast.ai/docs/gpu-instances/ssh).

## Set up environment

Setup the packages you will need:

`pip install transformers peft trl bitsandbytes scipy`

Clone the TRL repo for the training script

`git clone https://github.com/lvwerra/trl`

Log into HuggingFace on CLI

`huggingface-cli login`

Copy the auth token you created earlier (from https://huggingface.co/settings/tokens) and paste it into the prompt when asked. You can decline adding the token to your git credentials.

Fine tune!

`python trl/examples/scripts/sft_trainer.py --model_name meta-llama/Llama-2-7b-hf --dataset_name timdettmers/openassistant-guanaco --load_in_8bit --use_peft --batch_size 8 --gradient_accumulation_steps 1`

This will download the model weights automatically, so the first time you run, it will take a bit to actually start training. The dataset is specified there and it is using the openassistant-guanaco set. If you want to try loading in the 4bit version or change the batch size, modify that command.

You should end up seeing output like this:

```
{'loss': 1.6493, 'learning_rate': 1.4096181965881397e-05, 'epoch': 0.0}
{'loss': 1.3571, 'learning_rate': 1.4092363931762796e-05, 'epoch': 0.0}
{'loss': 1.5853, 'learning_rate': 1.4088545897644193e-05, 'epoch': 0.0}
{'loss': 1.4237, 'learning_rate': 1.408472786352559e-05, 'epoch': 0.0}
{'loss': 1.7098, 'learning_rate': 1.4080909829406987e-05, 'epoch': 0.0}
{'loss': 1.4348, 'learning_rate': 1.4077091795288384e-05, 'epoch': 0.0}
{'loss': 1.6022, 'learning_rate': 1.407327376116978e-05, 'epoch': 0.01}
{'loss': 1.3352, 'learning_rate': 1.4069455727051177e-05, 'epoch': 0.01}
```

## Summary

The price for performance on Vast when using an RTX 4090 is quite impressive when compared to an A100. In this post we have shown to easy it is to spin up a very low cost GPU ($0.20 per hour) and fine-tune the LLaMA 2 models.


LLaMA 2 fine-tuning on a Vast RTX 4090 for $0.50/hr outperforms an A100.

Fine-tuning the LLaMA 2 model on RTX 4090


In the rapidly evolving world of cloud computing, Vast.ai remains steadfast in our commitment to being the leader in cost-effective cloud GPU rental. Today, we are thrilled to announce our latest feature – Reserved Instance Discounts – aimed at helping our users get even better pricing for longer term reservations.

## On-Demand vs Reserved: Striking the Right Balance

In the on-demand world of cloud computing, users only pay for what they use and have the flexibility to end a contract at any time. While this offers unrivaled flexibility, the occupancy rate – or the fraction of time GPUs are rented – often clocks in at less than 100%, sometimes even as low as 50%. The resulting impact on provider revenue is noticeable as their hardware sits idle.

Clients willing to commit upfront to a longer rental term can now leverage our Reserved Instance Discounts feature which solves this idle problem. With this, clients can commit to an extended rental contract by pre-paying in advance. While they give up the on-demand liquidity, they receive a hefty discount in return. The provider controls the discount percentage, so each offer will vary. 

## How Reserved Instance Discounts Work

Here's the breakdown:

1. Users can deposit credits into a specific instance using either the [CLI](https://cloud.vast.ai/cli/) or website GUI (look for the **green tag**). The CLI gives users a detailed control over deposit amounts, while the GUI provides a user-friendly interface. The clickable green tag in the GUI is shown for participating machines in the bottom right of the instance card.

![green tag](/uploads/prepaidicon.png)

2. The discount schedule is a function of the time duration of the balance. Larger deposits unlock greater discounts, albeit with diminishing returns. The default discount schedule offers a 20% discount for a 1-month commitment, 30% for a 3-month commitment, and 40% for a 6-month commitment.

![](/uploads/reservedmodal.png)

3. To give an example, if an instance usually costs $1/hr on-demand, a 1-month prepaid rental with the default discount schedule would cost $576 instead of $720. That's a savings of 20%!

4. When additional funds are deposited into an existing reserved instance, everything is recalculated according to the new total balance, leading to a new discount rate.

The specific discount schedule can vary from provider to provider, with 40% at 6 months being the default setting.

## Link to the Documentation

For the complete description of how these works, read our documentation for [Reserved Instance Discounts](https://vast.ai/docs/gpu-instances/21-prepaid-plans).

## Seize the Savings

Our new Reserved Instance Discounts feature is here to change the way you use cloud computing, providing an avenue to commit to your long-term compute needs while reaping the benefits of considerable discounts. So whether you are a start-up scaling your operations or an established business with predictable computing demands, let Vast.ai help you make the most of your cloud computing budget. Your savings start today with Reserved Instance Discounts. Take control of your cloud computing journey and optimize your budget like never before.





Our on-demand rentals provide incredible value. For hosts that enable this feature, you can get further discounts by reserving an instance for months.

Stretching Your Cloud Computing Budget: Introducing Reserved Instance Discounts


As we delve deeper into the realm of artificial intelligence, one of the most impactful developments is the rise of large language models. These sophisticated tools, designed to understand and generate human-like text, have a myriad of applications spanning diverse fields. This article provides a comprehensive analysis of the key factors impacting their major use cases, including customer service, education, healthcare, legal, and marketing. We will also discuss the trade-offs and challenges associated with these different approaches, highlighting the importance of informed decision-making when choosing a language model.

## An Overview of Large Language Models

Large language models, such as OpenAI's GPT-3 and Google's BERT, are AI models trained on vast amounts of text data. Their capabilities range from understanding context in a conversation, translating languages, to even generating human-like text. The implications of these capabilities are far-reaching, with applications in various sectors.

## Major Use Cases: From Customer Service to Marketing

One of the significant applications of large language models is in customer service. These models can power chatbots and virtual assistants, providing fast, accurate, and personalized responses to customer inquiries. However, the trade-off here is the lack of human touch and potential misunderstandings due to the limitations in understanding complex human emotions and intentions.

In education, large language models can help develop intelligent tutoring systems, providing personalized learning experiences for students. While this promises a more scalable education solution, the risk lies in ensuring the accuracy and appropriateness of the content generated.

In the healthcare sector, large language models can assist in analyzing patient records and medical literature, providing recommendations or aiding in diagnoses. However, the stakes are high, as inaccurate or misinterpreted information can have serious consequences.

Legal professionals can use large language models to analyze legal documents, contracts, and case laws, which can automate some aspects of legal research. But the nuanced and critical nature of legal work means that outputs must be thoroughly reviewed for accuracy.

Lastly, in marketing, language models can generate creative marketing content and analyze consumer sentiment. The challenge here lies in ensuring that the content remains authentic and resonates with the target audience.

## Challenges and Considerations

Despite the impressive capabilities of large language models, their use is accompanied by several challenges. One notable issue is the risk of bias, as these models can inadvertently reproduce biases present in their training data. This risk underscores the importance of carefully curating training data and monitoring model outputs.

The selection of an appropriate language model often depends on the user's specific needs and constraints. While more powerful models can generate higher-quality outputs, they also require more computational resources, thus increasing costs.

## Conclusion

Large language models offer transformative potential across various sectors, from customer service to marketing. However, their implementation is not without challenges, and users must carefully weigh the trade-offs. By considering factors such as performance, cost, and the potential for bias, users can make informed decisions about the best model to use. Despite the complexities, the world of large language models holds immense promise and potential for advancement.

We take a deep dive into the pros and cons of large language models like OpenAI's GPT-4 and Google's BERT systems.

Blog