
When it comes to Falcon 180B, the epic new LLM set loose on the world by the folks at The Technology Innovation Institute (TII), you can’t be shy with the hardware!

This model is a beast - it is trained on more than 3 trillion tokens - mostly coming from TII’s “RefinedWeb” dataset.

The “180B” in the name hints at the incredible size and power of this model. While the B doesn’t stand for beast - it does reflect the 180 billion neural network parameters it uses.

That’s a lot. And it takes serious GPU RAM and computing power to operate.

This is where Vast.ai comes in. With Vast.ai you can rent the GPU power you need to run immense models like Falcon 180B.

So to get you up and running we’ll walk step by step through just how to get Falcon 180B running with Vast.ai. We will show you how to do it with the Web UI as well as directly with SSH.

In both cases - you’ll want to purchase some credits prior to starting, so that you will be able to run your instances.

## Use the Web UI to run the model!

### Step 1: Select the template you would like to use.

Click [templates](https://cloud.vast.ai/templates/) on the menu on the left hand side of the screen (expanding it with the “hamburger” icon on the top left if necessary).

From the template list, find the “Oobabooga” template and select it. This will run a simple webui for downloading, loading and chatting with the Falcon 180B model.

![](</uploads/OobaboogaLLMWebUI(LLaMA2).png>)

### Step 2: Select enough storage

Move the storage slider over to allocate storage for the instance. Storage can only be set before the instance is created, so this is an important step.

Falcon 180B requires differing amounts of storage depending on the quantization method you select.

We recommend at least 100GB for the model we recommend.

### Step 3: Find an appropriate machine - and click the Rent button to start it.

Falcon 180B requires a lot of GPU VRAM. With no quantization, the LLM requires ~640GB of GPU VRAM!

With quantization, the amount of GPU VRAM is reduced. However, the model quality degrades when more quantization is used. It is a balance. We recommend the Bloke’s quantized models - https://huggingface.co/TheBloke/Falcon-180B-Chat-GPTQ#provided-files-and-gptq-parameters

The linked chart shows the tradeoff between more quantization and quality. At the high end, the server will need 94.25 GB of GPU VRAM. On the low end, 70.54 GB.We recommend the main branch which uses 4 bit quantization and requires 95 GB of GPU VRAM along with 100 GB of disk space.

This will fit in a variety of configurations. You could use a 2X A6000/A40 or a 4X 3090/4090/A5000. The total GPU RAM needs to be over 95GB.

![](/uploads/C5FUug48n7mMT1m_aZPFEQalyeux2a8C-rthdQDdmuDFAVwApD-Q510pGghYczjegkYkMqOiVbmylzJ39FVt1Ve-TyW1L7qd56S3cBEof0eU3fbZI-PQAYTxp9GK.jpg)

Click the blue rent button to create and launch your instance.

### Step 3: Open the Web UI & download the model.

On the instance screen, once your instance has started, there will be a button labeled “Open” - this will take you to the Web UI.

Once the Web UI has loaded, on the top of the screen there will be a menu. Select “Model” from that menu.

On the right hand side of the screen there will then be a text box for downloading the model.

Enter the following: TheBloke/Falcon-180B-Chat-GPTQ

Press the Download button. This will take some time.
![](/uploads/HrKSLo7ao3bEGM1KW_IXvUQGnK9dNkDZGGdlcJnc4NRvzpVj7G64eJKKA2GSgEaaNWbtknyamaA9_10sTS0bo6G-qwvG53hEe6LT5Y-Mp4aBT7cdYZCcqh3wZYAR.png)

### Step 4: Load the model.

Slide all of the GPU memory sliders all the way to the right, so that you’re allocating all available memory to the model.

Once the download is complete the model will be listed in the drop down to the upper left. Select it and click load.

### Step 5: Use the model.

Click chat on the top menu. This will take you to a chat interface - you can now interact with the model in the familiar way.

## Use SSH to run the model!

An alternative way to run the model is to use SSH.

### Step 1: Set up your SSH access.

Ultimately this boils down to two things:

- Generating your key pair.
- Putting your public key on your account.

Some of this is platform specific - but detailed assistance can be found here: https://vast.ai/docs/instance-setup/ssh

### Step 2: Select the Oobabooga template.

Follow the same steps that as in Step 1 from the Web UI instructions

### Step 3: Find an appropriate machine - and click the Rent button to start it.

Follow steps 2 & 3 above to select an appropriate GPU server and match the total GPU VRAM with the size of the quantized model you want to run. Approximately 100GB of total VRAM should work.

### Step 4: SSH to the instance.

Once the instance has started - click the “connect to” button. This will bring up a screen that has SSH details - copy those to your terminal and execute them.

### Step 5: Install packages & Execute example Python code.

pip3 install transformers>=4.33.0 optimum>=1.12.0
pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/ # Use cu117 if on CUDA 11.7

At the command line for your instance (which you’ve SSH’d to) execute the following:

```
Python

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_name_or_path = "TheBloke/Falcon-180B-Chat-GPTQ"

model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                             device_map="auto",
                                             revision="gptq-3bit--1g-actorder_True")

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)

prompt = "Tell me about dogs"
prompt_template=f'''User: {prompt}
Assistant: '''

print("\n\n*** Generate:")

input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, do_sample=True, temperature=0.7, max_new_tokens=512)
print(tokenizer.decode(output[0]))
```

This sample code is from “TheBloke’s” Hugging Face card, at https://huggingface.co/TheBloke/Falcon-180B-Chat-GPTQ. It shows the system operating - though I’m sure you have many ideas on more interesting things to do with it, once you get it running.

Wrapping Up

So there you have it - quickly, painlessly spinning up the powerful Vast.ai platform and executing a cutting edge large language model. So whatever your AI use-case is, you can now be confident that you can make it a reality with minimal effort.


Running Falcon 180B on Vast


Here at Vast, we offer both **on-demand** and **interruptible** rental options. For most users, the on-demand option provides stable access to the GPU rented for the duration of the rental period. However for advanced users who want to save money and can have their job interrupted, interruptibles are an interesting option.

## On-Demand

**On-demand instances** have a fixed price set by the host. The advantage of renting on demand is that you have high priority and gain exclusive control over the GPU(s) for the lifetime of the instance. This duration of time is up to the client, but a maximum duration determined by the host appears on the offer card when searching the marketplace.

Once the designated time limit expires, the instance is automatically stopped. You can then copy data off of the stopped instance to a new instance or to your local machine. It is best to assume that your instance will be lost once its lifetime expires, but the host does have the option to renew the contract, adding more lifetime to your instance. Reasons not to do so may include taking the machine offline for scheduled maintenance or upgrades.

If you need an instance to run for many days or weeks at a time, such as for larger artificial intelligence or machine learning tasks, ensure that the maximum duration exceeds the estimated job completion time. Another best practice is to select a host with a high reliability score.

## Interruptible

Our **interruptible instances** use a direct bidding system to determine priority on a continual basis. This sets Vast.ai apart from others in the field. For instance, although AWS spot instances and GCE interruptible instances can similarly be interrupted by on-demand instances, they do not use a direct bidding system and some will terminate interruptible instances after 24 hours.

With Vast.ai, clients set a bid price for their instance, and the current highest bid determines the instance that runs; any others are paused. Higher bids are obviously favored to maximize revenue for hosts, but this system can nonetheless reduce clients’ costs by fifty percent or even more in certain use cases.

For any given interruptible instance, a higher bid means higher priority on the machine. Lower-priority instances are paused until their bid is raised enough to regain the highest priority or until a higher bid finishes up and is no longer running. On-demand instances will always take precedence over interruptible instances, as well.

## Important Considerations

As you get started with your GPU rental, the following are a few crucial points to keep in mind:

- The bidding method cannot be changed after an instance is rented.

- An interruptible instance can be abruptly paused at any time if another user places a higher bid or if an on-demand rental is created for the same resources. This stops all processes that are running, so it is recommended to save your work to disk and consider saving your outputs to cloud storage at regular intervals as well. You can still transfer data off a stopped instance.

- It’s best to limit the use of interruptible instances to fault-tolerant workloads capable of handling pauses in runtime. Some examples include batch jobs, code builds, load tests, background processing, data analysis, and optional tasks.

Lastly, we do offer another rental type where you can deposit credits to get [reserved instance discounts](/docs/instance-rental/prepaid-plans) anytime you like. This involves converting an on-demand rental into a prepaid reserved rental at a discount, ideal for users who have predictable longer-term computing needs and want to get the most out of their budget.

For more information about the above rental options and answers to any further questions you may have, we welcome you to reach out anytime via the chat box below or by emailing us at [support@vast.ai](mailto:support@vast.ai). We look forward to helping you find the GPU rental solution that best fits your unique needs!


Here at Vast, we offer both on-demand and interruptible rental options

On Demand vs Interruptible Rental Types


# Apple GPT: What We Know So Far

Apple has been secretly developing a generative AI tool called "Apple GPT." The tool's foundation is the Ajax framework, which Apple developed last year to streamline its machine learning development.

In simple terms, Apple GPT is a chatbot that can generate text, translate languages, create content, and answer your inquiries. Apple has not yet provided all the information on its intriguing project, but based on what is known so far, it will compete with even the most powerful AI chatbots, like ChatGPT, and might be made available as early as 2024.

## What Should We Expect From Apple GPT?

- According to Mark Gurman of Bloomberg, Apple uses AI technology for internal chatbot access for Apple workers and product prototyping.
- There is no easy way to incorporate the Apple GPT output into customer-facing new product features. Apple GPT's "Ajax" framework, created especially for huge language models like ChatGPT, Microsoft Bing, and Google Bard, is its beating heart. Ajax is based on Google's Jax machine learning framework, and it runs on Google Cloud.
- Historically, Apple has always taken a conservative, privacy-over-function stance when developing artificial intelligence and has frequently come under fire for having inferior skills to rivals.
- Apple CEO Tim Cook stressed the need for a careful and systematic approach to the development of AI while acknowledging that there are still problems in the field.

## Potential Features of Apple GPT

- Apple GPT might be used to enhance the precision and fluidity of Siri as well as other Apple services that rely on NLP (Natural Language Processing).
- Apple may employ Apple GPT to develop new goods and services, such as a chatbot to assist you at work or a writing helper to support your creative endeavors.
- Apple GPT might provide individualized replies to your queries or offer content to improve the user experience on its devices.

## How Effective Will Apple GPT Be?

It isn't easy to compare Apple GPT's performance to its rivals because it is currently under development and not yet offered to the marketplace.

According to a Bloomberg report, the chatbot remains relatively constant from ChatGPT, Bard, or other generative AI tools, according to multiple staff members who have tried it.
But as Apple is known for delaying the launch of new products until they are confident they offer a superior experience, we may infer that when Apple GPT is ultimately released, it will be a powerful competitor to the significant AI chatbots and have a distinct edge.

## Apple finally joins the AI revolution

It isn't the first time this year that Apple's possible AI projects have come to light; in April, it was reported that the company was developing an AI-powered health app with emotional analysis capabilities. Of course, Apple already uses AI in various ways across all of its devices.

For instance, Siri, the company's voice-activated personal assistant in all iPhones, is an example of artificial intelligence.

Apple CEO Tim Cook, who had earlier predicted that AI would be "huge," has also called attention to several privacy issues that, in his view, need to be resolved right now.

## When Will Apple GPT Be Available?

The million-dollar question people are clamoring to find out…

Insiders and rumor sites claim that Apple will make a "significant AI-related announcement" in 2024, which many believe would be regarding Apple GPT and the company's plans to make the platform available to consumers.

## Takeaway

Apple is aggressively investing in AI technology to stay competitive and provide cutting-edge solutions to its user base. However, details about the public release of Apple GPT and its possible capabilities are still unknown. Apple's strategic leap into the AI chatbot market opens the door for new developments in the near future as the realm of AI continues to develop.


Industry

Running Falcon 180B on Vast

On Demand vs Interruptible Rental Types

Apple GPT: What We Know So Far