
# How to Run Mixtral 8X7B

Mixtral is the latest open source model from Mistral AI. The model was release with no documentation, explanation or other comment from Mistral AI on Dec 8th via a single Tweet of the [magnet link](https://x.com/MistralAI/status/1733150512395038967?s=20). Mistral AI has several open source LLM models that are popular including Mistral 7B.

Mixtral 8X7B is notable in that it is a mixture of experts (MoE) model with exceptional ability. This guide uses some hacky implementations to get it to run. Once the model is out for a few months, it will certainly gain more support with open source tools.

To run it on Vast.ai, use a 8X 4090/3090 or a 4X A6000/A40 instance. You will need 120GB of total GPU RAM on the instance for the model. We use the typical development Pytorch recommended template and a modified version of Illama to run inference. This is a base model with no instruction fine tuning, so proper prompt techniques are helpful.

## Steps:

1. **Select Template**:

   - Choose the [Devel template](https://cloud.vast.ai/?ref_id=43484&template_id=e4c5e88bc289f4eecb0c955c4fe7430d) for Pytorch by clicking that link or selecting it in Templates.

2. **Rent a Server**:

   - Select a 8X 4090/3090 or 4X A40/A6000
   - Add at least 120GB of disk space
   - Click the Rent button
   - Purchase credits if needed.
   - Refer to the [quickstart guide](https://docs.vast.ai/quickstart) for help.

3. **SSH into the Machine**:

   - Obtain SSH information from the instance card by clicking on the >\_ button
   - Connect to the instance via the SSH command. It will look like this but have the port and server IP info of your instance:
     ```
     ssh -p <yourport> root@<yourserverip> -L 8080:localhost:8080
     ```

4. **Download Mixtral 8X7B Model Weights**:

   - Use a torrent file. This typically takes about 15 minutes, depending on server download speed.
   - To download the torrent file, run:
     ```
     apt install transmission-cli
     transmission-cli magnet:?xt=urn:btih:5546272da9065eddeb6fcd7ffddeef5b75be79a7&dn=mixtral-8x7b-32kseqlen&tr=udp%3A%2F%http://2Fopentracker.i2p.rocks%3A6969%2Fannounce&tr=http%3A%2F%http://2Ftracker.openbittorrent.com%3A80%2Fannounce
     ```

5. **Install Illama for Inference**:

   - Clone the repository:
     ```
     git clone https://github.com/dzhulgakov/llama-mistral
     ```
   - Install dependencies:
     ```
     pip install fire sentencepiece
     ```

6. **Run Mixtral 8X7B!**:

   - Load the model weights and run the prompts in `example_text_completion.py`. If you use 8 GPUS, append "--num-gpus 8" at the end of the run command like in the example below. If you use 2 GPUS, remove that line.:
     ```
     cd llama-mistral
     python example_text_completion.py ../Downloads/mixtral-8x7b-32kseqlen/ ../Downloads/mixtral-8x7b-32kseqlen/tokenizer.model --num-gpus 8
     ```
     And there you go! You will get the output of the prompts that are currently listed

7. **Modify Prompts**:
   - Install Nano:
     ```
     apt install nano
     ```
   - Edit the file:
     ```
     nano example_text_completion.py
     ```
   - Find and modify the prompts at the bottom.
   - Re-run the python file to get new responses.

Thank you for using Vast.ai to run the latest open source LLM, Mixtral 8X7B. Drop us a line and let us know what you think of the model.


Running the latest open source model from Mistral AI called Mixtral 8x7b on Vast.ai using 8X 4090s or 8X 3090s

How to run Mixtral 8X7B


# PyTorch vs TensorFlow: Which One Is Right For You? 

[PyTorch](https://pytorch.org/) and [TensorFlow](https://www.tensorflow.org/) are two of the most widely used deep learning libraries in the field of artificial intelligence. Both are supported on Vast.ai with easy to use templates. PyTorch, initially developed by Meta, offers an intuitive approach to building neural networks and is favored for its flexibility and ease of use in research. On the other hand, TensorFlow, developed by Google Brain, is designed for large-scale and complex machine learning models, with strong support for production environments. This article will compare PyTorch and TensorFlow, helping you understand which library may be more suitable for your project's needs and computational requirements.

# PyTorch Overview

PyTorch, initially developed by Meta, has quickly become a top choice for deep learning enthusiasts and professionals thanks to its dynamic computational graphing and deep integration with the Python language. It's specifically designed to facilitate rapid development and iteration of deep learning models, resonating with the workflow of many developers.

The true power of PyTorch comes when combined with NVIDIA's CUDA, a parallel computing platform that allows dramatic increases in computing performance by harnessing the power of GPUs. With CUDA, PyTorch can perform data-intensive computations at a fraction of the time it would take on traditional CPUs. This capability is very important when training complex models or dealing with massive datasets, a common task in AI research and development.

PyTorch also supports production needs with tools like TorchScript for model exporting and TorchServe for deployment. Its ecosystem is full of specialized libraries, fostering a collaborative environment where developers share tools and pre-built models, enhancing the PyTorch experience for both research and industry use.

# TensorFlow Overview

TensorFlow, developed by the Google Brain team, has also become one of the most widely used frameworks in the industry due to its scalable and flexible architecture. Its architecture centers around a static computational graph that simplifies and accelerates data processing. This feature, combined with its capacity for distributed computing, makes TensorFlow ideal for large-scale projects. In addition, transitioning models from the lab to the field is seamless with TensorFlow, thanks to tools like TensorFlow Serving for servers and TensorFlow Lite for mobile devices.

Furthermore, TensorFlow's integration with TensorFlow Extended (TFX) eases the creation of end-to-end machine learning workflows, making it more suitable for production use. The platform also includes comprehensive tools such as TensorBoard, which simplifies model analysis and debugging with its visualization capabilities.

The framework has fostered a massive community ranging from academic researchers to industry professionals. This diverse user base benefits from a large source of educational materials, add-ons from third parties, and forums for troubleshooting. Its widespread use in the corporate sector attests to TensorFlow's dependability and advanced capabilities. Ongoing input from this community contributes to TensorFlow's growth, keeping it at the forefront of AI application development.

# Comparing PyTorch and TensorFlow Metrics

### Performance Comparison

Both PyTorch and TensorFlow offer fast performance, but they do come with their own set of advantages and disadvantages. [A benchmark comparison revealed](https://viso.ai/deep-learning/pytorch-vs-tensorflow/) that PyTorch had a better performance compared to TensorFlow, particularly when offloading most of the computation to the cuDNN and cuBLAS libraries, which are essential components for GPU-accelerated computing​. However, performance can vary depending on the specific use case and hardware, with both frameworks capable of achieving high performance when properly optimized​​.

### Training Time and Memory Usage

[In a direct comparison](https://viso.ai/deep-learning/pytorch-vs-tensorflow/) utilizing CUDA, PyTorch outperforms TensorFlow in training speed, completing tasks in an average of 7.67 seconds against TensorFlow's 11.19 seconds. However, TensorFlow is more memory-efficient, using 1.7 GB of RAM during training compared to PyTorch’s 3.5 GB. Therefore, for quicker training, PyTorch is favorable, but for lower memory usage, TensorFlow is the better choice​.

### Ease of Use

PyTorch is often favored for its intuitive, Pythonic interface, making it easier for rapid prototyping, especially when utilizing CUDA for GPU acceleration. On the other hand, TensorFlow, although having a steeper learning curve, offers a more structured environment suitable for large-scale or commercial projects. Over time, TensorFlow has improved its usability, making it more approachable, but PyTorch is still considered more usable. In the end, PyTorch is simpler for research and development, while TensorFlow is structured for larger, production-oriented projects. However, regardless of which framework fits your project’s requirements, the computational requirements remain a substantial factor that can influence your choice.

# Addressing Computational Demands with Vast.ai

The computational intensity of working with PyTorch or TensorFlow can present challenges, especially when processing large datasets or training complex models. Both frameworks perform much better with high-performance GPUs, but they are not always readily accessible or economically feasible for every developer. [Vast.ai](https://vast.ai/) steps in to fill this gap by offering affordable and scalable GPU resources. With Vast.ai, developers and researchers can expedite training times and scale computing resources in tandem with their project demands, effectively leveraging the strengths of either framework without hardware limitations.

# Conclusion

In the end, your choice between PyTorch and TensorFlow should align with your project requirements: PyTorch for its user-friendly nature in research and development, and TensorFlow for its robustness in large-scale, production-level projects. Both frameworks have their merits and can be significantly enhanced by overcoming hardware limitations, a service that [Vast.ai](http://vast.ai/pricing) provides effectively. With the right tools and resources, you can ensure that your machine learning projects are not only innovative but also computationally feasible.


PyTorch vs TensorFlow: Which One Is Right For You? 

PyTorch vs TensorFlow: Which One Is Right For You


Writing a book is a daunting task.

Coming up with just the right words, and keeping track of the ideas that you’re weaving together - this is difficult work. The difficulty can even make it seem that there might be better ways to be spending your time.

A new generation of tools seeks to solve this challenge though - to allow you to bring your powerful intuition and vision to life, while leaving the management of the details to the machines.

This may sound too good to be true - but it’s not. Try the tools out for yourself and see. We’ll help you get started.

Setting some context is important, though - so we’ll go quickly over some of the basics, and then show you step-by-step how to get started.

KoboldAI is a tool that provides functionality specifically related to writing. KoboldAI in turn can leverage any number of models while producing its assistance. The one that we will use here is called Pygmalion.

And finally the magic dust that brings all of this technical wizardry together is Vast.ai. We will install the KoboldAI tooling with the Pygmalion model on the powerful Vast.ai GPUs - which we can rent for pennies-per-hour. This not only gives us a cost-effective way to put these tools to use - it also allows us to only pay for what we need.

## Step 1: Create a Vast.ai Account

If you don't already have a Vast.ai account, you'll need to create one. Visit [https://vast.ai](Vast.ai), sign up, and follow the on-screen instructions to create your account.

## Step 2: Add Credits

Before renting a virtual machine on Vast.ai, you'll need to add credits to your account. This is how you pay for the GPU resources you use. Simply click on your account balance and follow the prompts to add funds securely. Vast accepts credit card payments through Stripe and crypto through Crypto.com and Coinbase.

## Step 3: Running the Template

1. Browse Available Templates: Within the Vast.ai platform, take advantage of their pre-configured templates, which include the Pygmalion model. Visit the [Vast.ai Templates](https://cloud.vast.ai/templates/) section to explore these ready-to-use configurations.

2. Select the [KoboldAI + Pygmalion Template](https://cloud.vast.ai/?ref_id=43484&template_id=0930ce52c987ce85e981371fadacd7b1): Click that link, or locate and select the Pygmalion template from the list of available templates. These templates come pre-packaged with the necessary settings, eliminating the need for manual setup.
   [![ScreenShot](/uploads/1DCF197C-F94C-4EE0-9A60-DBD3ED8AB42B.jpeg)](/uploads/1DCF197C-F94C-4EE0-9A60-DBD3ED8AB42B.jpeg)

3. Customize as Needed: While the templates streamline the process, you still have the flexibility to customize certain parameters to suit your specific requirements, such as GPU type or resource allocation. Select your preferred storage size to find a machine for your needs.

4. Rent the Instance: Clicking the “Rent” button will start the instance.
   [![ScreenShot2](/uploads/B87F9218-AEA0-450C-BEFF-D5468A6C4212_4_5005_c.jpeg)](/uploads/B87F9218-AEA0-450C-BEFF-D5468A6C4212_4_5005_c.jpeg)

## Step 4: Write Your Next Great Novel!

Once your instance has started (this will take a few minutes), the final entries in the logs will show the URLs that can be used to access the system. They are labeled according to the type of the interface. Take one of these and plug it into your browser, and you can begin your work.

You will notice that one of these URLs is for use as an API - so if you are looking to programmatically include the capabilities of this tool into your own application, this is how you would do it.

### The Sky Is The Limit

This is just a small taste of the capability that might be provided by this and by similar tools. The flexible compute power offered by Vast.ai offers a cost-effective and straightforward way to get started - either with experimenting with workflows or with putting them to real use.


Exploring AI-Based Writing


When it comes to Falcon 180B, the epic new LLM set loose on the world by the folks at The Technology Innovation Institute (TII), you can’t be shy with the hardware!

This model is a beast - it is trained on more than 3 trillion tokens - mostly coming from TII’s “RefinedWeb” dataset.

The “180B” in the name hints at the incredible size and power of this model. While the B doesn’t stand for beast - it does reflect the 180 billion neural network parameters it uses.

That’s a lot. And it takes serious GPU RAM and computing power to operate.

This is where Vast.ai comes in. With Vast.ai you can rent the GPU power you need to run immense models like Falcon 180B.

So to get you up and running we’ll walk step by step through just how to get Falcon 180B running with Vast.ai. We will show you how to do it with the Web UI as well as directly with SSH.

In both cases - you’ll want to purchase some credits prior to starting, so that you will be able to run your instances.

## Use the Web UI to run the model!

### Step 1: Select the template you would like to use.

Click [templates](https://cloud.vast.ai/templates/) on the menu on the left hand side of the screen (expanding it with the “hamburger” icon on the top left if necessary).

From the template list, find the “Oobabooga” template and select it. This will run a simple webui for downloading, loading and chatting with the Falcon 180B model.

![](</uploads/OobaboogaLLMWebUI(LLaMA2).png>)

### Step 2: Select enough storage

Move the storage slider over to allocate storage for the instance. Storage can only be set before the instance is created, so this is an important step.

Falcon 180B requires differing amounts of storage depending on the quantization method you select.

We recommend at least 100GB for the model we recommend.

### Step 3: Find an appropriate machine - and click the Rent button to start it.

Falcon 180B requires a lot of GPU VRAM. With no quantization, the LLM requires ~640GB of GPU VRAM!

With quantization, the amount of GPU VRAM is reduced. However, the model quality degrades when more quantization is used. It is a balance. We recommend the Bloke’s quantized models - https://huggingface.co/TheBloke/Falcon-180B-Chat-GPTQ#provided-files-and-gptq-parameters

The linked chart shows the tradeoff between more quantization and quality. At the high end, the server will need 94.25 GB of GPU VRAM. On the low end, 70.54 GB.We recommend the main branch which uses 4 bit quantization and requires 95 GB of GPU VRAM along with 100 GB of disk space.

This will fit in a variety of configurations. You could use a 2X A6000/A40 or a 4X 3090/4090/A5000. The total GPU RAM needs to be over 95GB.

![](/uploads/C5FUug48n7mMT1m_aZPFEQalyeux2a8C-rthdQDdmuDFAVwApD-Q510pGghYczjegkYkMqOiVbmylzJ39FVt1Ve-TyW1L7qd56S3cBEof0eU3fbZI-PQAYTxp9GK.jpg)

Click the blue rent button to create and launch your instance.

### Step 3: Open the Web UI & download the model.

On the instance screen, once your instance has started, there will be a button labeled “Open” - this will take you to the Web UI.

Once the Web UI has loaded, on the top of the screen there will be a menu. Select “Model” from that menu.

On the right hand side of the screen there will then be a text box for downloading the model.

Enter the following: TheBloke/Falcon-180B-Chat-GPTQ

Press the Download button. This will take some time.
![](/uploads/HrKSLo7ao3bEGM1KW_IXvUQGnK9dNkDZGGdlcJnc4NRvzpVj7G64eJKKA2GSgEaaNWbtknyamaA9_10sTS0bo6G-qwvG53hEe6LT5Y-Mp4aBT7cdYZCcqh3wZYAR.png)

### Step 4: Load the model.

Slide all of the GPU memory sliders all the way to the right, so that you’re allocating all available memory to the model.

Once the download is complete the model will be listed in the drop down to the upper left. Select it and click load.

### Step 5: Use the model.

Click chat on the top menu. This will take you to a chat interface - you can now interact with the model in the familiar way.

## Use SSH to run the model!

An alternative way to run the model is to use SSH.

### Step 1: Set up your SSH access.

Ultimately this boils down to two things:

- Generating your key pair.
- Putting your public key on your account.

Some of this is platform specific - but detailed assistance can be found here: https://vast.ai/docs/instance-setup/ssh

### Step 2: Select the Oobabooga template.

Follow the same steps that as in Step 1 from the Web UI instructions

### Step 3: Find an appropriate machine - and click the Rent button to start it.

Follow steps 2 & 3 above to select an appropriate GPU server and match the total GPU VRAM with the size of the quantized model you want to run. Approximately 100GB of total VRAM should work.

### Step 4: SSH to the instance.

Once the instance has started - click the “connect to” button. This will bring up a screen that has SSH details - copy those to your terminal and execute them.

### Step 5: Install packages & Execute example Python code.

pip3 install transformers>=4.33.0 optimum>=1.12.0
pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/ # Use cu117 if on CUDA 11.7

At the command line for your instance (which you’ve SSH’d to) execute the following:

```
Python

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_name_or_path = "TheBloke/Falcon-180B-Chat-GPTQ"

model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                             device_map="auto",
                                             revision="gptq-3bit--1g-actorder_True")

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)

prompt = "Tell me about dogs"
prompt_template=f'''User: {prompt}
Assistant: '''

print("\n\n*** Generate:")

input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, do_sample=True, temperature=0.7, max_new_tokens=512)
print(tokenizer.decode(output[0]))
```

This sample code is from “TheBloke’s” Hugging Face card, at https://huggingface.co/TheBloke/Falcon-180B-Chat-GPTQ. It shows the system operating - though I’m sure you have many ideas on more interesting things to do with it, once you get it running.

Wrapping Up

So there you have it - quickly, painlessly spinning up the powerful Vast.ai platform and executing a cutting edge large language model. So whatever your AI use-case is, you can now be confident that you can make it a reality with minimal effort.


Running Falcon 180B on Vast


Here at Vast, we offer both **on-demand** and **interruptible** rental options. For most users, the on-demand option provides stable access to the GPU rented for the duration of the rental period. However for advanced users who want to save money and can have their job interrupted, interruptibles are an interesting option.

## On-Demand

**On-demand instances** have a fixed price set by the host. The advantage of renting on demand is that you have high priority and gain exclusive control over the GPU(s) for the lifetime of the instance. This duration of time is up to the client, but a maximum duration determined by the host appears on the offer card when searching the marketplace.

Once the designated time limit expires, the instance is automatically stopped. You can then copy data off of the stopped instance to a new instance or to your local machine. It is best to assume that your instance will be lost once its lifetime expires, but the host does have the option to renew the contract, adding more lifetime to your instance. Reasons not to do so may include taking the machine offline for scheduled maintenance or upgrades.

If you need an instance to run for many days or weeks at a time, such as for larger artificial intelligence or machine learning tasks, ensure that the maximum duration exceeds the estimated job completion time. Another best practice is to select a host with a high reliability score.

## Interruptible

Our **interruptible instances** use a direct bidding system to determine priority on a continual basis. This sets Vast.ai apart from others in the field. For instance, although AWS spot instances and GCE interruptible instances can similarly be interrupted by on-demand instances, they do not use a direct bidding system and some will terminate interruptible instances after 24 hours.

With Vast.ai, clients set a bid price for their instance, and the current highest bid determines the instance that runs; any others are paused. Higher bids are obviously favored to maximize revenue for hosts, but this system can nonetheless reduce clients’ costs by fifty percent or even more in certain use cases.

For any given interruptible instance, a higher bid means higher priority on the machine. Lower-priority instances are paused until their bid is raised enough to regain the highest priority or until a higher bid finishes up and is no longer running. On-demand instances will always take precedence over interruptible instances, as well.

## Important Considerations

As you get started with your GPU rental, the following are a few crucial points to keep in mind:

- The bidding method cannot be changed after an instance is rented.

- An interruptible instance can be abruptly paused at any time if another user places a higher bid or if an on-demand rental is created for the same resources. This stops all processes that are running, so it is recommended to save your work to disk and consider saving your outputs to cloud storage at regular intervals as well. You can still transfer data off a stopped instance.

- It’s best to limit the use of interruptible instances to fault-tolerant workloads capable of handling pauses in runtime. Some examples include batch jobs, code builds, load tests, background processing, data analysis, and optional tasks.

Lastly, we do offer another rental type where you can deposit credits to get [reserved instance discounts](/docs/instance-rental/prepaid-plans) anytime you like. This involves converting an on-demand rental into a prepaid reserved rental at a discount, ideal for users who have predictable longer-term computing needs and want to get the most out of their budget.

For more information about the above rental options and answers to any further questions you may have, we welcome you to reach out anytime via the chat box below or by emailing us at [support@vast.ai](mailto:support@vast.ai). We look forward to helping you find the GPU rental solution that best fits your unique needs!


Here at Vast, we offer both on-demand and interruptible rental options

Industry

How to run Mixtral 8X7B

PyTorch vs TensorFlow: Which One Is Right For You

Exploring AI-Based Writing

Running Falcon 180B on Vast

On Demand vs Interruptible Rental Types