
# How to Run Mixtral 8X7B

Mixtral is the latest open source model from Mistral AI. The model was release with no documentation, explanation or other comment from Mistral AI on Dec 8th via a single Tweet of the [magnet link](https://x.com/MistralAI/status/1733150512395038967?s=20). Mistral AI has several open source LLM models that are popular including Mistral 7B. 

Mixtral 8X7B is notable in that it is a mixture of experts (MoE) model with exceptional ability. This guide uses some hacky implementations to get it to run. Once the model is out for a few months, it will certainly gain more support with open source tools.

To run it on Vast.ai, use a 8X 4090/3090 or a 4X A6000/A40 instance. You will need 120GB of total GPU RAM on the instance for the model. We use the typical development Pytorch recommended template and a modified version of Illama to run inference. This is a base model with no instruction fine tuning, so proper prompt techniques are helpful. 

## Steps:

1. **Select Template**:
   - Choose the [Devel template](https://cloud.vast.ai/?ref_id=43484&template_id=e4c5e88bc289f4eecb0c955c4fe7430d) for Pytorch by clicking that link or selecting it in Templates.

2. **Rent a Server**:
   - Select a 8X 4090/3090 or 4X A40/A6000
   - Add at least 120GB of disk space
   - Click the Rent button
   - Purchase credits if needed.
   - Refer to the [quickstart guide](https://vast.ai/docs/overview/quickstart) for help.

3. **SSH into the Machine**:
   - Obtain SSH information from the instance card by clicking on the >_ button 
   - Connect to the instance via the SSH command. It will look like this but have the port and server IP info of your instance:
     ```
     ssh -p <yourport> root@<yourserverip> -L 8080:localhost:8080
     ```

4. **Download Mixtral 8X7B Model Weights**:
   - Use a torrent file. This typically takes about 15 minutes, depending on server download speed.
   - To download the torrent file, run:
     ```
     apt install transmission-cli
     transmission-cli magnet:?xt=urn:btih:5546272da9065eddeb6fcd7ffddeef5b75be79a7&dn=mixtral-8x7b-32kseqlen&tr=udp%3A%2F%http://2Fopentracker.i2p.rocks%3A6969%2Fannounce&tr=http%3A%2F%http://2Ftracker.openbittorrent.com%3A80%2Fannounce
     ```

5. **Install Illama for Inference**:
   - Clone the repository:
     ```
     git clone https://github.com/dzhulgakov/llama-mistral
     ```
   - Install dependencies:
     ```
     pip install fire sentencepiece
     ```

6. **Run Mixtral 8X7B!**:
   - Load the model weights and run the prompts in `example_text_completion.py`. If you use 8 GPUS, append "--num-gpus 8" at the end of the run command like in the example below. If you use 2 GPUS, remove that line.:
     ```
     cd llama-mistral
     python example_text_completion.py ../Downloads/mixtral-8x7b-32kseqlen/ ../Downloads/mixtral-8x7b-32kseqlen/tokenizer.model --num-gpus 8
     ```
	 And there you go! You will get the output of the prompts that are currently listed 

7. **Modify Prompts**:
   - Install Nano:
     ```
     apt install nano
     ```
   - Edit the file:
     ```
     nano example_text_completion.py
     ```
   - Find and modify the prompts at the bottom.
   - Re-run the python file to get new responses.

Thank you for using Vast.ai to run the latest open source LLM, Mixtral 8X7B. Drop us a line and let us know what you think of the model.


Running the latest open source model from Mistral AI called Mixtral 8x7b on Vast.ai using 8X 4090s or 8X 3090s

How to run Mixtral 8X7B


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support! 

With this latest update, we're welcoming both the Radeon and Instinct product lines from AMD to the existing array of NVIDIA machines available for rent on our marketplace. As we widen the range of supported hardware, we can help connect even more customers with the specific GPU compute requirements they're looking for. 

Here are the [AMD GPUs we have available now](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76).

## Now Supporting AMD Instinct and Radeon GPUs

The Instinct line is AMD's brand of data center GPUs, intended to accelerate deep learning, artificial neural network, and high-performance computing tasks, while the Radeon line is aimed at gaming and consumer-grade applications. With both now supported on Vast, this expansion means more versatility and options for all of our users.

Travis Cannell, COO at Vast.ai, highlights the strategic advantage of the update: "We are interested in partnering with managed service providers who have the Instinct line and who want to increase the utilization of their clusters, as well as the Radeon line for as our users have needs for remote gaming, rendering and more." 

Ultimately, our platform will now be better equipped to meet the diverse needs and preferences of our user base – from data centers to gamers and everyone in between. Whether you're scaling up operations or just starting out, we're here to support your computing journey.

## How to Get Started

To browse the active AMD listings on cloud.vast.ai, you will need to select an AMD template which will then filter results for AMD GPUs. Renters looking to leverage these new GPU options on Vast should use the ROCm template. The two templates supported now include an Ubuntu 22.04 image and a PyTorch image.

To see the active results, select the desired template:

- [AMD Ubuntu 22.04 ROCm 6.02](https://cloud.vast.ai/?ref_id=43484&template_id=9e190d88664e20818ee75eb33dce943c)
- [AMD ROCm PyTorch](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76)

You can also find the latest AMD templates by looking for AMD on the [template page](https://cloud.vast.ai/templates/).

![Vast.ai AMD Templates](/uploads/amd-templates.PNG)


## How to Host AMD Equipment

For those interested in hosting their GPUs, Vast.ai provides comprehensive instructions in our [Machine Hosting Setup guide](https://cloud.vast.ai/host/setup/). 

Whether you're setting up an AMD or NVIDIA machine, the step-by-step guide gives you all the information you need to get started. From system requirements and networking configuration to troubleshooting and how to list your machine, learn to optimize your hosting setup and get up and running as soon as possible.

## Leading the Way in GPU Rental Innovation

The addition of support for AMD GPUs comes at a time when Vast.ai has announced an incredible 265% year-over-year growth since 2019. This growth illustrates the heavy demand in the GPU rental market. With more than 350 independent hosts supplying 16,000+ GPUs – and growing – Vast.ai efficiently matches GPU supply and demand, ensuring that every user finds the best deal for their specific compute needs.

As Vast.ai leads the charge in GPU rental innovation, the addition of AMD support is more than just an upgrade – it's also a reflection of our commitment to the democratization of AI. Join us as we continue to make powerful computing more accessible to everyone.


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support!

Announcing AMD Support


## Background
vLLM is an open source framework for Large Language model inference. It specifically focuses on throughput for serving and batch workloads. This is important for building apps for multiple users and at scale.

vLLM provides an OpenAI compatible server, which means that you can integrate it into chatbots, and other applications

As companies build out their AI products, they often hit roadblocks like rate limits and cost for using these models. With vLLM on Vast, you can run your own models in the form factor you need, but with much more affordable compute. As inference grows in demand with agents and complicated workflows, vLLM on Vast shines for performance and affordability where you need it the most.


This guide will show you how to setup vLLM to serve an LLM on Vast. We reference a notebook that you can use [here](/uploads/serving_vllm_on_vast.ipynb)

## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account and add credits, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving an LLM, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU with decent RAM since we're going to serve a single small model. `vLLM` also requires Cuda version 12.1 or higher, so we will filter for that as well. We will query the vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.1' 
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 


```bash 
vastai create instance <instance-id> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 40 --args --model stabilityai/stablelm-2-zephyr-1_6b
```

## Connecting and Testing:
To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view.png)

We will copy over the IP address and the port into the cell below.
```bash
# This request assumes you haven't changed the model. If you did, fill it in the "model" value in the payload json below
curl -X POST http://<IP-Address>:<Port>/v1/completions -H "Content-Type: application/json"  -d '{"model" : "stabilityai/stablelm-2-zephyr-1_6b", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

## Advanced Usage: Serving a Quantized Llama-3-70b Model:
Now that we've spun up a model on vLLM, we can get into more complicated deployments. We'll work on serving this specific quantized Llama-3 70B [model](https://huggingface.co/casperhansen/llama-3-70b-instruct-awq).

With this quantized model, we can easilly serve this model on on 4 4090 GPU's.

### What is different this time around:
1. The model string - we need to use the new model id.
2. We're going to use 4 GPU's instead of just 1.
3. We need to provision much more space on our system to be able to download the full set of weights. 100 GB in this case should be fine.
4. We need to set up tensor parallelism inside vLLM to split up the model across these 4 gpus. 
5. We need to let vLLM know that this is a quantized model

First, we will search for instances that match our needs
``` bash
vastai search offers 'compute_cap >= 800 gpu_ram >= 24 num_gpus = 4 static_ip=true direct_port_count > 1 cuda_vers >= 12.1' 
```

In our instance creation, we will increase our disk usage to 100GB.

Then, we will tell vllm to: 1. use the specific model, 2. split across 4 GPU's, and 3. Let it know that it is in fact a quantized model.

``` bash
vastai create instance <Instance-ID> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 100 --args --model casperhansen/llama-3-70b-instruct-awq --tensor-parallel-size 4  --quantization awq 
```

### Other things to look out for with other configurations:

If you are downloading a model that needs authentication from the huggingface hub, passing `-e HF_token=<Your-Read-Only-Token>` within vast's `--env` variable string should help.

Sometimes the full context of a model can't be used given the space allocated for vLLM on the GPU + the models size. In those cases, you might want to increase `--gpu-memory-utilization`, or decrease the `max-model-len`. Increasing `--gpu-memory-utilization` does come with CUDA OutOfMemory Issues that can be hard to predict ahead of time.

We won't need either of these for this specific model and GPU configuration.

### Testing:

Copy the IP address from your instance once it is ready, and then we can use the following code to call it. Note that while your server might have ports ready, the model might not have downloaded yet as it is much larger this time. You can check the status of this via the logs to see when it has started serving.

```bash
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'casperhansen/llama-3-70b-instruct-awq',
    'prompt': 'Hello, how are you?',
    'max_tokens': 50,
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/completions', headers=headers, json=json_data)
print(response.content)
```


## Conclusions
Model inference is expensive, and leveraging more affordable compute/models makes a huge difference for engineering teams in terms of margins, and shipping velocity.

Using vLLM on Vast is perfect for this, pairing Vast's access to affordable compute with the simplicity and State of the Art throughput of the vLLM backend.

vLLM is a great beginning to building Generative AI Apps. We will continue to explore using this tool more with Vast in future posts.

Llama-3 is already ready to go on Vast to start experimenting and building!


Serving Online Inference with vLLM API on Vast.ai


Just this past year, NVIDIA introduced a groundbreaking security feature in the Hopper architecture to protect sensitive data, AI models, and applications in use. The new feature, called NVIDIA Confidential Computing, offers a hardware-based solution for securely processing data and code that is in use, preventing unauthorized access and modification. 

  

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible – while you also get to enjoy the cutting-edge acceleration of NVIDIA H200 and H100 Tensor Core GPUs.

  

### What Is NVIDIA Confidential Computing?

  

To put it simply, "confidential computing" (CC) protects data-in-use by performing computation in a trusted execution environment (TEE) that is both hardware-based and attested. 

  

The NVIDIA H200 and H100 GPUs have a CVM (confidential virtual machine) TEE on the CPU that is anchored in an on-die root of trust (RoT). When it boots in CC-On mode, hardware protections that provide confidentiality and integrity of code and data are enabled. (In this context, "confidentiality" means that an attacker cannot access any data or code, and "integrity" means that they cannot modify the execution.) 

  

Here's how it works:

  

1. A chain of trust is established via a GPU boot sequence with a secure and measured boot.
    

  

2. An SPDM (Security Protocol and Data Model) session facilitates a secure connection to the driver in a CPU TEE. 
    

  

3. An attestation report with a cryptographically signed set of measurements is generated, verifying system integrity.
    

  

In order to run a confidential workload, the GPU must be authenticated as a legitimate NVIDIA GPU that supports confidential computing (a process involving a device-unique private key fused onto each GPU corresponding to a certified public key); the GPU must not have been revoked for CC; and the GPU attestation report must be verified. 

  

Upon successful verification, the NVIDIA driver in the CVM establishes a secure channel, via a session key, with the GPU hardware TEE in order to transfer data, perform computation, and retrieve results. The CVM and GPU communicate through a shared memory region outside the CVM since the CPU prevents the GPU from directly accessing a CVM's memory. They use AES-GCM – an advanced encryption standard – to prevent the host system from reading this communication. 

  

Finally, the GPU copies and decrypts all inputs to its internal memory, where everything runs in plaintext and direct access is blocked. Performance counters are disabled as well, since these could be used for side-channel attacks. 

  

In this way, a confidential-computing environment is extended from a CVM (or a secure enclave) to a GPU, with attestation, encrypted communication, and memory isolation making it possible. 

  

### Benefits & Positive Impacts

  

The benefits of NVIDIA Confidential Computing are significant:

  

- Hardware-Based Security and Isolation – You can achieve full isolation of virtual machines (VMs) in any environment – on-premises, at the edge, or on the cloud. Your entire workload is secured with these built-in hardware firewalls that provide unprecedented protection. 
    

  

- Verifiability with Device Attestation – You will always rest assured that only authorized end users can deploy data and code for execution in the H200 or H100's TEE. Plus, through device attestation, you'll know that you're dealing with an authentic NVIDIA GPU, the firmware hasn't been modified, and updates have been done as expected. 
    

  

- Protection from Unauthorized Access – Your sensitive data, AI workloads, and intellectual property are protected, with confidentiality and integrity preserved at all times. All unauthorized entities (including the hypervisor, cloud provider, host operating system, and even anyone with physical access to the infrastructure) are blocked from viewing or modifying the AI application and data during execution. 
    

  

- No Application Code Change – As NVIDIA puts it, the new confidential computing feature "just works," without any code changes required for your GPU-accelerated workloads. 
    

  

As you can imagine, confidential computing-enabled GPUs usher in a whole host of use cases where security, privacy, and regulatory compliance are paramount. For instance, intellectual property protection is now possible for proprietary AI models, even when these models are distributed and deployed at scale on shared or remote infrastructure. 

  

Privacy and confidentiality can also now be preserved in AI training and inference – most notably in industries like healthcare, finance, and the public sector, where data may be sensitive and/or regulated. On top of that, collaboration between multiple parties can be done securely when building and improving AI models across participating sites, for use cases like medical imaging, drug development, and fraud detection. 

  

Another bonus: organizations, governments, and individuals can outsource AI workloads to a cloud provider while still ensuring protection from the underlying infrastructure. 

  

### Final Thoughts

  

It's exciting that confidential computing is now possible for even the most demanding workloads! Granted, for the vast majority of use cases, CC-enabled GPUs may arguably be overkill. But for those who need or prefer absolute confidentiality and security, NVIDIA Confidential Computing in the Hopper architecture can now provide that for you. 

  

At Vast.ai, we understand how important privacy and security can be for our customers. It is our hope that you'll find exactly what you need on our cost-effective, on-demand cloud GPU rental platform. 

  

In our global network of compute providers on Vast, some hosts are experimenting with NVIDIA Confidential Computing. Complete support for this feature is not yet available. Join our Discord for the latest product updates and up-to-date support.

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible 

Blog

How to run Mixtral 8X7B

How to Run Mixtral 8X7B

Steps:

Continue Reading: