
When you need high-powered resources to power complex calculations, the more you have the better. In a recent interview, Elon Musk claimed that he heard of a gigawatt class data center being built. Many responders think that Elon is referring to a rumor that –deep in the deserts of Kuwait–, the construction of a new data center is being built to compete with today’s most powerful compute power resources.

Elon Musk likely heard the rumor from [news](https://www.theinformation.com/articles/ai-agenda-the-mysterious-ai-data-center-startup-hiring-from-aws-azure-meta) of a startup poaching technology workers from AWS, Azure, Meta, and other large Silicon Valley companies. In 2023, the race to be a leader in artificial intelligence and machine learning became even more competitive. To be the best, organizations need immense GPU power, usually outsourced to the cloud for its quick deployment and scaling. Large cloud providers built the data centers to power calculations, and businesses can rent the compute resources necessary to power any project – from small personal projects to large enterprise-tier analytics.

The rumor of a Kuwait data center came in the summer of 2023. A company named Omniva –also known as Moneta Tech, and Moneta Systems– was interested in building the most powerful data center in the world. Their goal was to rival Google and Microsoft in technology. Originally, the data center idea was targeted towards cryptomining, but later became an idea to target businesses innovating for the latest artificial intelligence boom.

As more information became available, the gigawatt claims began to unravel. The facility was said to be about 430,000 square feet with total available power of 864 megawatts. A proposed budget of $200 million immediately sent red flags. As a benchmark, it’s reported that a Google data center costs between $1000 and $3000 per square foot. This means that at the very cheapest, a data center of Omniva’s size should cost $430 million, over double the proposed budget.

Cooling systems were ignored for manned spaces in exchange for employee cooling helmets. Fire systems were removed in exchange for fire extinguishers. Anyone with data center experience knows that the proposed planning and safety aren’t feasible.

Next is Omniva’s claim to offer gigawatts of power to customers. NVIDIA continues to dominate GPU performance and offer the best in compute power. To reach the compute power rumored to be available, Omniva would need the latest generation of GPUs, namely the new NVIDIA Blackwell B100. The B100s haven’t hit the market yet, but NVIDIA teased consumers with the promise that their next generation GPUs should launch in 2024.

As of today, any facility rumored to be built with gigawatt power has never been confirmed. The power promised in 2023 wasn’t available, and many of the plans presented to the public didn’t have the backing of professionals. The cloud GPU space still has the computational power to support artificial intelligence and cryptomining, but businesses and consumers should stick to a trusted provider.

For the absolute best in compute power with the latest in GPU technology, [Vast.ai](https://vast.ai/) has the resources you need for machine learning, artificial intelligence, and other high-powered analytics projects for your cloud GPU needs.


Coming up with just the right words, and keeping track of the ideas that you’re weaving together - this is difficult work. The difficulty can even make it seem that there might be better ways to be spending your time.

Is a Gigawatt Class Data Center Being Built in Kuwait?


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support! 

With this latest update, we're welcoming both the Radeon and Instinct product lines from AMD to the existing array of NVIDIA machines available for rent on our marketplace. As we widen the range of supported hardware, we can help connect even more customers with the specific GPU compute requirements they're looking for. 

Here are the [AMD GPUs we have available now](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76).

## Now Supporting AMD Instinct and Radeon GPUs

The Instinct line is AMD's brand of data center GPUs, intended to accelerate deep learning, artificial neural network, and high-performance computing tasks, while the Radeon line is aimed at gaming and consumer-grade applications. With both now supported on Vast, this expansion means more versatility and options for all of our users.

Travis Cannell, COO at Vast.ai, highlights the strategic advantage of the update: "We are interested in partnering with managed service providers who have the Instinct line and who want to increase the utilization of their clusters, as well as the Radeon line for as our users have needs for remote gaming, rendering and more." 

Ultimately, our platform will now be better equipped to meet the diverse needs and preferences of our user base – from data centers to gamers and everyone in between. Whether you're scaling up operations or just starting out, we're here to support your computing journey.

## How to Get Started

To browse the active AMD listings on cloud.vast.ai, you will need to select an AMD template which will then filter results for AMD GPUs. Renters looking to leverage these new GPU options on Vast should use the ROCm template. The two templates supported now include an Ubuntu 22.04 image and a PyTorch image.

To see the active results, select the desired template:

- [AMD Ubuntu 22.04 ROCm 6.02](https://cloud.vast.ai/?ref_id=43484&template_id=9e190d88664e20818ee75eb33dce943c)
- [AMD ROCm PyTorch](https://cloud.vast.ai/?ref_id=43484&template_id=a5ee2118a3dbb11155cb391ed9750a76)

You can also find the latest AMD templates by looking for AMD on the [template page](https://cloud.vast.ai/templates/).

![Vast.ai AMD Templates](/uploads/amd-templates.PNG)


## How to Host AMD Equipment

For those interested in hosting their GPUs, Vast.ai provides comprehensive instructions in our [Machine Hosting Setup guide](https://cloud.vast.ai/host/setup/). 

Whether you're setting up an AMD or NVIDIA machine, the step-by-step guide gives you all the information you need to get started. From system requirements and networking configuration to troubleshooting and how to list your machine, learn to optimize your hosting setup and get up and running as soon as possible.

## Leading the Way in GPU Rental Innovation

The addition of support for AMD GPUs comes at a time when Vast.ai has announced an incredible 265% year-over-year growth since 2019. This growth illustrates the heavy demand in the GPU rental market. With more than 350 independent hosts supplying 16,000+ GPUs – and growing – Vast.ai efficiently matches GPU supply and demand, ensuring that every user finds the best deal for their specific compute needs.

As Vast.ai leads the charge in GPU rental innovation, the addition of AMD support is more than just an upgrade – it's also a reflection of our commitment to the democratization of AI. Join us as we continue to make powerful computing more accessible to everyone.


As the world's first marketplace for cloud GPU rental, Vast.ai is excited to announce another first: we're expanding our offerings to include AMD support!

Announcing AMD Support


## Background
vLLM is an open source framework for Large Language model inference. It specifically focuses on throughput for serving and batch workloads. This is important for building apps for multiple users and at scale.

vLLM provides an OpenAI compatible server, which means that you can integrate it into chatbots, and other applications

As companies build out their AI products, they often hit roadblocks like rate limits and cost for using these models. With vLLM on Vast, you can run your own models in the form factor you need, but with much more affordable compute. As inference grows in demand with agents and complicated workflows, vLLM on Vast shines for performance and affordability where you need it the most.


This guide will show you how to setup vLLM to serve an LLM on Vast. We reference a notebook that you can use [here](/uploads/serving_vllm_on_vast.ipynb)

## Setup and Querying
First, we setup our environment and vast api key

```bash
pip install --upgrade vastai
```

Once you create your account and add credits, you can go [here](https://cloud.vast.ai/cli/) to find your API Key.

```bash
vastai set api-key <Your-API-Key-Here>
```

For serving an LLM, we're looking for a machine that has a static IP address, ports available to host on, plus a single modern GPU with decent RAM since we're going to serve a single small model. `vLLM` also requires Cuda version 12.1 or higher, so we will filter for that as well. We will query the vast API to get a list of these types of machines.

```bash
vastai search offers 'compute_cap > 800 gpu_ram > 20 num_gpus = 1 static_ip=true direct_port_count > 1 cuda_vers >= 12.1' 
```

## Deploying the Image:
The easiest way to deploy this instance is to use the command line. Copy and Paste a specific instance id you choose from the list above into `instance-id` below. 


```bash 
vastai create instance <instance-id> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 40 --args --model stabilityai/stablelm-2-zephyr-1_6b
```

## Connecting and Testing:
To connect to your instance, we'll first need to get the IP address and port number. Once your instance is done setting up, you should see something like this:
![Instance_view](/uploads/instance_view.png)

Click on the highlighted button to see the IP address and correct port for our requests.

![IP_address_view](/uploads/ip_address_view.png)

We will copy over the IP address and the port into the cell below.
```bash
# This request assumes you haven't changed the model. If you did, fill it in the "model" value in the payload json below
curl -X POST http://<IP-Address>:<Port>/v1/completions -H "Content-Type: application/json"  -d '{"model" : "stabilityai/stablelm-2-zephyr-1_6b", "prompt": "Hello, how are you?", "max_tokens": 50}'
```

You will see a response from your model in the output. Your model is up and running on Vast!

## Advanced Usage: Serving a Quantized Llama-3-70b Model:
Now that we've spun up a model on vLLM, we can get into more complicated deployments. We'll work on serving this specific quantized Llama-3 70B [model](https://huggingface.co/casperhansen/llama-3-70b-instruct-awq).

With this quantized model, we can easilly serve this model on on 4 4090 GPU's.

### What is different this time around:
1. The model string - we need to use the new model id.
2. We're going to use 4 GPU's instead of just 1.
3. We need to provision much more space on our system to be able to download the full set of weights. 100 GB in this case should be fine.
4. We need to set up tensor parallelism inside vLLM to split up the model across these 4 gpus. 
5. We need to let vLLM know that this is a quantized model

First, we will search for instances that match our needs
``` bash
vastai search offers 'compute_cap >= 800 gpu_ram >= 24 num_gpus = 4 static_ip=true direct_port_count > 1 cuda_vers >= 12.1' 
```

In our instance creation, we will increase our disk usage to 100GB.

Then, we will tell vllm to: 1. use the specific model, 2. split across 4 GPU's, and 3. Let it know that it is in fact a quantized model.

``` bash
vastai create instance <Instance-ID> --image vllm/vllm-openai:latest --env '-p 8000:8000' --disk 100 --args --model casperhansen/llama-3-70b-instruct-awq --tensor-parallel-size 4  --quantization awq 
```

### Other things to look out for with other configurations:

If you are downloading a model that needs authentication from the huggingface hub, passing `-e HF_token=<Your-Read-Only-Token>` within vast's `--env` variable string should help.

Sometimes the full context of a model can't be used given the space allocated for vLLM on the GPU + the models size. In those cases, you might want to increase `--gpu-memory-utilization`, or decrease the `max-model-len`. Increasing `--gpu-memory-utilization` does come with CUDA OutOfMemory Issues that can be hard to predict ahead of time.

We won't need either of these for this specific model and GPU configuration.

### Testing:

Copy the IP address from your instance once it is ready, and then we can use the following code to call it. Note that while your server might have ports ready, the model might not have downloaded yet as it is much larger this time. You can check the status of this via the logs to see when it has started serving.

```bash
import requests

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'model': 'casperhansen/llama-3-70b-instruct-awq',
    'prompt': 'Hello, how are you?',
    'max_tokens': 50,
}

response = requests.post('http://<Instance-IP-Address>:<Port>/v1/completions', headers=headers, json=json_data)
print(response.content)
```


## Conclusions
Model inference is expensive, and leveraging more affordable compute/models makes a huge difference for engineering teams in terms of margins, and shipping velocity.

Using vLLM on Vast is perfect for this, pairing Vast's access to affordable compute with the simplicity and State of the Art throughput of the vLLM backend.

vLLM is a great beginning to building Generative AI Apps. We will continue to explore using this tool more with Vast in future posts.

Llama-3 is already ready to go on Vast to start experimenting and building!


Serving Online Inference with vLLM API on Vast.ai


Just this past year, NVIDIA introduced a groundbreaking security feature in the Hopper architecture to protect sensitive data, AI models, and applications in use. The new feature, called NVIDIA Confidential Computing, offers a hardware-based solution for securely processing data and code that is in use, preventing unauthorized access and modification. 

  

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible – while you also get to enjoy the cutting-edge acceleration of NVIDIA H200 and H100 Tensor Core GPUs.

  

### What Is NVIDIA Confidential Computing?

  

To put it simply, "confidential computing" (CC) protects data-in-use by performing computation in a trusted execution environment (TEE) that is both hardware-based and attested. 

  

The NVIDIA H200 and H100 GPUs have a CVM (confidential virtual machine) TEE on the CPU that is anchored in an on-die root of trust (RoT). When it boots in CC-On mode, hardware protections that provide confidentiality and integrity of code and data are enabled. (In this context, "confidentiality" means that an attacker cannot access any data or code, and "integrity" means that they cannot modify the execution.) 

  

Here's how it works:

  

1. A chain of trust is established via a GPU boot sequence with a secure and measured boot.
    

  

2. An SPDM (Security Protocol and Data Model) session facilitates a secure connection to the driver in a CPU TEE. 
    

  

3. An attestation report with a cryptographically signed set of measurements is generated, verifying system integrity.
    

  

In order to run a confidential workload, the GPU must be authenticated as a legitimate NVIDIA GPU that supports confidential computing (a process involving a device-unique private key fused onto each GPU corresponding to a certified public key); the GPU must not have been revoked for CC; and the GPU attestation report must be verified. 

  

Upon successful verification, the NVIDIA driver in the CVM establishes a secure channel, via a session key, with the GPU hardware TEE in order to transfer data, perform computation, and retrieve results. The CVM and GPU communicate through a shared memory region outside the CVM since the CPU prevents the GPU from directly accessing a CVM's memory. They use AES-GCM – an advanced encryption standard – to prevent the host system from reading this communication. 

  

Finally, the GPU copies and decrypts all inputs to its internal memory, where everything runs in plaintext and direct access is blocked. Performance counters are disabled as well, since these could be used for side-channel attacks. 

  

In this way, a confidential-computing environment is extended from a CVM (or a secure enclave) to a GPU, with attestation, encrypted communication, and memory isolation making it possible. 

  

### Benefits & Positive Impacts

  

The benefits of NVIDIA Confidential Computing are significant:

  

- Hardware-Based Security and Isolation – You can achieve full isolation of virtual machines (VMs) in any environment – on-premises, at the edge, or on the cloud. Your entire workload is secured with these built-in hardware firewalls that provide unprecedented protection. 
    

  

- Verifiability with Device Attestation – You will always rest assured that only authorized end users can deploy data and code for execution in the H200 or H100's TEE. Plus, through device attestation, you'll know that you're dealing with an authentic NVIDIA GPU, the firmware hasn't been modified, and updates have been done as expected. 
    

  

- Protection from Unauthorized Access – Your sensitive data, AI workloads, and intellectual property are protected, with confidentiality and integrity preserved at all times. All unauthorized entities (including the hypervisor, cloud provider, host operating system, and even anyone with physical access to the infrastructure) are blocked from viewing or modifying the AI application and data during execution. 
    

  

- No Application Code Change – As NVIDIA puts it, the new confidential computing feature "just works," without any code changes required for your GPU-accelerated workloads. 
    

  

As you can imagine, confidential computing-enabled GPUs usher in a whole host of use cases where security, privacy, and regulatory compliance are paramount. For instance, intellectual property protection is now possible for proprietary AI models, even when these models are distributed and deployed at scale on shared or remote infrastructure. 

  

Privacy and confidentiality can also now be preserved in AI training and inference – most notably in industries like healthcare, finance, and the public sector, where data may be sensitive and/or regulated. On top of that, collaboration between multiple parties can be done securely when building and improving AI models across participating sites, for use cases like medical imaging, drug development, and fraud detection. 

  

Another bonus: organizations, governments, and individuals can outsource AI workloads to a cloud provider while still ensuring protection from the underlying infrastructure. 

  

### Final Thoughts

  

It's exciting that confidential computing is now possible for even the most demanding workloads! Granted, for the vast majority of use cases, CC-enabled GPUs may arguably be overkill. But for those who need or prefer absolute confidentiality and security, NVIDIA Confidential Computing in the Hopper architecture can now provide that for you. 

  

At Vast.ai, we understand how important privacy and security can be for our customers. It is our hope that you'll find exactly what you need on our cost-effective, on-demand cloud GPU rental platform. 

  

In our global network of compute providers on Vast, some hosts are experimenting with NVIDIA Confidential Computing. Complete support for this feature is not yet available. Join our Discord for the latest product updates and up-to-date support.

Whether deployed on-premises, at the edge, or on the cloud, your sensitive data can be vulnerable to both external attacks and internal threats. NVIDIA Confidential Computing gives you peace of mind knowing that the data and code you most want to protect are as secure as possible 

Blog

Is a Gigawatt Class Data Center Being Built in Kuwait?

Continue Reading: