GuidesAPIDiscord

Docs - Autoscaler

Architecture

Warning: The Autoscaler is currently in Beta, and is subject to changes, 'quirks', and downtime.

The Autoscaler manages groups of instances, called autogroups, to serve an application and scale up or down according to load (as defined via the vast-pyworker metrics code). The autoscaler and accompanying GPU worker code automates instance management (creation,deletion,stopping,starting,etc), performance/capacity measurement, and error handling.

The general system architecture for a complete application using Vast.ai Autoscaling has a few main components:

  1. the autoscaler (Vast.ai)
  2. the loadbalancer (Vast.ai)
  3. gpu worker code (We provide a framework and examples, you customize/replace as needed)
  4. application website (yours)

An example workflow (for a consumer LLM app), works like this:

  1. A customer initiates a request through your website
  2. Your website makes a call to https://run.vast.ai/route/ with your endpoint, apikey, any optional params (cost)
  3. The /route/ endpoint returns a suitable worker address
  4. Your website makes a call to your GPU worker's specific API endpoint, like {worker_address}/generate (or /generate-stream/ etc), passing the info returned by /route, along with any request params (prompt, etc)
  5. Your website then returns the results to the client browser (or does whatever else you want)

We currently use this 2-step routing process for simplicity and flexibility. It also has the additional benefit that you don't need to route all your prompts (or other user data) through any central server provided by vast.ai, as our load-balancer doesn't need those details to route your request.

The /route/ endpoint signs the messages it returns with its public key (available at https://run.vast.ai/pubkey/ ) so the GPU worker then can check this signature to validate requests (to prevent others from using your workers).

We likely will add an optional proxy service in the future to optionally reduce this 2-step process to a single step, but for now its a 2-step process (Since you probably need to proxy customer traffic anyway, the 2-step method may reduce latency)

A full example walkthrough for LLM inference using HuggingFace TGI for the worker backend is available in this guide