Warning
The Autoscaler is currently in Beta and may experience changes, quirks, and downtime.
Worker instances run the Vast PyWorker code, which is configured to support several backends. The currently supported backends are:
Each backend has its own launch script and expected environment variables. We provide custom templates that configure everything needed to set up an autogroup for these backends, available here. To modify default parameters, refer to the README files for each backend in the Vast PyWorker repository:
After launch, these images run inference code in a separate process accessible as an API server. The Vast PyWorker backend code wraps HTTP calls to this inference process, meaning each backend can support any endpoint that the underlying inference API server supports. However, each endpoint requires defining a wrapper function in the appropriate backend.py file.
To add an endpoint to an existing backend, follow the instructions in the PyWorker Extension Guide. This guide's architecture can also be used to write new backends, where the underlying inference code is accessible through an HTTP server in its own process.
You can see the currently supported endpoints for each backend:
To support additional endpoints, follow the guide and fork the repository to add your own code. These wrapper endpoints have the same name, input arguments, and output format as the underlying endpoint.
The authentication information returned by https://run.vast.ai/route/ must be included in the request dictionary to the endpoint but will be filtered out before forwarding to the backend server. When Vast’s autoscaling server returns a server address from the /route/ endpoint, it provides a unique signature with your request. The authentication server verifies this signature to ensure that only authorized clients can send requests to your server.
For more detailed information and advanced configuration, please visit the Vast PyWorker repository.