Below are template_hash values for various autoscaler-compatible templates. These templates handle all the necessary configuration to run different images in API mode, search for suitable machines, and communicate successfully with the autoscaling server. Refer to the quickstart for a guide on using these templates.

Customizing Your Own Templates: For all templates, some variables such as HF_TOKEN must be set. You can create a new private template by modifying relevant variables in the create template command. Then, set your autogroup to use that template by specifying the new template_hash of your newly created template.

If the existing templates don't meet your needs, don't hesitate to contact us, and we’ll assist you in creating an autoscaler-compatible template.

TGI #

The text-generation-inference image is used for serving LLMs.

Required Variables: #

• HF_TOKEN: your Hugging Face API token with read permissions, used to download gated models
• MODEL_ID: ID of model you want to use for inference. See here for available options

Endpoints #

1. /generate: Generates the LLM's response to a given prompt in a single request.
2. /generate_stream: Streams the LLM's response token by token.

Both endpoint take the same API payload:

Note: The max_new_tokens parameter, rather than the prompt size, impacts performance. For example, if an instance is benchmarked to process 100 tokens per second, a request with max_new_tokens = 200 will take approximately 2 seconds to complete.

template_hash: dcd0920ffd9d026b7bb2d42f0d7479ba

template link

Comfy UI #

The ai-dock's Comfy UI template is used as the base image for running the pyworker server for image generation.

Required Variables: #

• HF_TOKEN: your Hugging Face API token with read permissions, used to download gated models
• COMFY_MODEL: Text2Image model to be used, options: sd3, flux

template_hash: f8aac491656a1d1040e7dfc5f8fcf059

template link

Endpoints #

1. /prompt: Uses the default comfy workflow defined under workers/comfyui/misc/default_workflows

   • payload example:

   {
     "prompt": "a fluffy cat",
     "width": 1024,
     "height": 1024,
     "steps": 28,
     "seed": 123456789
   }
   

2. /custom_workflow: Allows the client to send their own comfy workflow with each API request.

   

   If you want to use a custom comfy workflow, you should create a regular Comfy UI instance, create your workflow using the UI, then enable dev mode in Comfy UI's settings, and export your workflow in the API format

   • payload example:

   {
     // values in custom_fields should match their respective values in the workflow.
     // these values are used to estimate workload
     "custom_fields": {
       "width": 1024,
       "height": 1024,
       "steps": 28
     },
     "workflow": {
       "3": {
         "inputs": {
           "seed": 156680208700286,
           "steps": 20,
           "cfg": 8,
           "sampler_name": "euler",
           "scheduler": "normal",
           "denoise": 1,
           "model": ["4", 0],
           "positive": ["6", 0],
           "negative": ["7", 0],
           "latent_image": ["5", 0]
         },
         "class_type": "KSampler"
       },
       "4": {
         "inputs": {
           "ckpt_name": "sd3_medium_incl_clips_t5xxlfp16.safetensors"
         },
         "class_type": "CheckpointLoaderSimple"
       },
       "5": {
         "inputs": { "width": 512, "height": 512, "batch_size": 1 },
         "class_type": "EmptyLatentImage"
       },
       "6": {
         "inputs": {
           "text": "beautiful scenery nature glass bottle landscape, purple galaxy bottle",
           "clip": ["4", 1]
         },
         "class_type": "CLIPTextEncode"
       },
       "7": {
         "inputs": { "text": "text, watermark", "clip": ["4", 1] },
         "class_type": "CLIPTextEncode"
       },
       "8": {
         "inputs": { "samples": ["3", 0], "vae": ["4", 2] },
         "class_type": "VAEDecode"
       },
       "9": {
         "inputs": { "filename_prefix": "ComfyUI", "images": ["8", 0] },
         "class_type": "SaveImage"
       }
     }
   }