
Qwen 3.5 is Alibaba's latest model family, featuring a hybrid architecture that combines Gated DeltaNet (linear attention) with standard attention in a 3:1 ratio. This design enables dramatically faster inference — up to 8.6x at 32K context and 19x at 256K — while maintaining strong reasoning capabilities. The medium models released February 24, 2026 include:

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Total Params</th>
      <th>Active Params</th>
      <th>Architecture</th>
      <th>VRAM (BF16)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Qwen3.5-122B-A10B</td>
      <td>122B</td>
      <td>10B</td>
      <td>MoE</td>
      <td>~244 GB</td>
    </tr>
    <tr>
      <td>Qwen3.5-35B-A3B</td>
      <td>35B</td>
      <td>3B</td>
      <td>MoE</td>
      <td>~66 GB</td>
    </tr>
    <tr>
      <td>Qwen3.5-27B</td>
      <td>27B</td>
      <td>27B</td>
      <td>Dense</td>
      <td>~54 GB</td>
    </tr>
  </tbody>
</table>

The MoE models route each token through 8 of 256 experts plus 1 shared expert, so only a fraction of the total parameters are active per token. The 35B-A3B model activates just 3B parameters per token while having 35B total — giving you large-model quality at small-model inference cost. All three models are Apache 2.0 licensed, so no HuggingFace token is needed.

In this guide, we'll deploy Qwen3.5-35B-A3B on Vast.ai using SGLang, which has day-0 Qwen3.5 support as of v0.5.9, on a single A100 80GB GPU.

## Deploying on Vast.ai

Install the Vast.ai CLI and set your API key:

```bash
pip install --upgrade vastai
export VAST_API_KEY="YOUR_KEY_HERE"
vastai set api-key $VAST_API_KEY
```

Search for a suitable GPU. The model needs ~66 GB VRAM in BF16, so an 80 GB card works:

```bash
vastai search offers "gpu_ram>=80 num_gpus=1 direct_port_count>=1 rentable=true disk_space>=200 cuda_vers>=12.2" -o dph
```

Deploy the model using SGLang's Docker image. Replace `YOUR_INSTANCE_ID` with an offer ID from the search results:

```bash
vastai create instance YOUR_INSTANCE_ID \
    --image lmsysorg/sglang:latest \
    --env '-p 8000:8000' \
    --disk 200 \
    --onstart-cmd "python3 -m sglang.launch_server \
        --model-path Qwen/Qwen3.5-35B-A3B \
        --host 0.0.0.0 --port 8000 \
        --tp-size 1 \
        --context-length 32768 \
        --reasoning-parser qwen3 \
        --mem-fraction-static 0.85"
```

Key flags:
- `--reasoning-parser qwen3` enables the model's built-in thinking mode, separating reasoning from the final answer
- `--mem-fraction-static 0.85` allocates 85% of GPU memory for KV cache
- `--context-length 32768` sets a 32K context window (the model supports up to 262K natively)

The model will download (~70 GB) and load automatically. Check `vastai logs YOUR_INSTANCE_ID` — you'll see "The server is fired up and ready to roll!" when it's serving.

## Calling the Model

Find your instance's IP and port in the [Instances tab](https://cloud.vast.ai/instances/) of the Vast.ai console. Click the IP address button to see the port mapping for 8000/tcp.

Install the OpenAI SDK:

```bash
pip install --upgrade openai
```

Then call the model:

```python
from openai import OpenAI

VAST_IP_ADDRESS = "YOUR_IP"
VAST_PORT = "YOUR_PORT"

client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B",
    messages=[
        {"role": "user", "content": "What are three benefits of mixture-of-experts models? Be concise."}
    ],
    max_tokens=1024,
    temperature=0.7
)

print(response.choices[0].message.content)
```

### Response

```
1. **Computational Efficiency:** Only a subset of parameters is activated per input,
   reducing inference costs while maintaining a massive total parameter count.
2. **Enhanced Performance:** The larger effective model capacity allows for better
   accuracy on complex and diverse tasks compared to dense models of similar compute.
3. **Specialization:** Individual experts can learn to handle specific input patterns
   or domains, improving the model's robustness and adaptability.
```

With `--reasoning-parser qwen3`, the model's thinking process is automatically separated from the final answer. The thinking tokens appear in `reasoning_content` in the API response, while the clean answer appears in `content`.

## Other Models

You can swap `Qwen/Qwen3.5-35B-A3B` for the other medium models:

- **Qwen3.5-122B-A10B**: Larger MoE model, 122B total / 10B active. At BF16 the full weights need ~244 GB, so you'll need multiple GPUs (e.g., 4× A100 80GB with `--tp-size 4`). Alternatively, GGUF quantizations are available — the Q4_K_M variant is ~70 GB and fits on a single 80GB card.
- **Qwen3.5-27B**: Dense model, all 27B parameters active. Needs ~54 GB VRAM. Fits on a single A100 or H100.

## Conclusion

Qwen3.5-35B-A3B packs 35B parameters into a model that only activates 3B per token, and its hybrid DeltaNet architecture delivers substantially faster inference than standard transformers. With SGLang and a single A100 on Vast.ai, you can have it serving in under 10 minutes.

For production use, consider increasing `--context-length` — the model supports up to 262K natively, with 128K+ recommended for best reasoning performance. GGUF quantizations (Q4_K_M at ~22 GB) are also available for running on consumer GPUs like the RTX 4090.

## Useful Resources

- [Qwen3.5-35B-A3B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)
- [Qwen3.5-122B-A10B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)
- [Qwen3.5-27B on Hugging Face](https://huggingface.co/Qwen/Qwen3.5-27B)
- [SGLang Documentation](https://docs.sglang.ai/)
- [Vast.ai Documentation](https://docs.vast.ai/)


Deploy Qwen 3.5 medium models on Vast.ai using SGLang, including setup, launch commands, and API usage.

Running Qwen 3.5 Medium Models on Vast.ai


Liquid AI just released [LFM2-24B-A2B](https://huggingface.co/LiquidAI/LFM2-24B-A2B), a 24 billion parameter foundation model that only activates 2.3 billion parameters per token. That ratio — less than 10% of total parameters active at inference time — makes it one of the most aggressively sparse mixture-of-experts models available today.

What makes LFM2 different from other MoE models isn't just the sparsity. The architecture is a hybrid design that replaces most of the attention layers with gated short convolution blocks. Out of 40 total layers, only 10 use grouped query attention — the other 30 are convolutional. This is a fundamentally different approach from transformer-only models, and it's what allows Liquid AI to push the active parameter count so low while maintaining competitive quality across benchmarks like MMLU-Pro, GPQA Diamond, IFEval, and MATH-500.

Liquid AI, an MIT spinoff, has scaled this architecture from 350M to 24B parameters following log-linear scaling laws. LFM2 is the first model in the family large enough to be broadly useful — and with day-one support for vLLM, SGLang, and llama.cpp, it's ready to deploy today.

> **Note**: LFM2-24B-A2B is an early checkpoint trained on 17 trillion tokens without reinforcement learning or extensive post-training. A post-trained version (LFM2.5-24B-A2B) is coming.

## Model Overview

<table>
  <thead>
    <tr>
      <th>Property</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Developer</td>
      <td><a href="https://www.liquid.ai">Liquid AI</a></td>
    </tr>
    <tr>
      <td>Model</td>
      <td>LFM2-24B-A2B</td>
    </tr>
    <tr>
      <td>Architecture</td>
      <td>Hybrid MoE: 30 convolutional + 10 GQA attention layers, 64 experts, top-4 routing</td>
    </tr>
    <tr>
      <td>Total Parameters</td>
      <td>24B</td>
    </tr>
    <tr>
      <td>Active Parameters</td>
      <td>2.3B per token</td>
    </tr>
    <tr>
      <td>Context Length</td>
      <td>32,768 tokens</td>
    </tr>
    <tr>
      <td>Training Data</td>
      <td>17T tokens (mixed BF16/FP8)</td>
    </tr>
    <tr>
      <td>License</td>
      <td><a href="https://www.liquid.ai/lfm-open-license-v1-0">LFM Open License v1.0</a></td>
    </tr>
    <tr>
      <td>HuggingFace</td>
      <td><a href="https://huggingface.co/LiquidAI/LFM2-24B-A2B">LiquidAI/LFM2-24B-A2B</a></td>
    </tr>
  </tbody>
</table>

## Deploy LFM2-24B-A2B on Vast.ai with vLLM

At BF16, the model weights are ~48 GB — it fits on a single A100-80GB with room for a 32K context KV cache. No quantization or multi-GPU setup needed.

### Prerequisites

- A [Vast.ai](https://cloud.vast.ai) account with credits
- The Vast CLI installed and configured:

```bash
pip install --upgrade vastai  # make sure you're on v0.5.0+
vastai set api-key <YOUR_API_KEY>
```

### Find a GPU

Search for a single GPU with at least 80 GB VRAM and direct port access:

```bash
vastai search offers "gpu_ram>=80 num_gpus=1 direct_port_count>=1 cuda_vers>=12.4 cuda_vers<13" -o dph
```

> **Important**: Use a machine with **CUDA 12.x** (not 13.x). The vLLM nightly image is built against CUDA 12 and will fail with a driver mismatch on CUDA 13 machines.

### Launch the Instance

Pick an instance ID from the search results and deploy with vLLM. LFM2 requires a newer version of the `transformers` library than what ships in the vLLM image, so we upgrade it at startup:

```bash
vastai create instance <INSTANCE_ID> \
  --image vllm/vllm-openai:nightly \
  --env "-p 8000:8000" \
  --disk 100 \
  --onstart-cmd "pip install 'transformers>=5.0.0' && \
    vllm serve LiquidAI/LFM2-24B-A2B \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 32768 --dtype auto \
    --trust-remote-code"
```

This upgrades the tokenizer library, pulls the model from HuggingFace, and starts an OpenAI-compatible API server on port 8000. Initial startup takes several minutes while the model downloads (~48 GB for the BF16 weights).

Once the instance is running, find your IP and port in the [Vast console](https://cloud.vast.ai/instances/) — click the IP button on your instance to see the port mapping.

> **Why `nightly` and `transformers>=5`?** LFM2's tokenizer uses the `TokenizersBackend` class introduced in `transformers 5.0`. The stable vLLM image pins `transformers<5`, so the tokenizer fails to load. Using `nightly` with a transformers upgrade resolves this. This was tested with vLLM nightly build `v0.16.0rc2`. Once a stable vLLM release ships with `transformers>=5.0.0` support, you can switch to `vllm/vllm-openai:latest` and drop the `pip install` step.

## Call the API

```bash
curl http://<VAST_IP>:<PORT>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "LiquidAI/LFM2-24B-A2B",
    "messages": [{"role": "user", "content": "What makes mixture-of-experts models efficient?"}],
    "max_tokens": 300
  }'
```

## Cleanup

When you're done, destroy the instance to stop charges:

```bash
vastai destroy instance <INSTANCE_ID>
```

## Resources

- [LFM2-24B-A2B on HuggingFace](https://huggingface.co/LiquidAI/LFM2-24B-A2B)
- [LFM2-24B-A2B GGUF Quantizations](https://huggingface.co/LiquidAI/LFM2-24B-A2B-GGUF)
- [Liquid AI Blog: LFM2-24B-A2B](https://www.liquid.ai/blog/lfm2-24b-a2b)
- [Liquid AI Models](https://www.liquid.ai/models)
- [Vast.ai Documentation](https://docs.vast.ai/)
- [vLLM Documentation](https://docs.vllm.ai/)


Liquid AI released LFM2-24B-A2B, a sparse 24B MoE model with 2.3B active parameters per token.

Liquid AI's LFM2 Just Dropped — Here's How to Run It on Vast.ai


dstack is an open-source GPU orchestration platform that automates instance provisioning and lifecycle management across cloud providers. This guide walks through using dstack with Vast.ai as the backend, combining declarative infrastructure with competitive GPU marketplace pricing.

## What Makes dstack Special?

Key features include:
- **Infrastructure as Code**: Define GPU requirements, pricing limits, and workloads in YAML
- **Automatic Provisioning**: dstack finds and provisions the best available instances
- **Cost Controls**: Set `max_price` to cap hourly costs automatically
- **Built-in Proxy**: Access services through dstack's authenticated endpoint

## What's in the New Guide

Our latest documentation walks you through deploying language models with dstack and vLLM on Vast.ai.

**Complete Setup Instructions**
- Installing and configuring dstack with your Vast.ai API key
- Starting the dstack server and CLI
- Creating service configurations for vLLM deployments

**Working Service Configurations**
- Ready-to-use YAML for deploying Qwen3-30B-A3B
- GPU memory and pricing parameters
- Real deployment outputs showing what to expect

**API Integration Examples**
- Python code using the OpenAI SDK
- cURL examples for testing
- Streaming response implementations

## Why dstack + Vast.ai?

The combination gives you orchestration on top of a GPU marketplace:

- **Simplified Workflow**: No manual instance hunting or environment setup
- **Cost Optimization**: dstack finds the cheapest instance meeting your requirements
- **Flexible Pricing**: Access Vast.ai's competitive rates with automatic cost caps
- **Production-Ready API**: vLLM provides OpenAI-compatible endpoints

The guide demonstrates deploying Qwen3-30B-A3B on an H100 80GB—provisioned with a single command.

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Teams** wanting reproducible, version-controlled GPU deployments
- **Developers** building LLM applications who want simple infrastructure management
- **Anyone** tired of manually provisioning and configuring GPU instances

## Get Started

The complete guide is now available in our documentation:

**[Read: Deploy LLMs with dstack and vLLM on Vast.ai →](https://docs.vast.ai/examples/serving-infrastructure/dstack-vllm)**

Whether you're new to GPU orchestration or looking for a better way to manage LLM deployments, this guide provides everything you need to get started with dstack on Vast.ai.

Ready to try it? [Sign up for Vast.ai](https://vast.ai) and follow the guide to deploy your first model.


Learn how to deploy language models with dstack and vLLM on Vast.ai, combining declarative infrastructure with competitive GPU marketplace pricing.

Deploy LLMs with dstack on Vast.ai


DR-Tulu is AI2's open-source research agent—an 8B model that autonomously plans research strategies, searches the web, reads pages, and synthesizes comprehensive answers with citations. Unlike typical LLMs with tools bolted on, DR-Tulu was trained end-to-end with its MCP tools, making it a compelling open alternative to proprietary research APIs. This guide walks through deploying DR-Tulu on Vast.ai, where you can run the full agent stack on affordable GPU instances and integrate it into your own agentic applications.

## What Makes DR-Tulu Special?

Key features include:
- **Native MCP Integration**: Trained with web search and page reading tools from the start
- **Autonomous Research**: Plans strategies, gathers sources, and synthesizes cited answers
- **Efficient Architecture**: 8B parameters with strong research capabilities

## What's in the New Guide

Our latest documentation walks you through deploying DR-Tulu on Vast.ai with a split architecture: vLLM for GPU inference on Vast.ai, MCP backend running locally for tool execution.

**Complete Deployment Instructions**
- Instance selection with the Vast.ai CLI
- vLLM configuration with the correct version (critical for compatibility)
- MCP backend setup with required API keys
- Workflow configuration to connect the components

**Three Ways to Use DR-Tulu**
- Interactive chat for exploratory research
- Batch evaluation against built-in datasets
- Python API for programmatic integration

## Why This Architecture?

The split deployment keeps GPU-intensive inference on Vast.ai while running tool orchestration locally. This means:

- Your API keys (Serper, Jina) stay on your machine (or wherever you deploy the MCP backend)
- Model inference doesn't require sending data to third-party APIs
- You can modify the MCP backend without redeploying the model
- Lower Vast.ai costs since you only need GPU for inference

## Who Should Use This Guide?

This deployment guide is perfect for:
- **Researchers** who need cited, source-backed answers at scale
- **Developers** building applications that require web research capabilities
- **Teams** evaluating open alternatives to proprietary research APIs

## Get Started

The complete guide is now available in our documentation:

**[Read: Running DR-Tulu on Vast.ai →](https://docs.vast.ai/examples/mcp/dr-tulu)**

Whether you're exploring agentic AI architectures or ready to deploy your own research assistant, this guide provides everything you need to get DR-Tulu running on Vast.ai infrastructure.

Ready to try it? [Sign up for Vast.ai](https://vast.ai) and follow the guide to deploy your first instance.


Learn how to deploy DR-Tulu, AI2's open-source research agent, on Vast.ai for autonomous web research with citations.

Posts about: Tutorial

Running Qwen 3.5 Medium Models on Vast.ai

Liquid AI's LFM2 Just Dropped — Here's How to Run It on Vast.ai

Deploy LLMs with dstack on Vast.ai

Deploy Deep Search with DR-Tulu on Vast.ai