K2 Horizon MoVA 36B-A4B

K2 Horizon MoVA 36B-A4B is the sparse member of IFM's K2 Horizon family. It is a Mixture-of-Experts model that also uses Mixture-of-Values attention (MoVA), storing 36B parameters and running about 4B per token. IFM releases it fully open: the final checkpoint, the intermediate training checkpoints, the training data and recipe, and the training code are all public.

Key Features

• Sparse and efficient - routed experts in the feed-forward layers plus routed value experts inside attention, so only a small slice of the model runs for each token.
• 512K-token context - native 524,288-token context from the midtraining stages onward.
• Selectable reasoning depth - thinking effort can be set per request to high (the default), medium, or low.
• Tool calling - supports JSON, XML, and typed-XML tool-call formats, chosen per request.
• Intermediate checkpoints - checkpoints from every training stage are released, so capability changes can be studied across training rather than at a single point.
• Fully open - training data, recipe, and code are published alongside the weights.

Multi-Turn Conversations

The server returns the model's thinking in the reasoning field, separate from the answer in content. When you send a conversation back, every earlier assistant turn must carry that thinking in reasoning (or reasoning_content), including turns that only make tool calls. An assistant turn without it is rejected with HTTP 400 "Assistant message is missing a thinking field".

{
  "model": "IFM/K2-Horizon-MoVA-36B-A4B",
  "messages": [
    {"role": "user", "content": "What is 17 x 23?"},
    {"role": "assistant", "content": "17 x 23 = 391.", "reasoning": "17 x 20 = 340, 17 x 3 = 51, 340 + 51 = 391."},
    {"role": "user", "content": "Now divide that by 7."}
  ]
}

If your client does not keep the reasoning, send "reasoning": "" on each earlier assistant turn. An empty string is accepted; the model then sees that turn's answer with an empty thinking block.

Training

The model was pretrained on 22.9T tokens at an 8K sequence length. Four midtraining stages then extended the context to 32K, 128K, and finally 512K, with the last stage shifting the data mix toward agentic and reasoning data. Two phases of supervised fine-tuning at 512K followed, the second on a high-quality subset of the first with learning-rate decay.

Benchmark Highlights

On its published evaluations, IFM reports 26.8 on tau3-Banking for agentic tool use, 58.6 on Terminal-Bench 2.1 for agentic terminal use, and 38.9 on SciCode. On scientific reasoning it reaches 80.8 on GPQA Diamond and 25.2 on Humanity's Last Exam without tools. For long-context reasoning it scores 66.3 on AA-LCR, and it posts a 69.2 non-hallucination rate on AA-Omniscience.

Use Cases

• Agentic workflows and tool use
• Coding and terminal automation
• Scientific and multi-step reasoning
• Long-document analysis over very long contexts
• Research on open models and on how capabilities develop during training

Deploy K2 Horizon MoVA 36B-A4B on Vast.ai to run a fully open, efficient Mixture-of-Experts model with vLLM.