Granite 4.0 H Small

Granite 4.0 H Small is a 32-billion-parameter long-context instruct model from the Granite Team at IBM. It is finetuned from Granite-4.0-H-Small-Base using a combination of permissively licensed open-source instruction datasets and internally collected synthetic data, refined through supervised finetuning, reinforcement-learning alignment, and model merging. Granite 4.0 instruct models emphasize improved instruction following and tool-calling, making them well suited to enterprise and agentic applications.

Architecture

Granite 4.0 H Small uses a decoder-only Mixture-of-Experts transformer built on a hybrid attention design. Its core components are Grouped-Query Attention, Mamba2 state-space layers, MoE feed-forward blocks with shared experts, SwiGLU activations, RMSNorm, and shared input/output embeddings. The hybrid layout pairs a small number of full-attention layers with a majority of Mamba2 layers (4 attention layers to 36 Mamba2 layers), which keeps the KV cache small even at long context and makes the model efficient to serve across a wide range of sequence lengths. As a sparse MoE, only a fraction of its total experts are activated per token, so it delivers the quality of a large model with the runtime cost closer to a much smaller one.

Key Features

• Native context window of 131,072 tokens for long-document and long-conversation workloads.
• Hybrid Mamba2/transformer architecture with a very low KV-cache footprint at long context.
• Strong tool-calling and function-calling support using an OpenAI-style function definition schema.
• Multilingual coverage across English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese.

Use Cases

• AI assistants and chatbots for business and general-purpose domains.
• Agentic and tool-using workflows that call external functions and APIs.
• Retrieval-Augmented Generation over enterprise knowledge bases.
• Summarization, text classification, and structured text extraction.
• Question answering and multilingual dialogue.
• Code-related tasks, including Fill-In-the-Middle completions.

Granite 4.0 H Small is available on Vast with vLLM and SGLang for full-precision and FP8 serving, and with llama.cpp for quantized GGUF deployments.

For full model details, benchmarks, and usage examples, see the model card on HuggingFace.