SAM 2: Segment Anything in Images and Videos

Segment Anything Model 2 (SAM 2) is Meta FAIR's foundation model for promptable visual segmentation in images and videos, introduced in the paper "SAM 2: Segment Anything in Images and Videos". Given a prompt — a click, a box, or a mask — it produces a precise segmentation mask for the indicated object, and it can also segment everything in an image automatically with no prompt at all. This template runs the SAM 2.1 Hiera Large checkpoint, the largest and most accurate of the improved SAM 2.1 weights published at facebook/sam2.1-hiera-large.

Overview

SAM 2 extends the original Segment Anything Model from images to video by treating an image as a one-frame video. The design is a simple transformer with streaming memory: each frame is encoded once, and a memory attention module conditions the current frame on the ones before it, so an object selected on one frame can be tracked and re-segmented across an entire clip in real time. A model-in-the-loop data engine, where annotators and the model refine each other, was used to build the SA-V dataset, the largest video segmentation dataset released to date, and SAM 2 trained on it delivers strong performance across a wide range of tasks and visual domains.

Key Features

• Promptable segmentation - Point clicks (positive and negative), bounding boxes, and input masks all steer the prediction; a few clicks are usually enough to isolate a subject cleanly
• Automatic mask generation - With no prompt, the automatic mask generator sweeps a grid of points across the image and returns a mask for every distinct object it finds
• Zero-shot generality - As a segmentation foundation model, it handles objects and scenes it was never explicitly trained on, without fine-tuning
• Video object tracking - The streaming-memory architecture propagates masks across frames, tracking one or many objects through a video from prompts on a single frame
• A family of checkpoints - SAM 2.1 ships tiny, small, base-plus, and large variants; Hiera Large, served here, is the most accurate of the four while remaining fast enough for interactive use

Architecture

SAM 2 pairs a hierarchical Hiera vision transformer encoder with a lightweight prompt encoder and mask decoder derived from the original SAM design. For video, a memory encoder compresses previous frames' predictions into a memory bank, and memory attention lets the current frame attend to it, which is what allows masks to follow an object through occlusion and motion. The SAM 2.1 checkpoints are a retrained suite with improved accuracy over the original SAM 2 release; the Hiera Large variant used here has 224.4 million parameters, so it loads in seconds and segments an image in a fraction of a second on a modern GPU.

Use Cases

• Interactive object cutouts: click a subject to get a clean mask for compositing or editing
• Automatic scene decomposition: segment every object in an image for analysis or labeling pipelines
• Dataset annotation: bootstrap segmentation labels far faster than manual polygon tracing
• Masking for generative editing: feed SAM 2 masks into inpainting and regional-prompt workflows
• Video rotoscoping and object tracking: select an object once and propagate its mask through a clip
• Robotics and vision research: a strong off-the-shelf segmentation signal for downstream systems

On This Template

The template runs SAM 2.1 Hiera Large inside ComfyUI via dedicated SAM 2 nodes. A ready-made workflow is preloaded: load any image and the automatic mask generator segments every object it finds, saving a color-coded segmentation overlay and a black-and-white mask of everything it found. The workflow is also available through the built-in API wrapper, which returns both images. The same node pack also provides point- and box-prompted segmentation and video segmentation nodes, so the preloaded checkpoint can be wired into selective masking or tracking workflows without downloading anything else.