Segment Anything (SAM) and Dense Prediction

Segmentation used to be the most data-hungry task in vision. Every new domain - aerial imagery, medical scans, retail catalogues - meant collecting hundreds of thousands of pixel-perfect masks and training a fresh U-Net. Kirillov et al's 2023 Segment Anything paper changed the framing: train one model on a billion masks, make segmentation promptable (give me the mask for this point, this box, this region), and ship a foundation model that generalises to new domains zero-shot. SAM is to segmentation what CLIP is to classification.

Promptable segmentation as a foundation task

The task definition is deceptively simple. Given an image and a prompt (point, box, mask, or text), return a valid mask. "Valid" means the model picks the most sensible segment containing the prompt even when ambiguous - if you click on a person's shirt, the model should be able to return the shirt, the upper body, or the whole person. SAM emits three candidate masks per prompt with confidence scores to handle this ambiguity explicitly.

This framing has three consequences:

• It is interactive by construction. The model is built to be steered, which is exactly the workflow product teams want.
• It is task-general. The same model handles instance segmentation, object boundary refinement, and selection tooling.
• It is data-bootstrappable. The promptable interface lets you build a data engine where the model proposes masks, humans correct them, and corrections feed back into training. SAM's SA-1B dataset (1.1B masks on 11M images) was built this way.

The encoder + lightweight decoder split

SAM's architecture is designed around a specific deployment pattern: the image is encoded once, then prompted many times.

The image encoder produces a 64x64x256 feature grid that gets cached. Every subsequent click, box, or refinement uses the cached features and runs only the cheap decoder. Interactive selection becomes responsive - sub-100ms per click - because the heavy lifting happens once upfront.

The mask decoder is a small two-way transformer: image features cross-attend to prompt tokens, prompt tokens cross-attend to image features, and output tokens predict the mask via a small MLP. The whole decoder is intentionally tiny so that prompt latency stays interactive.

SAM 2 and video

SAM 2 (Ravi et al, August 2024) extends the promptable design to video. The core changes:

• Streaming memory. A small memory bank holds features from recent frames; the current frame attends to it. Mask propagation across frames is automatic.
• Single unified model for image and video. A single image is just a one-frame video.
• Trained on SA-V, a 51k-video, 643k-masklet dataset built with a new video data engine.

Reported gains over SAM 1 on images alone: 6x faster, slightly better mIoU. On video, 3x fewer user interactions for equivalent quality vs prior video-segmentation pipelines. The image encoder is also smaller (Hiera, a hierarchical ViT variant) which helps throughput.

Where SAM falls down

The failure modes are predictable from how it was trained:

• Fine-grained boundaries. SAM was trained on photographic masks where pixel-precise edges were rare in the labels. Hair, fur, wisps of smoke, transparent surfaces - SAM under-segments these. Specialised matting models (MODNet, BiRefNet) still beat it.
• Medical and satellite imagery. SA-1B is natural images. CT scans, MRI, histopathology, aerial multispectral - SAM works far worse out of the box than supervised baselines. MedSAM and the SAM-Med2D / SAM-Med3D fine-tunes recover much of the gap but the zero-shot story is weak.
• Semantic labelling. SAM gives you a mask, not a class. Pairing it with a CLIP-like classifier (Grounded-SAM, Segment Anything in High Quality) is the usual recipe for open-vocabulary segmentation.
• Very small objects. Below ~32 pixels SAM's feature resolution starts to dominate the error.

What segmentation foundation models change for product workflows

The before-and-after for a team that needs custom segmentation:

The annotation acceleration alone has reorganised internal vision teams. The frozen-encoder, fine-tune-decoder pattern means a small team can adapt SAM to a new vertical without retraining the expensive part. Most production "AI-powered selection" features (background removal, magic wand, video rotoscoping) shipped in 2023-2024 are SAM under the hood.

Where it falls down (the limits of "anything")

• Anything except the things SAM was trained for. "Anything" is marketing - it means anything in SA-1B-style natural images. Medical, scientific, and remote-sensing domains need fine-tuning.
• The encoder is heavy. 450ms per image on an A100 is fine for interactive tools, expensive for high-throughput batch processing. The smaller ViT-B variant is 4x faster with measurable quality loss.
• No instance count. SAM segments one thing per prompt. Counting all sheep in an image requires a separate detector to generate the prompts.

Further reading

• Segment Anything - Kirillov et al 2023, the original SAM paper with the promptable task definition and SA-1B dataset.
• SAM 2: Segment Anything in Images and Videos - Ravi et al 2024, the streaming-memory video extension.
• facebookresearch/segment-anything - reference SAM implementation and checkpoints.