Labeling Every Pixel: Semantic vs Instance Segmentation | Visual AI Learning Guide

Compare the two panels. On the left, every pixel is colored by its class — all the people share one color, so the two figures blend into a single region. That is semantic segmentation: it tells you what each pixel is, but not which object it belongs to. On the right, the same scene is split into separate masks, one per object. The two people now have distinct regions even though their class is identical. So the question to ask is: do you need to know what is where, or do you need to count and separate individual things? The first is semantic, the second is instance.

0:00 / 0:00

Auto-play

Segmentation goes finer than boxes. Instead of a rectangle around an object, the model assigns a class to every pixel in the image. Semantic segmentation produces one label map: every pixel is tagged as road, sky, person, car, and so on. It does not distinguish between two people — both are simply labeled "person," and the two regions merge into one. Instance segmentation adds that distinction. Each individual object gets its own mask, so two people are two separate regions even though they share the same class. The difference matters in practice: a crowd-counting task needs instance segmentation, because it must know how many people there are, while a scene-parsing task that only needs to know which areas are road and which are sidewalk is satisfied by semantic segmentation. Both approaches produce pixel-level output, but only instance segmentation keeps objects apart.