Image Annotation Types: Bounding Boxes, Polygons and Segmentation

By Stefan, founderData types and tasks4 min read

Image annotation teaches models to see. The main types differ in how precisely they mark where things are: a tag says what is in the image, a bounding box draws a rectangle around each object, a polygon traces its outline, segmentation assigns every pixel and keypoints mark specific points. More precision gives the model more information, but takes much more time per image.

The best choice is the simplest type that gives the model what it needs.

Each Type Explained

Image tags (classification). One or more labels for the whole image: "contains damage", "indoor", "product: shoe". The fastest type, enough when the model only needs to know what, not where.

Bounding boxes. A rectangle drawn tightly around each object, with a class. Fast and widely supported, good for finding and counting objects. Less precise for odd shapes: a box around a diagonal pole is mostly background.

Polygons. A shape traced with points along an object's outline. Much more precise than boxes for irregular objects such as dents, roofs or leaves, but slower to draw.

Segmentation. Every pixel is assigned to a class. It comes in two forms:

Form What it marks Example
Semantic segmentation The class of every pixel, without separating individual objects All road pixels are "road", all car pixels are "car"
Instance segmentation Each individual object's pixels separately Car 1, car 2 and car 3 each get their own mask

Segmentation is the most detailed and slowest type, needed when exact shape or area matters, as in medical imaging or self-driving.

Keypoints. Specific points on an object: the joints of a body, the corners of a document, facial landmarks. Used for pose estimation, measuring and aligning.

Lines and polylines. Lines along thin objects such as road markings, cables or cracks.

Choosing the Right Type

Match the type to what the model must do:

The model must… Use Example
Say what is in the image Tags Is this product photo acceptable for the shop?
Find and count objects Bounding boxes How many people are in the queue?
Measure the size or shape of something Polygons or instance segmentation How large is the dent on this car door?
Understand every part of a scene Semantic segmentation Which pixels are road, pavement or obstacle?
Track pose or alignment Keypoints Is the worker lifting with a straight back?
Follow thin structures Lines Where are the cracks in this wall?

Mixing types is common. A retail project might tag the shelf, draw boxes around products and outline only the damaged ones.

Test before deciding. If you're unsure whether boxes are precise enough, label a small sample both ways and compare model results. It costs far less than relabeling a whole dataset later.

Rules for Consistent Image Annotation

Write these into the guidelines, with pictures:

  • Tightness: boxes touch the object's outermost visible pixels, with no extra margin.
  • Occlusion: for partly hidden objects, mark only the visible part, or the estimated full extent; choose one.
  • Truncation: objects cut off by the image edge are still labeled; say whether to flag them.
  • Minimum size: objects smaller than a set size (for example, a few pixels across) are skipped or marked as "too small".
  • Groups: a crowd or a pile of boxes is labeled object by object, or as one group; decide which.
  • Reflections and pictures of objects: a car in a mirror or on a poster is labeled or not; decide.
  • Ambiguous classes: a van vs a truck needs a rule, ideally with a visual example of each.

Measuring quality with IoU. For boxes and masks, quality is measured with intersection over union (IoU): the overlap between the annotator's shape and the gold shape, divided by their combined area.

latex IoU = \frac{\text{area of overlap}}{\text{area of union}}

For example, two 100 × 100 pixel boxes shifted 10 pixels sideways overlap by 9.000 pixels, out of a combined 11,000, so IoU = 0.82. An IoU of 1 means a perfect match. Projects set a threshold above which a box counts as correct; agree on it before checking work. See Quality Control in Data Labeling.

What Changes with Video

Video annotation uses the same types, plus time. Annotators:

  • track each object from frame to frame, keeping the same ID even when it disappears and comes back
  • label events and actions with start and end times, such as "person enters" or "car stops"
  • usually draw on key frames and let the tool fill the frames between, correcting where it drifts

The main extra rules are about identity (is this the same person after they walk behind a pillar?) and about when an event starts and ends.

Precise Shapes, Checked

AnnoTrue annotators follow clear tightness and occlusion rules and we report IoU against gold shapes with every batch. Try it on your images with a free pilot.