Image annotation teaches models to see. The main types differ in how precisely they mark where things are: a tag says what is in the image, a bounding box draws a rectangle around each object, a polygon traces its outline, segmentation assigns every pixel and keypoints mark specific points. More precision gives the model more information, but takes much more time per image.
The best choice is the simplest type that gives the model what it needs.
Each Type Explained
Image tags (classification). One or more labels for the whole image: "contains damage", "indoor", "product: shoe". The fastest type, enough when the model only needs to know what, not where.
Bounding boxes. A rectangle drawn tightly around each object, with a class. Fast and widely supported, good for finding and counting objects. Less precise for odd shapes: a box around a diagonal pole is mostly background.
Polygons. A shape traced with points along an object's outline. Much more precise than boxes for irregular objects such as dents, roofs or leaves, but slower to draw.
Segmentation. Every pixel is assigned to a class. It comes in two forms:
| Form | What it marks | Example |
|---|---|---|
| Semantic segmentation | The class of every pixel, without separating individual objects | All road pixels are "road", all car pixels are "car" |
| Instance segmentation | Each individual object's pixels separately | Car 1, car 2 and car 3 each get their own mask |
Segmentation is the most detailed and slowest type, needed when exact shape or area matters, as in medical imaging or self-driving.
Keypoints. Specific points on an object: the joints of a body, the corners of a document, facial landmarks. Used for pose estimation, measuring and aligning.
Lines and polylines. Lines along thin objects such as road markings, cables or cracks.
Choosing the Right Type
Match the type to what the model must do:
| The model must… | Use | Example |
|---|---|---|
| Say what is in the image | Tags | Is this product photo acceptable for the shop? |
| Find and count objects | Bounding boxes | How many people are in the queue? |
| Measure the size or shape of something | Polygons or instance segmentation | How large is the dent on this car door? |
| Understand every part of a scene | Semantic segmentation | Which pixels are road, pavement or obstacle? |
| Track pose or alignment | Keypoints | Is the worker lifting with a straight back? |
| Follow thin structures | Lines | Where are the cracks in this wall? |
Mixing types is common. A retail project might tag the shelf, draw boxes around products and outline only the damaged ones.
Test before deciding. If you're unsure whether boxes are precise enough, label a small sample both ways and compare model results. It costs far less than relabeling a whole dataset later.
Rules for Consistent Image Annotation
Write these into the guidelines, with pictures:
- Tightness: boxes touch the object's outermost visible pixels, with no extra margin.
- Occlusion: for partly hidden objects, mark only the visible part, or the estimated full extent; choose one.
- Truncation: objects cut off by the image edge are still labeled; say whether to flag them.
- Minimum size: objects smaller than a set size (for example, a few pixels across) are skipped or marked as "too small".
- Groups: a crowd or a pile of boxes is labeled object by object, or as one group; decide which.
- Reflections and pictures of objects: a car in a mirror or on a poster is labeled or not; decide.
- Ambiguous classes: a van vs a truck needs a rule, ideally with a visual example of each.
Measuring quality with IoU. For boxes and masks, quality is measured with intersection over union (IoU): the overlap between the annotator's shape and the gold shape, divided by their combined area.
latex
IoU = \frac{\text{area of overlap}}{\text{area of union}}
For example, two 100 × 100 pixel boxes shifted 10 pixels sideways overlap by 9.000 pixels, out of a combined 11,000, so IoU = 0.82. An IoU of 1 means a perfect match. Projects set a threshold above which a box counts as correct; agree on it before checking work. See Quality Control in Data Labeling.
What Changes with Video
Video annotation uses the same types, plus time. Annotators:
- track each object from frame to frame, keeping the same ID even when it disappears and comes back
- label events and actions with start and end times, such as "person enters" or "car stops"
- usually draw on key frames and let the tool fill the frames between, correcting where it drifts
The main extra rules are about identity (is this the same person after they walk behind a pillar?) and about when an event starts and ends.
Precise Shapes, Checked
AnnoTrue annotators follow clear tightness and occlusion rules and we report IoU against gold shapes with every batch. Try it on your images with a free pilot.