Quality Control in Data Labeling: Methods That Work

By Stefan, founderThe process, step by step4 min read

Reliable labeling quality comes from several checks running at the same time, each catching a different kind of problem: automatic checks for mechanical mistakes, hidden gold items for accuracy, overlap between annotators for unclear rules, reviewer sampling for judgment errors and batch acceptance before anything is delivered.

No single check is enough. Gold items miss problems on items unlike the gold set. Overlap shows disagreement but not who is right. Reviewers can't see everything. Together, they give a picture you can trust and act on.

The Five Layers of Quality Control

Layer How it works What it catches What it misses
1. Automatic checks Rules run on every item as it is saved Empty fields, wrong formats, boxes outside the image, impossibly fast work Wrong judgments that look valid
2. Hidden gold items A small share of items with known answers mixed into the work Each annotator's accuracy, without them knowing which items are tested Errors on kinds of items not in the gold set
3. Overlap A share of items labeled by two or more people independently Unclear rules and inconsistent annotators Which answer is right, when both are wrong the same way
4. Reviewer sampling An experienced reviewer checks a sample of each person's work Judgment errors, misread guidelines, sloppy patterns Problems outside the sample
5. Batch acceptance Each batch is checked against an agreed error limit before delivery Batches that fall below the standard Nothing, if the earlier layers work; it is the final gate

Layer 1 costs almost nothing once set up, so use it everywhere. Layer 2 is the backbone of accuracy measurement; see Gold Standard Datasets: How to Build and Use Them. Layer 3 is the best early warning that guidelines are unclear; see Inter-Annotator Agreement Explained Simply. Layer 4 catches what numbers can't: a reviewer notices why someone gets things wrong. Layer 5 protects the client.

Check new annotators more closely. Review all of a new person's first items, not a sample. Most bad habits show up early and are easy to fix early.

Setting Sample Sizes and Thresholds

There is no universal number, but there are sensible starting points that you adjust as the project shows its risks.

Check A common starting point Raise it when
Gold items Around 5% of items Accuracy matters a lot, or annotators are new
Overlap Around 10% of items Agreement is low or labels are subjective
Reviewer sampling A share of each annotator's work every week Someone's scores drop, or guidelines just changed
New annotators 100% of their first items, for example the first 50 The task is complex or high-stakes
Batch acceptance A random sample per batch, checked against the guidelines The client's own use is high-stakes

These are the starting points we use at AnnoTrue; your project may need more or less.

Small samples give rough answers. A sample error rate is an estimate and its uncertainty shrinks as the sample grows. For example, at 95% confidence:

Sample checked Errors found Observed error rate Likely range of the true rate
100 items 3 3% about 1.0% to 8.5%
500 items 15 3% about 1.8% to 4.9%
2.000 items 60 3% about 2.3% to 3.8%

So if your acceptance limit is 5%, finding 3 errors in 100 items doesn't prove the batch is fine. Check more items when the result is close to the limit.

Agree on what counts as an error. Before any checking, define error severity: a wrong label is an error; a box two pixels loose may not be. Write it down, or reviewers and clients will count differently.

Acting on the Results

Checks are only useful if something happens when they find a problem. Match the response to the pattern:

What the checks show Likely cause What to do
One annotator's accuracy drops Fatigue, misunderstanding, or rushing Specific feedback with examples; closer review for a while
Everyone gets one label wrong The guideline for that label is unclear Fix the rule, add examples, tell the whole team, log the change
Agreement drops across the board New kinds of items in the data, or a recent guideline change Review recent flags; add edge-case rules
Very fast work with normal accuracy on gold Possible gaming of gold items Check non-gold work closely; refresh the gold set
Repeated failures after feedback Not a good fit for this task Move the person to a different task, or off the project

Feedback should be fast and specific. "Item 4821: this is Fees and pricing, not Billing dispute, because the customer asks a question rather than complaining (rule 5.2)" teaches. "Your accuracy is low" doesn't.

Decide what happens to bad work. Agree in advance whether items found wrong are fixed by the same person, by someone else, or relabeled from scratch and who pays. At AnnoTrue, failed work is redone by other annotators and a batch that misses the agreed accuracy is fixed free of charge within 5 working days.

Be fair to annotators. Most quality problems start in the guidelines, not with the people. Check that first.

What a Quality Report Should Contain

Every delivered batch should come with a short report. Ask your vendor for at least:

  • items delivered and items flagged or skipped, with reasons
  • accuracy on gold items, overall and per label
  • agreement on overlap items and the most common disagreements
  • reviewer findings: main error types and what was done
  • guideline changes that applied to this batch
  • batch acceptance result against the agreed limit

A vendor that can't produce these numbers isn't measuring them.

Five Layers on Every Project

AnnoTrue runs all five layers on every project and sends a quality report with each batch. See it in practice with a free pilot, before you commit.