Common Annotation Mistakes and How to Avoid Them

By Stefan, founderWorking as an annotator3 min read

Most annotation mistakes come from a few habits: reading guidelines once and working from memory, judging by tone or fluency instead of facts, guessing instead of flagging and rushing. None of them are about intelligence and all of them can be fixed with simple routines.

This article lists the mistakes reviewers see most, grouped by type, with the habit that prevents each one.

Mistakes with Guidelines

Mistake What it looks like The habit that prevents it
Working from memory Applying last week's rule after it changed Read the change log at the start of every session
Skimming the definitions Mixing up two similar labels for days Re-read the definitions of the two closest labels whenever unsure
Following your instinct over the rule "This feels like a complaint", although the rule says it's a question Apply the rule; raise your disagreement in the question channel
Missing small words Overlooking "not", "only", "except" in a rule or a task Read instructions twice, slowly
Inventing your own rule for an unclear case Consistent, but different from everyone else Flag the item and ask; don't decide alone
Ignoring examples Applying the definition in a way the examples contradict Treat the examples as part of the rule

The common thread: guidelines are a reference, not something you read once. Keep them open next to the task.

Mistakes When Evaluating AI Answers

AI answers are written to sound convincing. That makes these mistakes especially common:

Mistake Example The habit that prevents it
Trusting a fluent answer A confident, well-written answer with a wrong number gets a high score Check every fact, number and step before scoring
Preferring the longer answer The longer of two answers wins, although the short one is correct and complete Ask "does it answer the question correctly?" before "is it thorough?"
Preferring the first answer shown A wins more often simply because it appears first Read both fully before forming a view
Judging style before substance A friendly answer beats a plain, correct one Follow the priority order: correctness, then instructions, then clarity, then tone
Missing ignored instructions The user asked for three bullet points and got two paragraphs, but it isn't penalised Re-read the prompt after reading the answer
Vague reasons "A is better overall" Name the specific problem: "B states the wrong deadline"
Using AI to check AI Asking another chatbot which answer is better Never; it breaks the rules and copies the other model's errors

If these look familiar, they are the same biases researchers have found in AI judges, which is one reason careful human evaluation is valuable. See How to Evaluate LLM Outputs.

Mistakes in Precision Tasks

Task Common mistake Prevention
Entity tagging Inconsistent span edges: including a title one time, not the next Learn the boundary rules; check the first and last character of every span
Documents Marking the wrong occurrence of a value, such as the subtotal instead of the total Know the definition of each field; check that numbers add up
Bounding boxes Loose boxes with extra background, or boxes that cut off part of the object Zoom in on edges; follow the tightness rule
Occluded objects Sometimes drawing the hidden part, sometimes not Apply the occlusion rule every time
Transcription Cleaning up speech in a verbatim project, or guessing unclear words Keep fillers if the rules say so; use [inaudible] instead of guessing
Dates and numbers Swapping day and month, or using the wrong format Follow the format rules; when in doubt, flag

Mistakes in Work Habits

  • Rushing to hit a number. Speed without accuracy gets caught by quality checks and fixing it costs more than it saved.
  • Working while tired. Accuracy falls at the end of long sessions. Stop before it does.
  • Not reading feedback. The same correction three times is a pattern reviewers notice.
  • Guessing instead of flagging. A guess looks like a real answer and hides the problem.
  • Logging time you didn't work. Hours are compared with output and padding is one of the quickest ways to lose access.

A Two-Minute Self-Check Routine

Before submitting each item, ask yourself:

  • Did I read the whole item and the whole instruction?
  • Does my answer follow the rule, not just my instinct?
  • Did I check the facts, numbers and dates?
  • Are my spans, boxes or fields exactly where the rules say?
  • Is my reason specific enough for a reviewer to check?
  • Should I have flagged this instead?

At first it slows you down. After a week, it becomes automatic and your accuracy shows it.

Feedback That Helps You Improve

At AnnoTrue, reviewers give specific feedback on your work and you can appeal decisions you think are wrong. Sign up for free.