Most annotation mistakes come from a few habits: reading guidelines once and working from memory, judging by tone or fluency instead of facts, guessing instead of flagging and rushing. None of them are about intelligence and all of them can be fixed with simple routines.
This article lists the mistakes reviewers see most, grouped by type, with the habit that prevents each one.
Mistakes with Guidelines
| Mistake | What it looks like | The habit that prevents it |
|---|---|---|
| Working from memory | Applying last week's rule after it changed | Read the change log at the start of every session |
| Skimming the definitions | Mixing up two similar labels for days | Re-read the definitions of the two closest labels whenever unsure |
| Following your instinct over the rule | "This feels like a complaint", although the rule says it's a question | Apply the rule; raise your disagreement in the question channel |
| Missing small words | Overlooking "not", "only", "except" in a rule or a task | Read instructions twice, slowly |
| Inventing your own rule for an unclear case | Consistent, but different from everyone else | Flag the item and ask; don't decide alone |
| Ignoring examples | Applying the definition in a way the examples contradict | Treat the examples as part of the rule |
The common thread: guidelines are a reference, not something you read once. Keep them open next to the task.
Mistakes When Evaluating AI Answers
AI answers are written to sound convincing. That makes these mistakes especially common:
| Mistake | Example | The habit that prevents it |
|---|---|---|
| Trusting a fluent answer | A confident, well-written answer with a wrong number gets a high score | Check every fact, number and step before scoring |
| Preferring the longer answer | The longer of two answers wins, although the short one is correct and complete | Ask "does it answer the question correctly?" before "is it thorough?" |
| Preferring the first answer shown | A wins more often simply because it appears first | Read both fully before forming a view |
| Judging style before substance | A friendly answer beats a plain, correct one | Follow the priority order: correctness, then instructions, then clarity, then tone |
| Missing ignored instructions | The user asked for three bullet points and got two paragraphs, but it isn't penalised | Re-read the prompt after reading the answer |
| Vague reasons | "A is better overall" | Name the specific problem: "B states the wrong deadline" |
| Using AI to check AI | Asking another chatbot which answer is better | Never; it breaks the rules and copies the other model's errors |
If these look familiar, they are the same biases researchers have found in AI judges, which is one reason careful human evaluation is valuable. See How to Evaluate LLM Outputs.
Mistakes in Precision Tasks
| Task | Common mistake | Prevention |
|---|---|---|
| Entity tagging | Inconsistent span edges: including a title one time, not the next | Learn the boundary rules; check the first and last character of every span |
| Documents | Marking the wrong occurrence of a value, such as the subtotal instead of the total | Know the definition of each field; check that numbers add up |
| Bounding boxes | Loose boxes with extra background, or boxes that cut off part of the object | Zoom in on edges; follow the tightness rule |
| Occluded objects | Sometimes drawing the hidden part, sometimes not | Apply the occlusion rule every time |
| Transcription | Cleaning up speech in a verbatim project, or guessing unclear words | Keep fillers if the rules say so; use [inaudible] instead of guessing |
| Dates and numbers | Swapping day and month, or using the wrong format | Follow the format rules; when in doubt, flag |
Mistakes in Work Habits
- Rushing to hit a number. Speed without accuracy gets caught by quality checks and fixing it costs more than it saved.
- Working while tired. Accuracy falls at the end of long sessions. Stop before it does.
- Not reading feedback. The same correction three times is a pattern reviewers notice.
- Guessing instead of flagging. A guess looks like a real answer and hides the problem.
- Logging time you didn't work. Hours are compared with output and padding is one of the quickest ways to lose access.
A Two-Minute Self-Check Routine
Before submitting each item, ask yourself:
- Did I read the whole item and the whole instruction?
- Does my answer follow the rule, not just my instinct?
- Did I check the facts, numbers and dates?
- Are my spans, boxes or fields exactly where the rules say?
- Is my reason specific enough for a reviewer to check?
- Should I have flagged this instead?
At first it slows you down. After a week, it becomes automatic and your accuracy shows it.
Feedback That Helps You Improve
At AnnoTrue, reviewers give specific feedback on your work and you can appeal decisions you think are wrong. Sign up for free.