Good annotation guidelines tell a careful newcomer exactly which label to choose for any item, including the awkward ones, without needing to ask. They do it with short definitions, real examples, explicit rules for overlaps, a list of edge cases and a clear instruction for what to do when unsure.
Guidelines are the single biggest lever on label quality. Two annotators with clear guidelines agree far more often than two experts working from memory. This article gives you a template, a worked example and the rules that make guidelines easy to follow.
A Template You Can Copy
Most good guidelines follow the same structure. Use these sections in this order:
| Section | What it contains | Typical length |
|---|---|---|
| 1. Purpose | What the labels are for, in two or three sentences | A short paragraph |
| 2. The task in one line | Exactly what the annotator does with each item | One sentence |
| 3. Label list | Every label, with a one-line definition | One line per label |
| 4. Label details | For each label: definition, 2–3 clear examples, 1–2 "not this" examples | Half a page per label |
| 5. Decision rules | What wins when an item fits two labels | A short numbered list |
| 6. Edge cases | Known unusual items and the rule for each | A table, growing over time |
| 7. When unsure | How to flag an item or ask a question and how fast you answer | A few lines |
| 8. Format rules | Box tightness, span boundaries, punctuation, date formats | As needed |
| 9. Quality | How work is checked and what counts as an error | A few lines |
| 10. Change log | Every change, with its date | Grows over time |
Put the label list near the top. Annotators return to it constantly. A one-page summary at the start, with details below, saves hours over a project.
Explain the purpose. When annotators know what the labels are for, they make better calls on items the guidelines didn't foresee.
A Worked Example: One Label, Written Badly and Well
Imagine a project that sorts customer emails for a bank. Here is the label "Billing dispute" written two ways.
Written badly:
Billing dispute: emails about billing problems.
It sounds clear, but it leaves every real question open. Is a question about a fee a dispute? What about a refund request? A double charge that was already fixed?
Written well:
Billing dispute The customer says a charge on their account is wrong and wants it reversed or explained.
Examples:
- "I was charged twice for the same payment on 04.05.2026."
- "Why did you take a €15 fee? I never agreed to it."
- "This transaction isn't mine. Please cancel it." (also flag as Possible fraud)
Not this label:
- "How much is the fee for a transfer abroad?" → Fees and pricing (a question, not a complaint)
- "Thanks, the refund arrived." → No action needed
Rule: if the customer disputes a charge and reports a blocked card, choose Card problem. The card team handles both.
The second version takes a minute longer to read and saves hundreds of wrong labels. Notice the three parts: a definition that says what the customer wants, real examples in the customer's own words and examples that look similar but belong elsewhere.
Writing Rules That Annotators Actually Follow
- One rule per line. Long paragraphs hide rules. Break them into short, numbered statements.
- Use examples from your real data. Invented examples are too clean; annotators need to see the messy versions they will meet.
- Show what a label is not. The closest wrong label is where most errors happen. Put the two side by side.
- Make decision rules explicit. "If both apply, choose X" removes guessing. Order matters, so say which rule wins.
- Avoid vague words. "Mostly", "usually", "clearly" and "relevant" mean different things to different people. Replace them with a test: "more than half the text", "the first request in the email".
- Show, don't only tell, for visual tasks. A picture of a correctly drawn box teaches more than a paragraph about box tightness.
- Never punish "not sure". If annotators fear flagging items, they guess and guesses look like real answers.
- Keep the language simple. Many annotators read the guidelines in their second language. Short sentences and common words reduce mistakes.
- Write rules for the model's goal, not your personal taste. If two labels lead to the same action, merging them is better than writing a fine distinction.
For LLM evaluation tasks, add a priority order: for example, correctness first, then whether the answer follows the instructions, then clarity, then tone. Without it, annotators weigh these differently and agreement drops. See How to Evaluate LLM Outputs.
Testing and Maintaining Your Guidelines
Guidelines are never right the first time. Plan for at least one round of changes.
Do a dry run before the pilot. Give 20 to 50 items to two people who didn't write the guidelines. Collect every question they ask and every item where they disagree. Each one points to a gap: a missing example, a vague word or a missing rule.
Use the pilot to measure. Agreement between annotators, broken down by label, shows exactly where the guidelines are weak. Low agreement on one label usually means its definition overlaps with a neighbour. See Inter-Annotator Agreement Explained Simply.
Keep a change log. Every change gets a date, a short description and, if it changes past labels, a note on whether earlier items must be relabeled. Without it, nobody knows which rule applied to which batch.
Grow the edge-case table. When annotators flag a new kind of item, decide the rule, add it to the table with an example and tell the whole team. The edge-case table often becomes the most valuable part of the document. See Handling Edge Cases and Ambiguity in Labeling Projects.
Version the document. Name versions (v1.0, v1.1) and store each one with the batches it applied to. Months later, when you retrain or add data, you will need to know.
Let Us Test Your Guidelines
In a free pilot, AnnoTrue reviews your guidelines, labels a sample within 10 working days and reports every question and disagreement we found, so you know exactly what to fix before scaling.