Inter-annotator agreement measures how often two or more people give the same label to the same item, working independently. High agreement means the guidelines are clear enough for different people to reach the same answer. Low agreement means the task, the labels or the guidelines leave too much room for interpretation.
Agreement is not the same as accuracy: two people can agree and both be wrong. But it is the best early warning that something in your project is unclear and it doesn't need answers known in advance.
Why Simple Percent Agreement Misleads
The obvious measure is the share of items where both annotators chose the same label. It is easy to understand and worth reporting, but it has a flaw: some agreement happens by pure chance.
Imagine a task where 95% of emails are "not spam". Two annotators who don't read the emails at all and just pick "not spam" 95% of the time at random, would still agree on about 90% of items. The number looks excellent, yet they agreed on nothing that matters.
The fewer labels there are and the more one label dominates, the more agreement chance alone produces. That is why researchers use measures that subtract chance agreement, such as Cohen's kappa.
Cohen's Kappa, with a Worked Example
Cohen's kappa compares the agreement you observed with the agreement you would expect by chance:
latex
\kappa = \frac{p_o - p_e}{1 - p_e}
where pₒ is the observed agreement and pₑ the agreement expected by chance. A kappa of 1 means perfect agreement; 0 means no better than chance.
Example. Two annotators label the same 100 emails as either Complaint or Question:
| B: Complaint | B: Question | Total (A) | |
|---|---|---|---|
| A: Complaint | 40 | 10 | 50 |
| A: Question | 5 | 45 | 50 |
| Total (B) | 45 | 55 | 100 |
Step 1: observed agreement. They agree on 40 + 45 = 85 emails, so pₒ = 0.85.
Step 2: chance agreement. Annotator A says Complaint 50% of the time, B says it 45% of the time. By chance, both say Complaint 0.50 × 0.45 = 0.225 of the time and both say Question 0.50 × 0.55 = 0.275 of the time. So pₑ = 0.225 + 0.275 = 0.50.
Step 3: kappa. (0.85 − 0.50) / (1 − 0.50) = 0.35 / 0.50 = 0.70.
So 85% raw agreement becomes a kappa of 0.70 once chance is removed. The table also shows where they disagree: A calls 10 emails complaints that B calls questions. That points straight at the guideline rule that separates the two labels.
Other Measures and How to Read the Numbers
| Measure | Use it when | Notes |
|---|---|---|
| Percent agreement | Always, as a simple first look | Easy to explain, but ignores chance |
| Cohen's kappa | Two annotators, categories | The most common chance-corrected measure |
| Fleiss' kappa | Three or more annotators, categories | Extends the same idea to more people |
| Krippendorff's alpha | Any number of annotators, missing labels, or ordered scales such as 1–5 ratings | The most flexible; widely used in research |
| Span or box overlap measures | Highlighted text or drawn shapes | Measures how much two marks overlap, not just whether labels match |
How to read kappa. A widely quoted scale from Landis and Koch (1977) calls 0.41–0.60 "moderate", 0.61–0.80 "substantial" and 0.81–1.00 "almost perfect" agreement. On that scale, our example's 0.70 is substantial. But treat such scales with care: the authors gave no evidence for the cut-offs and kappa values depend on the number of labels and how evenly they are used (source).
How to read alpha. Krippendorff suggested relying on data with alpha of at least 0.800 and using data between 0.667 and 0.800 only for tentative conclusions, with a higher bar when the stakes are high (source).
The practical rule. Agree on a target for your task during the pilot, then watch the trend. A drop in agreement is often more informative than the absolute number. Subjective tasks, such as rating tone or helpfulness, naturally agree less than factual ones, such as extracting an invoice date.
What to Do When Agreement Is Low
- Find where people disagree. Build a table like the one above for each pair of labels. Disagreement is rarely spread evenly; it clusters on two or three label pairs.
- Read the disputed items together. Ask both annotators why they chose what they did. The answer usually reveals a missing rule or a vague word.
- Fix the guidelines. Add a decision rule, more examples, or merge labels the model doesn't need to tell apart. See How to Write Annotation Guidelines.
- Check the people. If one annotator disagrees with everyone, they may need feedback or retraining.
- Measure again on a fresh sample after the changes.
Agreement Reported with Every Batch
AnnoTrue labels a share of every project twice and reports agreement and the main disagreements with each delivery. Try it with a free pilot.
Sources
- Cohen's kappa, Wikipedia, citing Landis and Koch (1977), "The measurement of observer agreement for categorical data", Biometrics
- Krippendorff's alpha, Wikipedia