Handling Edge Cases and Ambiguity in Labeling Projects

By Stefan, founderThe process, step by step4 min read

Edge cases are items your guidelines didn't plan for: an email in two languages, a blurry photo, a chatbot answer that is half right. Handle them with a simple loop: annotators flag the item instead of guessing, one owner decides the rule, the rule goes into the guidelines with an example and the whole team is told.

Edge cases are where most labeling errors and disagreements live. A project that handles them well can reach high consistency even on messy real-world data.

The Kinds of Edge Cases

Edge cases come in a few recognisable types. Knowing them helps you plan for them.

Type What it looks like Example
Fits two labels The item matches more than one definition An email that both complains about a fee and asks to close the account
Fits no label Nothing in the label set applies A blank page, spam, a message in another language
Poor quality The content can't be read or heard properly A blurry scan, a noisy recording, a photo cut off at the edge
Borderline It sits on the line between two labels A review that is mildly positive with one complaint
Rare but important Seldom seen, but costly to get wrong A message hinting at fraud or self-harm
Outside the rules' assumptions Something the guidelines never imagined A chatbot answer that refuses politely but includes the requested information anyway
Expert-only Only a specialist can judge it A contract clause that is unusual but legal

The first two types can be planned for from the start: write a decision rule for overlaps and give annotators a label for "none of these". The rest are found during the work.

Finding Edge Cases Early

The earlier you find an edge case, the fewer items are labeled inconsistently before the rule exists.

  • Look at the data before writing guidelines. Skim a few hundred random items and note anything odd.
  • Ask the people who know the data. Support agents, lawyers or doctors on the client side know the strange cases by heart.
  • Run a dry run. Two people label 20 to 50 items; every question is a candidate edge case.
  • Use the pilot's disagreements. Items where annotators disagreed are edge cases by definition. See Inter-Annotator Agreement Explained Simply.
  • Make flagging easy. A one-click "not sure" or "flag" button in the tool, with a short comment, turns every annotator into an edge-case finder.
  • Watch the model. After training, items the model gets wrong often turn out to be edge cases nobody decided.

The Decision Process: Flag, Decide, Document, Tell

  1. Flag. The annotator marks the item as unclear, adds a one-line note and moves on. No guessing.
  2. Decide. One owner, usually the client's project lead or a senior reviewer agreed with the client, decides the rule. One owner keeps decisions consistent.
  3. Document. The rule goes into the guidelines' edge-case table, with the example and the date.
  4. Tell. The whole team gets the new rule, not just the person who asked.
  5. Fix the past, if needed. Decide whether items labeled before the rule need a second look. Items flagged earlier are relabeled under the new rule.

Speed matters. If questions wait days for an answer, annotators either stop or start guessing. Agree on a response time, for example within one working day.

An example edge-case table for an email classification project:

# Case Rule Example Added
E-01 Email in two languages Label by the main request, whatever its language "Hello, quiero cerrar mi cuenta" → Close account v1.1
E-02 Complaint plus account closure Choose Close account; add the tag Complaint "Your fees are a joke. Close my account." v1.1
E-03 Forwarded chain with several requests Label the newest message only A reply on top of three older emails v1.2
E-04 Possible fraud mentioned anywhere Always add the Possible fraud flag, whatever the main label "There's a payment I don't recognise" v1.2

This table often becomes the most valuable part of the guidelines.

When an Item Is Genuinely Ambiguous

Some items have no single right answer, even for experts: a sarcastic review, a joke that may or may not be offensive, two AI answers of equal quality. Forcing one label can hide useful information.

Options for these cases:

  • Allow a "both" or "tie" answer where the task permits it, for example "A and B are equally good" in comparisons.
  • Collect several labels and keep them all. The spread of opinions is itself useful data for subjective tasks.
  • Add a confidence field so annotators can mark items they are unsure about.
  • Exclude them from the test set if the test needs clear right answers and say so in your dataset notes.

The key is to decide on purpose, write the decision down and apply it consistently.

Fast Answers to Every Question

At AnnoTrue, annotators flag unclear items instead of guessing and every new rule is added to your guidelines and shared with the whole team. See how it works with a free pilot.