Named Entity Recognition Annotation: A Practical Guide

By Stefan, founderData types and tasks4 min read

Named entity recognition (NER) annotation means highlighting the names, dates, amounts and other specific items in a text and labeling each with its type, so a model can learn to find them on its own. Success depends on three decisions made before anyone starts: which entity types you need, exactly where each entity starts and ends and how to handle entities inside other entities.

NER sounds mechanical, but small inconsistencies, such as including a title in one name and not in another, add up across thousands of documents and confuse the model.

Designing the Entity Schema

The schema is the list of entity types and their definitions. Start from what the model will do with the entities, not from every type you can think of.

A typical general-purpose schema:

Type Definition Examples Not this type
PERSON A named, real or fictional individual Maria Ingram, Dr Porter "the customer", "my manager"
ORGANIZATION A named company, institution or group Banca Transilvania, the EU Commission "the bank" without a name
LOCATION A named place Austin, Michigan Avenue "home", "the office"
DATE A specific date or period 03.03.2026, Q2, next Friday "recently"
AMOUNT A sum of money with its currency $450, $1.200 "a lot of money"
ID An account, policy, order or reference number Order 48213, IBAN phone numbers, if you want them separate

Rules for a good schema:

  • Each type gets a definition, examples and "not this" examples. The "not this" column prevents most errors.
  • Avoid overlapping types. If a hospital could be both ORGANIZATION and LOCATION, decide which wins, or define it by context.
  • Add domain types only when needed. A legal project may need PARTY, COURT and CLAUSE_TYPE; a medical one DRUG, DOSE and CONDITION.
  • Keep the list short. Every type you add slows annotation and invites confusion.

Boundary and Nesting Rules

Most NER disagreements are not about the type, but about where the entity starts and ends. Decide these rules in writing, with examples:

Question Example A common rule
Titles and honorifics "Dr Maria Ingram" Exclude titles: tag "Maria Ingram"
Currency "$450", "450 dollars" Include the currency: "$450", "450 dollars"
Punctuation "Ingram," Never include trailing punctuation
Legal forms "Acme LLC" Include legal forms: "Acme LLC"
Possessives "Maria's account" Tag "Maria" only
Dates in pieces "from 3 to 05.03.2026" Tag as one DATE span, or two; decide and stay consistent
Abbreviations "BT" for Banca Transilvania Tag as ORGANIZATION if it clearly refers to it
Pronouns "she", "the company" Don't tag, unless the project includes coreference

Nested entities are entities inside other entities: "University of Chicago" is an organisation containing a location. Most tools and models handle either flat tagging (one span, the outer entity) or nested tagging (both). Choose one:

  • Flat: tag only the longest entity. Simpler, faster, enough for most projects.
  • Nested: tag both. Needed when the model must find the inner entity too, for example every city mentioned anywhere.

Whatever you choose, the same rule must apply to every annotator and every document.

Measuring NER Quality

NER quality is measured by comparing an annotator's spans with gold spans, using three numbers:

  • Precision: of the entities the annotator tagged, how many are correct?
  • Recall: of the entities that should have been tagged, how many did the annotator find?
  • F1: a single score that balances the two (their harmonic mean).

Example. A document has 10 gold entities. The annotator tags 9 spans and 8 of them match a gold entity exactly, with the right type.

Measure Calculation Result
Precision 8 correct / 9 tagged 0.89
Recall 8 correct / 10 in gold 0.80
F1 2 × 0.89 × 0.80 / (0.89 + 0.80) 0.84
Exact or partial match?

An exact match needs the same start, end and type. A partial match gives credit when spans overlap, for example "Dr Maria Ingram" against "Maria Ingram". Report both: a big gap between them means your boundary rules need work, not your annotators.

Look at errors by type. If recall on DATE is low, annotators may be missing relative dates such as "next Friday". If precision on ORGANIZATION is low, they may be tagging unnamed references like "the bank". Each pattern points to a specific rule to fix.

NER in Specialised Fields

General entity types are rarely enough for professional documents. Each field brings its own types and its own need for annotators who understand them.

Field Typical entity types Why expertise matters
Legal Parties, courts, dates of effect, clause types, obligations, amounts Telling a party from a mere mention needs legal reading skills
Medical Conditions, drugs, doses, procedures, body parts, test results Abbreviations and context change meaning; patient data is highly sensitive
Finance Instruments, tickers, amounts, rates, counterparties, dates Similar-looking numbers mean very different things
HR and recruiting Skills, job titles, employers, degrees, dates Varied CV formats and job-title synonyms

For medical and other sensitive texts, personal data must be removed or tightly controlled before annotation; see GDPR and AI Training Data. For domain experts, see Expert Annotation.

Precise Spans, Measured

AnnoTrue reports precision, recall and F1 per entity type with every batch, so you see exactly where the data is strong and where it needs work. Try it with a free pilot.