There are six broad families of data annotation, one per kind of data: text, documents, images, video, audio and AI model outputs. Within each family, the type you choose depends on what the model must learn, from one tag per item to detailed markup of every part of it. More detail means better information for the model, but also more time and cost per item.
| Family | Main annotation types |
|---|---|
| Text | Classification, sentiment, named entities, intent, relations, span highlighting |
| Documents | Field extraction, page classification, table extraction, key-value pairs |
| Images | Tags, bounding boxes, polygons, segmentation, keypoints |
| Video | Object tracking, event and action labels |
| Audio | Transcription, speaker labels, sound events, language and emotion tags |
| AI outputs | Rating, ranking, comparison, writing ideal answers, red-teaming |
Text Annotation
Text annotation teaches models to understand written language. It is the most common starting point, because so much business data is text: emails, chats, reviews, tickets, contracts.
| Type | What the annotator does | Example |
|---|---|---|
| Classification | Picks one or more categories for the whole text | "Where is my refund?" → Billing |
| Sentiment | Marks the feeling expressed | "Delivery was late but the staff were lovely" → Mixed |
| Intent | Marks what the writer wants to do | "Can I change my flight to Friday?" → Change booking |
| Named entities | Highlights names, places, dates, amounts | "Maria Ingram paid $450 on 03.03.2026" |
| Relations | Links entities that belong together | Links "$450" to "Maria Ingram" as "amount paid by" |
| Span highlighting | Marks the exact words that support a label | Highlights "late" as the reason for a complaint |
Where it's used: routing support tickets, moderating content, searching documents, training chatbots and extracting facts from reports.
What makes it hard: sarcasm, mixed feelings, slang and texts that fit two categories at once. Good guidelines decide these cases in advance. More in Text Annotation: Classification, Entities and Sentiment and Named Entity Recognition Annotation: A Practical Guide.
Document Annotation
Document annotation combines text with layout. The model must learn not only what words say, but where they sit on the page: the total at the bottom of an invoice, the signature block of a contract, a value inside a table.
| Type | What the annotator does | Example |
|---|---|---|
| Field extraction | Marks the value of each required field | Invoice number, date, supplier, VAT, total |
| Key-value pairs | Links a label on the page to its value | "Due date:" → "30.11.2026" |
| Table extraction | Marks rows, columns and cells | Each line item on a purchase order |
| Page classification | Tags each page or document by type | Invoice, receipt, contract, ID card |
| Handwriting and stamps | Transcribes or flags non-printed content | A handwritten amount next to a stamp |
Where it's used: invoice and receipt processing, contract review, insurance claims, loan applications and customer onboarding.
What makes it hard: poor scans, different layouts from every supplier, numbers that appear several times and sensitive personal data on many pages. More in Document Annotation: Extracting Data From Invoices and Forms.
Image and Video Annotation
Image annotation teaches models to see. The types differ mainly in how precisely they mark where something is and precision costs time.
| Type | What the annotator does | Example | Relative effort |
|---|---|---|---|
| Image tags | Picks labels for the whole image | "Damaged" or "Not damaged" | Lowest |
| Bounding boxes | Draws a rectangle around each object | A box around every car in a street photo | Low to medium |
| Polygons | Traces the outline of each object with points | The exact shape of a dent on a car door | Medium to high |
| Segmentation | Assigns every pixel to an object or class | Road, pavement, sky and people in a driving scene | Highest |
| Keypoints | Marks specific points on an object | Joints on a person's body, corners of a document | Medium |
Video annotation adds time. Annotators track the same object from frame to frame, keep its identity when it disappears and comes back and label events such as "person enters" or "car stops". Tools often interpolate between frames, so annotators correct key frames rather than draw every one.
Where it's used: quality inspection in factories, retail shelf monitoring, medical imaging, maps, agriculture and driver assistance.
What makes it hard: objects that overlap or are partly hidden, unclear edges, small objects and keeping box tightness consistent across a team. More in Image Annotation Types: Bounding Boxes, Polygons and Segmentation.
Audio Annotation
Audio annotation teaches models to understand speech and sound. Most of it depends on careful listening and strong language skills.
| Type | What the annotator does | Example |
|---|---|---|
| Transcription | Writes down exactly what is said, following format rules | "um I'd like to [inaudible] my order" |
| Speaker labels | Marks who speaks when | Agent 0:00–0:12, Customer 0:12–0:31 |
| Time stamps | Aligns text with the moment it is spoken | Each sentence linked to its start time |
| Sound events | Tags non-speech sounds | Doorbell, glass breaking, music |
| Language and accent | Tags the language, dialect or accent | English, Southern US accent |
| Emotion or tone | Tags how something is said | Calm, frustrated, urgent |
Where it's used: voice assistants, call-centre analytics, subtitles, meeting notes and speech recognition for less common languages.
What makes it hard: background noise, people talking over each other, names and technical terms and deciding how to write hesitations and mistakes. More in Audio and Speech Transcription for AI: What to Know.
AI Output Annotation
This family is about judging and improving what AI models produce, mostly the answers of language models. It became central once chat assistants arrived and it needs reasoning more than speed.
| Type | What the annotator does | Example |
|---|---|---|
| Rating | Scores one answer on set dimensions | Accuracy 2/5: the answer gives the wrong VAT rate |
| Comparison | Chooses the better of two answers and explains why | A is better: B ignores the requested word limit |
| Ranking | Orders several answers from best to worst | Ranks four summaries of the same article |
| Writing ideal answers | Writes the response the model should have given | A clear, correct answer to a customer's tax question |
| Rewriting | Fixes a flawed answer | Corrects the facts and tone of a draft reply |
| Red-teaming | Tries to make the model fail safely and records how | Prompts that trick the model into giving unsafe advice |
Where it's used: fine-tuning and RLHF, model evaluation before each release, safety testing and checking AI features in products.
What makes it hard: convincing answers that are subtly wrong, expert topics and the temptation to judge style over substance. Expert annotators are often needed. More in What Is RLHF?, How to Evaluate LLM Outputs and Writing Fine-Tuning Data.
How to Pick the Right Type
Choose the simplest annotation that gives the model what it needs. Three questions settle most cases:
- Does the model need to know where, or only what? If a yes or no per item is enough, don't pay for boxes or highlighted spans.
- How precise must "where" be? A box is often enough to count or find objects; outlines are needed to measure size or shape.
- Does judging the item need expertise? Medical, legal and technical content needs qualified annotators, whatever the annotation type.
When unsure, run a small pilot with two levels of detail and compare model results. It is cheaper than labeling the whole dataset twice.
Not sure which type you need?
Tell us what your model should do and we'll suggest the simplest approach, then prove it on your data with a free pilot, delivered within 10 working days.