Document annotation marks where each piece of information sits on a page, such as the invoice number, the total, the supplier and each line item, so a model can learn to extract them automatically. It combines reading with layout: the model must learn not just what a number says, but which number on the page is the total.
The work is detailed and rule-heavy. Documents from different senders look different, the same value often appears several times and scans are rarely perfect. Clear rules for these situations are what make the data useful.
What Gets Annotated
| Element | What the annotator does | Invoice example |
|---|---|---|
| Fields | Marks the value of each required field, usually with a box on the page and the text | Invoice number, issue date, due date, supplier, buyer, VAT number, total |
| Key-value pairs | Links a printed label to its value | "Amount due:" → "$1.250" |
| Tables | Marks rows, columns and cells and links cells to their headers | Each line item: description, quantity, unit price, VAT, amount |
| Page or document type | Classifies each page or document | Invoice, credit note, receipt, delivery note, contract |
| Special content | Flags or transcribes handwriting, stamps, signatures | A handwritten correction next to the total |
Most projects combine several of these. An accounts-payable model, for example, needs the document type, a set of header fields and the full line-item table.
Decide the output format early. Some projects need only the extracted value ("1250.00"); others also need its position on the page, so the model learns where to look. Position data takes longer to annotate, but layout-aware models need it.
The Rules That Matter Most
Document guidelines live or die on a handful of decisions. Write each one down with an example.
| Situation | Example | Rule to decide |
|---|---|---|
| A value appears several times | The total is printed in the table and again at the bottom | Which occurrence to mark: the first, the last, the one next to the label? |
| Several similar values | Subtotal, VAT, total and amount paid | Exact definitions, e.g. "Total = the final amount payable including VAT" |
| Value split across lines | A supplier address on three lines | One span covering all lines, or one per line |
| Number formats | "1.250,00" vs "1,250.00" | Mark as printed; normalise in a separate field if needed |
| Dates | "03.04.2026" could be 03.04.2026 or 04.03.2026 | Use the document's country convention; flag if unclear |
| Missing value | No due date on the invoice | Leave empty and mark as not present, never guess |
| Unreadable value | A smudged total | Mark as unreadable, don't infer it from other numbers |
| Corrections | A printed amount crossed out and rewritten by hand | Which one counts: usually the correction, flagged |
"Not present" and "unreadable" are different. The first tells the model the document has no such field; the second tells it the field exists but can't be read. Mixing them confuses the model.
Messy Documents and Personal Data
Real documents are messy. Plan the sample and the rules for:
- phone photos taken at an angle, with shadows or cut-off edges
- faint, low-resolution or skewed scans
- multi-page documents where the table continues on the next page
- layouts that change from one supplier to the next
- several languages, sometimes on the same page
- stamps and signatures over printed text
A training set built only from clean PDFs produces a model that fails on the first phone photo. Include the messy cases in proportion to how often they really occur.
Documents are full of personal data. Invoices, contracts and forms carry names, addresses, ID numbers, bank details and signatures. Before annotation:
- remove or mask what the model doesn't need to learn
- if the fields must stay (because the model must find them), limit access to a small, trained team and block downloads
- sign an NDA and a data processing agreement with any vendor
See How to Anonymize Data Before Sending It for Annotation and GDPR and AI Training Data.
Quality Checks for Document Annotation
Documents allow some checks that other data types don't:
- Arithmetic checks: line items should add up to the subtotal and subtotal plus VAT to the total. A mismatch points to a wrong field or a misread number.
- Format checks: dates are valid dates, VAT numbers match the country's format, totals are numbers.
- Cross-field checks: the due date comes after the issue date.
- Field-level accuracy on gold documents, reported per field, since some fields are much harder than others.
- Reviewer sampling focused on the hard fields and the messy documents.
These automatic checks catch many errors before a reviewer ever looks. See Quality Control in Data Labeling.
Documents Handled with Care
AnnoTrue annotates invoices, receipts, contracts and forms with field-level quality reports, under an NDA and a data processing agreement, with data stored in the EU. See the quality before you commit, with a free pilot.