The Data Labeling Process: A Step-by-Step Guide

By Stefan, founderPillar guide · The process, step by step11 min read

A data labeling project has nine steps: define the goal and labels, write guidelines, prepare and protect the data, set up the tool, select and train annotators, run a pilot, scale up with quality control, deliver and accept batches, then close the loop. The pilot is the step teams most often skip and skipping it is the most expensive mistake in the whole process.

This guide walks through each step: what to do, what to decide and what usually goes wrong. It is written for teams labeling data in-house and for teams working with a vendor, since the steps are the same either way.

The nine steps at a glance

  1. Define the goal and the label set
  2. Write the annotation guidelines
  3. Prepare and protect the data
  4. Choose the tool and set up the workflow
  5. Select and train annotators
  6. Run a pilot
  7. Scale up with quality control running
  8. Deliver, accept and fix batches
  9. Close the loop

If you are new to the topic, start with What Is Data Labeling? The Complete Guide for AI Teams.

Step 1: Define the Goal and the Label Set

Start from what the model must do, not from the data you happen to have. A clear goal decides which labels you need, how detailed they must be and how good is good enough.

Answer these four questions in writing before anything else:

  • What decision will the model make? For example: "Route each incoming support email to the right team."
  • What labels does it need to learn that? List them, with a one-line definition each.
  • What does one item look like? One email, one page, one image, one conversation?
  • What accuracy do you need? A model that suggests labels for a human to confirm can tolerate more errors than one that acts alone.

Keep the label set as small as the goal allows. Every extra label adds confusion with its neighbours and slows the work. If two labels lead to the same action downstream, merge them. You can always split a label later; untangling a confused one is harder.

Decide the level of detail. "Does this image contain a damaged part?" is a yes or no. "Where exactly is the damage?" needs a box or an outline per item and takes many times longer. Choose the simplest form that serves the goal.

Plan for "none of the above". Real data always contains items that fit no label: spam, blank pages, another language. Give annotators a label for these, or they will force items into the closest wrong one.

Step 2: Write the Annotation Guidelines

Guidelines are the single biggest lever on label quality. Two annotators with the same clear guidelines will agree far more often than two experts working from memory.

Good guidelines contain:

  1. The purpose, in two sentences. Annotators make better calls on unclear items when they know what the labels are for.
  2. Each label, defined. A short definition, then at least two clear examples and one example of what it is not.
  3. Decision rules for overlaps. "If an email mentions both a blocked card and a refund, label it Card problem."
  4. Edge cases. The unusual items you already know about and the rule for each.
  5. What to do when unsure. A flag, a "not sure" label or a question channel and never a guess.
  6. Format rules. How tight a box should be, whether to include punctuation in a highlighted name, how to write dates.
  7. A change log. Rules change after the pilot. Every change gets a date, so everyone knows which version applied to which batch.

Write for a careful newcomer. Short sentences, one rule per line, real examples from your own data. If a rule needs a paragraph to explain, it probably needs an example instead.

Test the guidelines before the pilot. Ask two people who didn't write them to label 20 items. Every question they ask is a gap in the document.

A template and a worked example are in How to Write Annotation Guidelines That Annotators Can Follow.

Step 3: Prepare and Protect the Data

Only send annotators what they need to see and agree in writing how the data is handled before anyone opens it.

Select a representative sample. The items you label should look like the data the model will meet in real use: the same mix of topics, languages, image quality and rare cases. A dataset of only clean, easy items trains a model that fails on messy real ones.

Remove personal data you don't need. Names, emails, phone numbers, account numbers and faces can often be removed or replaced without changing the task. If the task needs them, for example when the model must find names, limit who can see the data and for how long. See How to Anonymize Data Before Sending It for Annotation.

Put the paperwork in place first. With an outside vendor, that means an NDA and, if the data contains any personal data from the EU, a data processing agreement under the GDPR. See GDPR and AI Training Data.

Agree on access and deletion. Write down:

  • where the data is stored and in which country
  • who can access it and whether they can download or copy it
  • how long it is kept after the project ends and how its deletion is confirmed

Split off a test set early. Keep a portion of the labeled data aside for testing the model and never train on it. Otherwise you will measure how well the model remembers, not how well it works.

Step 4: Choose the Tool and Set Up the Workflow

The right tool is the one that fits your data type and your security rules and that annotators can use quickly. If your team already uses an annotation tool, a good vendor should be able to work inside it.

What to check in a tool:

Need Question to ask
Data types Does it handle your format well: long documents, PDFs, video, audio, chat conversations?
Label types Does it support what you need: classes, spans, boxes, polygons, rankings, free text?
Quality features Can it mix in gold items, send items to two annotators and route work to reviewers?
Security Where is data stored? Can you control who sees what and block downloads?
Speed Are there keyboard shortcuts and pre-filled options? Small delays add up over thousands of items.
Export Can you export labels in the format your training code expects?

Open-source tools such as Label Studio cover most text, image and audio tasks and can be hosted on your own servers. Commercial platforms add more automation and project management.

Set up the workflow, not just the tool. Decide how items flow: who labels, what share goes to a second annotator, who reviews and where flagged items go. Write it down so the whole team follows the same path.

More detail is in How to Choose a Data Annotation Tool.

Step 5: Select and Train Annotators

The people doing the labeling matter more than the tool. Choose them for the task, test them before they touch real data and train them on your guidelines.

Match people to the task. Most text and image tasks need careful generalists with strong reading skills and attention to detail. Some need more:

  • Language skills: native or near-native speakers for any task that depends on tone, slang or grammar.
  • Subject experts: lawyers for contract clauses, doctors or nurses for clinical notes, engineers for code. See Expert Annotation: How Lawyers, Doctors and Engineers Train AI.
  • Cultural context: local knowledge for content moderation, names and addresses.

Test before real work. A short task built from your own data, with known answers, shows quickly who follows the guidelines. It also tells you which parts of the guidelines people misread.

Train on your guidelines, then check early work closely. Walk new annotators through the guidelines with examples, then review all of their first items, not a sample. Feedback in the first days shapes the habits that last for the whole project.

Keep the same people. Annotators get faster and more accurate as they learn your data. A team that changes every week keeps paying the learning cost again. Fair pay and steady work keep people on a project; ask any vendor how they handle both.

Step 6: Run a Pilot

A pilot is a small, real run of the project, usually a few hundred items, done before you commit to the full volume. It is the cheapest place to find problems, because every problem found later is multiplied by the size of the dataset.

What to measure in a pilot:

Measure What it tells you
Accuracy on gold items Whether annotators apply the guidelines correctly
Agreement between annotators Whether the guidelines are clear enough for people to reach the same answer
Errors by label Which labels are confused with each other and need better definitions or merging
Items per hour The real speed, which gives you the real cost and timeline of the full project
Questions and flags Gaps in the guidelines and edge cases nobody planned for

Fix, then measure again if needed. Most pilots end with changes to the guidelines: a merged label, a new rule, extra examples. If the changes are large, run a second small round before scaling.

Use the pilot to judge a vendor. A pilot shows how a vendor communicates, how fast it answers questions and how honest its quality report is. That is worth more than any sales call.

A checklist and a sample pilot report are in How to Run a Data Labeling Pilot (and What to Measure).

Step 7: Scale Up with Quality Control Running

When the pilot numbers are good, add volume and people, but keep the quality checks running all the time. Quality drops quietly as projects grow: new people join, fatigue sets in and rare cases pile up.

A solid setup uses several layers, because each catches different problems:

  1. Automatic checks catch mechanical errors at once: empty fields, wrong formats, boxes outside the image, tasks done impossibly fast.
  2. Hidden gold items, a small share of the work, measure each annotator's accuracy without them knowing which items are tested.
  3. Overlap, where a share of items is labeled by two people independently, shows where the guidelines leave room for different answers.
  4. Reviewer sampling puts an experienced reviewer on a sample of every annotator's work and on all of a new annotator's first items.
  5. Batch acceptance checks each delivery against an agreed error limit before it leaves the team.

Act on what the checks show. A falling score for one person means feedback or retraining. A falling score for everyone on one label means the guidelines need fixing. Record each change in the guidelines' change log.

Keep a question channel open. Annotators will keep finding new edge cases. Answer them quickly, add the answer to the guidelines and share it with the whole team.

More on each method in Quality Control in Data Labeling: Methods That Work and Gold Standard Datasets: How to Build and Use Them.

Step 8: Deliver, Accept and Fix Batches

Deliver in batches, not all at once and agree beforehand how each batch is accepted or sent back. Small batches mean small surprises.

Every batch should come with a short quality report:

  • number of items and how many were flagged or skipped, with reasons
  • accuracy on gold items
  • agreement between annotators on overlapping items
  • reviewer findings: the most common errors and what was done about them
  • guideline changes that applied to this batch

Check a sample yourself. Take a random sample, label it or check it against the guidelines and count errors. If the error rate is within the agreed limit, accept the batch. If not, send it back with examples.

Agree the fix in advance. The contract or statement of work should say what happens when a batch fails: who fixes it, within what time and at whose cost. At AnnoTrue, if a delivered batch misses the agreed accuracy, we fix it free of charge within 5 working days.

Watch for drift between batches. Compare reports over time. A slow decline in agreement or a rising share of flags usually means new kinds of items are appearing in the data and the guidelines need to catch up.

Step 9: Close the Loop

A labeling project doesn't really end when the last batch is accepted. The model's results show where the data was weak and the data you no longer need should be deleted.

Learn from the model. After training, look at where the model makes mistakes. Errors often cluster around one label, one kind of item or one source of data. That tells you what to label next: more examples of the weak cases, rather than more of what the model already does well. This is the basic idea behind Human-in-the-Loop AI.

Keep the guidelines alive. Store the final guidelines, the change log and the gold items with the dataset. The next round of labeling, maybe months later and with different people, will need them to produce consistent labels.

Document the dataset. Record where the data came from, how it was sampled, which guideline version applied, who labeled it and what quality was measured. Under the EU AI Act, providers of high-risk AI systems must follow data governance practices that cover how training data was collected and prepared, including labeling (Article 10). See The EU AI Act and Training Data.

Delete what you agreed to delete. At the end of the project, the vendor should delete or return your data as agreed and confirm it in writing. Keep that confirmation.

Who Does What and How Long It Takes

The client owns the goal and the final word on the rules; the labeling team owns the people, the process and the quality numbers. The table shows a typical split of work for a project with an outside vendor. The durations are an example for a project of a few thousand text items, not a promise: real timelines depend on volume and difficulty.

Step Client Labeling team Example duration
1. Goal and labels Decides Advises 1–3 days
2. Guidelines Approves Drafts or improves 2–5 days
3. Data preparation Selects and anonymizes data; signs NDA and DPA Confirms storage and access rules 1–5 days
4. Tool and workflow Provides tool access, if using its own Sets up workflow and quality checks 1–2 days
5. Annotators Reviews requirements Selects, tests and trains 2–5 days
6. Pilot Reviews results, answers questions Labels sample, reports 1–2 weeks
7. Scale-up Answers new edge cases Labels with quality control Depends on volume
8. Delivery Checks samples, accepts or returns Delivers batches with reports; fixes failures Ongoing
9. Close the loop Analyzes model errors; requests deletion Deletes data, confirms in writing Days

Start with a Free Pilot

AnnoTrue runs the first steps of this process with you for free: we review your guidelines, label a sample within 10 working days and send a quality report. An NDA is signed before we see any data.

Sources