How to Run a Data Labeling Pilot (and What to Measure)

By Stefan, founderThe process, step by step4 min read

A data labeling pilot is a small, real run of your project, usually a few hundred items, done before you commit to the full volume. It answers four questions with numbers: are the guidelines clear, how accurate is the work, how fast does it go and what will the full project really cost?

A pilot is cheap insurance. Every problem it finds would otherwise be multiplied by the size of your dataset. And if you are choosing a vendor, a pilot on your own data tells you more than any sales call.

Planning the Pilot

Decide three things before any item is labeled.

1. What you want to learn. Write down the questions the pilot must answer. Typical ones:

  • Can annotators apply our guidelines consistently?
  • Which labels get confused with each other?
  • How many items per hour can a trained annotator do?
  • Is this vendor's quality and communication good enough to scale with?

2. The sample. Choose items that look like your real data, not the cleanest ones. Include:

  • the common cases, in roughly their real proportions
  • some rare and difficult items you already know about
  • a set of gold items with known correct answers, labeled carefully by your own expert

For most text tasks, a few hundred items is enough to measure speed and spot guideline problems. For slow tasks, like long AI answers or detailed image outlines, fewer items can work, as long as they cover the same variety.

3. Success criteria. Agree in writing what "good enough to scale" means, for example:

Measure Example target
Accuracy on gold items Your agreed level, e.g. 95%
Agreement between annotators A level that matches the task's difficulty
Open guideline questions All answered and added to the guidelines
Turnaround Delivered within the agreed days

The example targets are only examples. Set yours from what the model needs: a test set needs higher accuracy than a first training set.

Running the Pilot, Step by Step

  1. Sign the paperwork. NDA and, if there is personal data, a data processing agreement. See GDPR and AI Training Data.
  2. Prepare the data. Remove personal data the task doesn't need; set up access in the tool.
  3. Brief the team. Walk annotators through the guidelines and the purpose of the labels.
  4. Label a warm-up set. 20 to 50 items, fully reviewed, with feedback. Don't count these in the results.
  5. Label the pilot sample. Mix in the gold items without telling annotators which they are. Have a share of items labeled by two people independently.
  6. Log every question. Every question and flagged item is a finding about the guidelines.
  7. Review. A reviewer checks a sample of every annotator's work.
  8. Report. Put the numbers, the open questions and the recommended guideline changes in one short report.
  9. Meet and decide. Go through the report together and agree on next steps.

A small pilot usually takes one to two weeks from data access to report, depending on the task and how fast questions get answered.

What to Measure and What a Pilot Report Looks Like

Measure How it is calculated What it tells you
Accuracy on gold items Share of gold items labeled correctly Whether the guidelines are applied correctly
Agreement between annotators Share of overlapping items with the same label, or a statistic such as Cohen's kappa Whether the guidelines are clear enough to give the same answer
Errors by label Which labels are mistaken for which others Labels to define better, merge or split
Speed Items per hour, after the warm-up Real cost and timeline of the full project
Flag rate Share of items flagged as unclear How much the guidelines still leave open
Questions List of questions, with answers Rules to add to the guidelines

A sample pilot report (illustrative numbers, for a project sorting 500 support emails into 8 topics):

Item Result
Items labeled 500, plus 50 overlap items
Accuracy on 40 gold items 93% (37 of 40)
Agreement on overlap items 88%
Most confused labels "Billing dispute" vs. "Fees and pricing" (11 of 23 disagreements)
Speed 115 emails per hour after warm-up
Flagged items 4%, mostly emails in two languages
Recommended changes Add a decision rule for fee questions vs. disputes; add a rule for mixed-language emails
Estimated full project 20.000 emails ≈ 174 labeling hours, plus overlap and review

One short page like this is enough to decide what to do next.

Reading the Results: Go, Fix, or Stop

Every pilot ends with one of three decisions.

Decision When Next step
Go Targets met, questions answered, small guideline fixes only Update the guidelines, agree price and timeline, scale up
Fix and repeat Accuracy or agreement below target, mostly on a few labels Rewrite the weak rules, merge confused labels, run a second small round
Stop or rethink Low agreement everywhere, or the task can't be defined clearly Revisit the goal and the label set; the problem is the task, not the people

Look at errors, not just scores. A 93% score can hide a pattern, such as one label that is almost always wrong. The confused-labels list is usually the most useful part of the report.

Judge the vendor too. Did they ask good questions? Was the report honest about problems? Did they meet the deadline? How a vendor behaves in a small pilot is how they will behave at scale.

Use the measured speed for the price. A quote based on measured speed is far more reliable than one based on a guess. See How Much Does Data Labeling Cost?

Try a Free Pilot

AnnoTrue pilots are free and delivered within 10 working days, one per company, with an NDA signed first. You get the labeled sample and a report like the one above.