Audio and Speech Transcription for AI: What to Know

By Stefan, founderData types and tasks4 min read

Transcription for AI turns recorded speech into text that follows strict, consistent rules, so a speech recognition model can learn exactly how sounds map to words. Unlike transcription for people to read, it often keeps the hesitations, repetitions and mistakes, because the model needs to learn speech as it really is.

The key decisions are the transcription style, the format rules for everything that isn't a plain word and how quality is measured.

Verbatim or clean verbatim?

The first decision is how literally to write what people say.

Style What it keeps Typical use
Full verbatim Every word, filler, false start, repetition and stutter Training speech recognition on natural speech
Clean verbatim The words as spoken, minus fillers and false starts; grammar not corrected Call analytics, meeting notes, subtitles
Edited Meaning only; grammar and wording tidied Summaries and documents for people to read; rarely used for training speech models

The same sentence in each style:

  • Full verbatim: "um so I I wanted to, uh, ask about the the fee on my card"
  • Clean verbatim: "So I wanted to ask about the fee on my card"
  • Edited: "The caller asked about a card fee."

Many speech recognition projects use full verbatim or a defined version of it. Whatever you choose, write it down with examples, because annotators naturally drift toward cleaning up speech.

Format Rules to Decide in Advance

Everything that isn't a clear, ordinary word needs a rule. These are the usual ones:

Situation Example rule
Unclear speech Write [inaudible]; never guess
Partly clear word Write your best guess in brackets with a question mark: [card?]
Speaker changes New line with a label: Agent:, Customer:
Overlapping speech Mark with [overlap] and transcribe both speakers
Non-speech sounds Tags from a fixed list: [laughter], [noise], [music], [silence]
Numbers Write as spoken: "fifteen euro", not "€15", so the model learns the sounds
Spelled-out letters Separate with spaces: "I O N E S C U"
Foreign words Write in their own spelling and tag the language if needed
Cut-off words Mark with a dash: "I wan- I wanted"
Timestamps At each speaker change, or every set number of seconds

Numbers deserve special care. For speech models, writing numbers as words teaches the link between sound and text. For analytics, digits may be more useful. Choose one, because mixed styles confuse both people and models.

Speaker labels need their own rules: generic labels (Speaker 1, Speaker 2) or roles (Agent, Customer) and what to do when you can't tell who is speaking.

Measuring Transcription Quality

The standard measure is the word error rate (WER): the number of word mistakes compared with a correct reference transcript, divided by the number of words in the reference.

latex WER = \frac{S + D + I}{N}

where S is substituted words, D deleted (missed) words, I inserted (extra) words and N the number of words in the reference. Lower is better; 0 means a perfect match.

Example. The reference is "I wanted to ask about the fee on my card" (10 words). A transcript reads "I want to ask about fee on my credit card". It has one substitution ("want" for "wanted"), one deletion ("the") and one insertion ("credit"). WER = (1 + 1 + 1) / 10 = 30%.

Other useful checks:

  • Tag accuracy: are [inaudible], [noise] and speaker labels used correctly?
  • Speaker attribution: is each line given to the right speaker?
  • Timestamp alignment: do timestamps match where the words are actually spoken?
  • Format compliance: numbers, spelling and punctuation follow the guidelines.

WER treats all errors as equal, but some matter more than others: a wrong account number is worse than a missed "um". Track important words separately when they matter to your use case.

Languages, Accents and Privacy

Use native speakers. Accents, dialects, slang and local names are where non-native transcribers make the most mistakes. For a call-center dataset from the American South, you want transcribers who grew up hearing how people in Texas, Georgia and Alabama actually speak.

Represent the people who will use the system. If the training data includes only one accent, the model will work worse for everyone else. Plan the sample across ages, regions, accents and recording conditions.

Voices and recordings are personal data. A voice can identify a person and calls often contain names, addresses and card numbers. Before transcription:

  • remove or bleep segments the model doesn't need
  • agree whether transcribers type personal details or replace them with tags such as [NAME] or [CARD_NUMBER]
  • if a voice is used to identify who is speaking, that is biometric data under the GDPR, which needs extra legal care

See GDPR and AI Training Data and How to Anonymize Data Before Sending It for Annotation.

Native-Speaker Transcription

AnnoTrue works with native speakers, follows your format rules exactly and reports quality with every batch. Try it on your recordings with a free pilot.