How to Evaluate LLM Outputs: Rubrics, Ratings and Rankings

By Stefan, founderData types and tasks4 min read

To evaluate a language model's answers well, people judge them against a written rubric, in one of three formats: rating a single answer on several dimensions, comparing two answers and picking the better one, or ranking several answers from best to worst. The rubric defines what "good" means, such as correct, complete, safe and clear and in what order those qualities matter.

Good evaluation data does two jobs: it measures how well a model works before you ship it and it becomes training data that makes the next version better.

The Three Formats

Format How it works Strengths Weaknesses
Rating Score one answer on each dimension, e.g. 1–5 for accuracy, helpfulness, safety Detailed; shows why an answer is weak; good for tracking a model over time People use scales differently; one person's 4 is another's 3
Pairwise comparison Show two answers to the same prompt; pick the better one and say how much better Easier and more consistent than absolute scores; the format used for preference training Says which is better, not whether either is good enough
Ranking Order three or more answers from best to worst Captures more preferences per prompt Slower; middle positions are often unclear

Use ratings for testing whether a model is good enough to ship and for finding which dimension is weak.

Use comparisons for preference data (RLHF, DPO) and for choosing between two model versions. See What Is RLHF?

Use rankings when you generate several answers per prompt and want the most preference information per prompt.

Whatever the format, ask for a short written reason. It lets reviewers check the judgment and tells model builders what to fix.

Building a Rubric

A rubric turns "is this a good answer?" into questions people can answer the same way. It needs three things: dimensions, a scale with anchors and a priority order.

A sample rubric for a customer-support assistant:

Dimension Question 1 (poor) 3 (acceptable) 5 (excellent)
Correctness Are all facts, numbers and steps right? One or more serious errors Minor imprecision, nothing misleading Fully correct
Instruction following Does it do what was asked, in the requested format? Ignores the request Mostly follows it Follows it exactly
Completeness Does it cover what the user needs? Misses the main point Covers the main point Covers everything needed, nothing extra
Safety and policy Does it avoid harmful, private or off-policy content? Clear violation Borderline No issues
Clarity and tone Is it clear and in the right tone? Confusing or rude Clear enough Clear, concise, right tone

Anchor every score. Describe what a 1, a 3 and a 5 look like, ideally with a real example of each. Unanchored scales drift from person to person.

Set a priority order. For example: correctness and safety first, then instruction following, then completeness, then clarity. A fluent answer with a wrong number should lose to a plain answer that is right.

Define "fail" conditions. Some problems make an answer unacceptable whatever its other scores: a made-up source, unsafe advice, leaking personal data. List them and make them override everything else.

Human judges or AI judges?

Many teams now use a strong language model to grade other models' answers, often called "LLM-as-a-judge". It is fast and cheap and it can be useful. A well-known 2023 study (Zheng and others) found that a strong model judge agreed with human preferences over 80% of the time on general chat questions, about as often as humans agreed with each other.

The same study also documented the judges' weaknesses:

  • Position bias: preferring the answer shown first (or second), regardless of quality
  • Verbosity bias: preferring longer answers
  • Self-enhancement bias: favouring answers that resemble the judge model's own style
  • Limited reasoning: missing errors in maths and logic

Where people are still needed:

  • Specialist content. Only a lawyer can reliably spot a wrong legal answer that sounds right; the same goes for medicine, finance and code.
  • Setting the standard. Human judgments are what AI judges are checked against. Without a human-labeled set, you can't know if your AI judge is any good.
  • New or sensitive topics, where the judge model may share the blind spots of the model being judged.
  • Training data for preferences, where errors would be learned and amplified.

A common setup combines both: an AI judge screens large volumes and people evaluate a sample, the hard cases and everything specialised and keep checking the AI judge against human scores.

Running a Human Evaluation Project

  1. Collect realistic prompts, including hard, ambiguous and adversarial ones, not just easy questions.
  2. Write the rubric, with anchored scales, a priority order and fail conditions.
  3. Randomise the order of answers in comparisons, so position doesn't bias people either.
  4. Hide which model wrote which answer.
  5. Give annotators time and tools to check facts. Rushed evaluators reward confident answers.
  6. Route specialist prompts to experts. See Expert Annotation.
  7. Double-rate a share of items and measure agreement per dimension; see Inter-Annotator Agreement Explained Simply.
  8. Read the written reasons, not just the scores. They show patterns no number reveals.

Human Evaluation, Done Carefully

AnnoTrue evaluators are tested on fact-checking and rubric use and experts handle specialist prompts. Every judgment comes with a written reason, by a person, never by AI. Try it with a free pilot.

Sources