LLM evaluation metrics answer five questions: is the output correct, is it grounded in the source, is it useful, is it safe and in the right format, and can you afford it. Pick metrics by question, not by popularity.
BLEU and ROUGE rarely fit modern LLM apps. They reward word overlap, and a correct answer worded differently scores badly.
LLM-as-a-judge works, within limits. In the MT-Bench study, a strong judge agreed with human preferences over 80% of the time, about as often as humans agree with each other, but judges favor longer answers and their own style.
Small test sets produce noisy scores. On 100 test cases, a score of 80% has a margin of roughly plus or minus 8 points, so 82% against 85% may mean nothing.
A working LLM evaluation framework runs offline before release and online after it. Offline catches regressions; online catches the problems your test set never imagined.
A team changes one line in a prompt. The demo still looks fine. Two weeks later, support tickets show the assistant has started inventing refund rules that appear nowhere in the policy documents. Nothing measured that, so nothing caught it.
Most guides to LLM evaluation metrics list twenty scores and stop. The harder questions are which score catches which failure, whether an AI judge can be trusted to grade your outputs, and how many test cases you need before a number means anything.
This guide answers those. It groups LLM evaluation metrics by the question each one answers, matches them to common use cases, shows how to check an LLM judge against people, and ends with a working LLM evaluation framework a production team can run on every change.
What are LLM evaluation metrics?
LLM evaluation metrics are scores that measure the quality of a language model's outputs against defined criteria, such as correctness, grounding in source documents, relevance to the question, safety, format, speed and cost. Production teams use them to decide whether a prompt, model or retrieval change makes the product better or worse before users see it.
There are two layers of LLM evaluation, and mixing them up is a common mistake:
Model evaluation compares base models on public benchmarks such as MMLU or HumanEval. It tells you which model is generally stronger.
Application evaluation tests your product on your own tasks, data and rules. It tells you whether your assistant answers your customers correctly.
A model that tops a public benchmark can still fail your use case. This guide covers application evaluation, which is the one production teams own. If your system calls tools and takes actions, the agent-specific measures (tool selection, task completion, consistency across runs) are covered in our AI agent evaluation framework guide.
The 5 questions LLM evaluation metrics answer
Every useful metric answers one plain question. Group them this way and the choice gets easier.
1. Is the output correct? Accuracy metrics
Use these when a right answer exists and can be checked.
Exact match. The output equals the expected answer. Good for classification, short extraction and yes or no questions.
Precision, recall and F1. For extraction and labeling, these show whether the model misses items or adds wrong ones.
Execution checks. For code and SQL, run the output and test the result. A query that runs and returns the right rows beats any text similarity score.
Numeric tolerance. For calculations, accept answers within a set margin rather than requiring identical text.
2. Is the output grounded? Faithfulness and RAG metrics
Use these whenever the model answers from retrieved documents, which covers most business assistants.
Faithfulness. The share of claims in the answer that the source documents support. The Ragas paper (Es et al., 2023) defines it by splitting the answer into claims and checking each against the retrieved context. This is the main hallucination metric for RAG systems.
Context precision. Of the passages retrieved, how many were relevant to the question.
Context recall. Of the facts needed for a correct answer, how many the retrieved passages contained.
Hallucination rate. The share of answers containing at least one unsupported claim. Easier to explain to a stakeholder than an average faithfulness score.
These separate two failures that look identical to a user. A wrong answer with poor context recall is a retrieval problem. A wrong answer with good context and low faithfulness is a generation problem. Fixing the wrong one wastes weeks.
3. Is the output useful? Relevance and quality metrics
A grounded, correct answer can still miss the point.
Answer relevancy. Whether the response addresses the question asked, rather than a nearby one.
Completeness. Whether every part of a multi-part question got an answer.
Rubric scores. Criteria you write in plain language, such as "uses the customer's order details" or "gives a next step", scored by a judge model. G-Eval, from a 2023 Microsoft paper, is the best-known version of this method.
Semantic similarity. Scores such as BERTScore compare meaning rather than exact words, so a correct paraphrase is not punished.
4. Is the output safe and in the right format? Guardrail metrics
These are usually pass or fail, and a single failure can matter more than any average.
Format validity. Does JSON parse, does it match the schema, are required fields present. Any downstream system depends on this.
Toxicity and policy violations. Harmful, biased or off-brand content.
Sensitive data leakage. Personal data, internal notes or system prompt text appearing in outputs.
Refusal rate. How often the model declines. Too high frustrates users; too low on risky requests is a liability.
5. Can you afford it? Operational metrics
Quality that arrives too slowly or costs too much still fails in production.
Latency. Time to first token and total response time, measured at the 95th percentile, not the average.
Cost per request. Token spend per answer, including retries and judge calls.
Escalation rate. The share of conversations handed to a person, which turns quality into a staffing cost.
LLM evaluation metrics by use case
Most teams need four to six metrics, not twenty. Start from the use case.
Use case
Main metrics
Supporting metrics
Usually skip
RAG assistant or knowledge bot
Faithfulness, context recall, answer relevancy
Context precision, hallucination rate, latency
BLEU, ROUGE
Customer support chatbot
Answer relevancy, rubric score for policy and tone, escalation rate
Faithfulness, refusal rate
Exact match
Summarization
Faithfulness, completeness, rubric score
ROUGE as a rough check, length
Exact match
Classification or routing
Accuracy, precision and recall per class
Confusion matrix, cost per request
Judge scores
Data extraction
Field-level F1, format validity
Exact match per field
Semantic similarity
Code or SQL generation
Execution pass rate
Format validity, latency
BLEU, ROUGE
If a metric does not map to a failure that would hurt your users or your business, drop it. A dashboard of twenty scores trains people to ignore all of them.
Reference-based vs reference-free LLM evaluation
A reference-based metric compares the output to a known correct answer. A reference-free metric judges the output on its own, or against the source documents, with no ideal answer written in advance.
Type
How it works
Examples
Best for
Weakness
Reference-based
Compares output with a written correct answer
Exact match, F1, BLEU, ROUGE, BERTScore
Tasks with one right answer
Needs labeled data; punishes valid alternatives
Reference-free
Scores output against criteria or source context
Faithfulness, answer relevancy, rubric scores
Open-ended answers, live traffic
Depends on a judge model that can be wrong
Why BLEU and ROUGE fall short for LLM apps
BLEU (2002) and ROUGE (2004) count overlapping words with a reference text. They were built for machine translation and summarization research, where outputs were close to a reference. Modern LLMs phrase correct answers in many ways, so an accurate answer can score low and a fluent wrong one can score high. The G-Eval paper reports that these metrics correlate poorly with human judgment, especially on open-ended tasks. Keep them, if at all, as a cheap rough signal for summarization.
How to use LLM-as-a-judge without fooling yourself
LLM-as-a-judge means using a capable model to score another model's outputs against a rubric. It is the only practical way to grade open-ended answers at scale, and it is also the easiest place to fool yourself.
The research supports it with conditions. The MT-Bench study (Zheng et al., NeurIPS 2023) found that a GPT-4 judge agreed with human preferences more than 80% of the time, matching the rate at which people agreed with each other. The same paper documented three biases: judges prefer the answer shown first, prefer longer answers, and favor outputs from their own model family. The G-Eval paper reported a Spearman correlation of 0.514 with human ratings on summarization, well above older metrics, and still only moderate agreement.
Five steps make a judge trustworthy enough to gate releases:
Write narrow criteria. "Is every number in the answer present in the source?" is checkable. "Is this a good answer?" is not.
Use a scale with defined points, or pass and fail. Describe what a 1, 3 and 5 look like. Vague scales drift.
Calibrate against people. Have two team members label 50 to 100 outputs. Measure how often the judge agrees with them, and how often they agree with each other. If the judge agrees far less often than the people do, fix the rubric before trusting it.
Control known biases. Swap answer order in pairwise comparisons, cap or normalize length, and use a judge from a different model family than the one being tested where possible.
Re-check after changes. A new judge model or edited rubric is a new instrument. Recalibrate on the same labeled sample.
Code checks beat judges wherever they apply. If JSON validity, a required field or a banned phrase can be tested with code, test it with code. It is faster, cheaper and never changes its mind.
How many test cases do you need for LLM evaluation?
You need enough test cases that a real change in quality is larger than random noise. With small sets, most score differences between two prompts are noise.
The table shows the approximate margin of error, at 95% confidence, for a measured pass rate of 80%.
Test cases
Approximate margin of error
What it can reliably detect
50
About ±11 points
Only large changes, such as 80% falling to 60%
100
About ±8 points
Moderate changes; 82% against 85% is noise
200
About ±5.5 points
Most changes teams care about
400
About ±4 points
Small regressions on important flows
These figures use the standard binomial approximation and are illustrative. Three practical rules follow from them:
Start with 20 to 50 cases and grow. A small set built from real failures beats an empty one. Anthropic's January 2026 guidance on evals makes the same point: start small and grow the set from production problems.
Report the margin with the score. "84%, ±6" stops a team from celebrating noise.
Compare the same cases. Running old and new versions on an identical set and counting cases that flipped is far more sensitive than comparing two averages.
Offline vs online LLM evaluation
Offline evaluation runs a fixed test set before release. Online evaluation scores real traffic after release. You need both, because each catches what the other misses.
Offline evaluation
Online evaluation
When it runs
Before every prompt, model or retrieval change
Continuously on live traffic
What it uses
Curated test cases with expected outcomes
Sampled real conversations, user feedback, escalations
What it catches
Regressions on known cases
New question types, drift, problems no one predicted
Main metrics
Accuracy, faithfulness, rubric scores, format validity
Hallucination rate on samples, thumbs-down rate, escalation rate, latency, cost
The two connect. Every failure found online becomes a new offline test case, so the same bug cannot ship twice.
How to build an LLM evaluation framework
An LLM evaluation framework is the set of test cases, metrics, graders, thresholds and routines that decide whether a change can ship. Five steps produce a working one:
Write down the failures that matter. List the five or six ways the product could hurt a user or the business: invented policy, wrong figures, broken output format, unsafe content, slow responses.
Choose one metric per failure. Use the use-case table above. Prefer code checks, then calibrated judges, then human review for the hardest cases.
Build the test set from real inputs. Pull questions from logs, support tickets and sales calls. Add edge cases and every bug you have already fixed.
Set release thresholds. Hard gates for pass or fail checks such as format validity and data leakage. Margin-aware gates for averages, so a drop smaller than the noise does not block a release.
Run it on every change and every model update. Wire the suite into the deployment pipeline, sample live traffic each week, and review a small batch by hand each month to keep the judge honest.
Most teams already know which answers would embarrass them. Few have a test set that catches those answers before a release. Codiste builds that evaluation layer around your own data and failure cases, and tells you plainly where an off-the-shelf tool already covers it.
Common mistakes with LLM evaluation metrics
These are the patterns that make scores look healthy while the product gets worse:
Optimizing the metric instead of the product. A team tunes prompts until the judge score rises, and the answers get longer and blander. Spot-check outputs, not only numbers.
Trusting public benchmarks for product decisions. A higher MMLU score says little about whether a model follows your refund policy.
Using one average for everything. A 92% average can hide a category, such as billing questions, failing half the time. Break scores down by question type.
Never recalibrating the judge. Judge models get updated too. A rubric that matched people in March may not in October.
A test set nobody updates. If production failures never become test cases, the suite measures last year's product.
Evaluating without the real context. Tests that skip retrieval, or use clean documents instead of the messy ones in production, pass easily and predict little.
Best LLM evaluation tools development
These tools run the metrics described here. They save time on scoring and dashboards, but none of them decides which failures matter to you.
Codiste: designs and builds evaluation frameworks around a client's own data, rules and failure cases, including test sets, calibrated judges and release gates wired into the deployment pipeline, then hands the system to the client's team. Our AI agent development services include evaluation as part of every build.
DeepEval: open-source Python framework with ready-made metrics including G-Eval, faithfulness and answer relevancy, designed to run like unit tests.
Ragas: open-source library focused on RAG metrics such as faithfulness, context precision and context recall.
promptfoo: open-source command-line tool for comparing prompts and models against test cases, with assertions and red-team checks.
Langfuse and Arize Phoenix: open-source tracing and observability with evaluation on live traffic.
Braintrust and LangSmith: commercial platforms combining datasets, experiments, judges and production monitoring.
Conclusion
LLM evaluation metrics are only as useful as the failures they are chosen to catch. Teams that start from a list of popular scores end up with a busy dashboard and the same bugs. Teams that start from the five or six ways their product could hurt someone end up with fewer metrics and far better releases.
Three habits separate evaluation that works from evaluation that reassures. First, match the metric to the question: correctness, grounding, usefulness, safety, cost. Second, treat an LLM judge as an instrument that needs calibration, checked against your own people and re-checked whenever the judge or rubric changes. Third, respect the noise. A few points of difference on a hundred cases is usually nothing, and comparing versions on the same cases tells you more than any average.
The test set matters more than the tooling. Build it from real conversations and real failures, and add every production bug to it the week you find it. A modest suite that grows this way will outperform a polished framework fed with invented examples.
Start this week with the smallest version that could work: twenty real questions, one grounding metric, one format check, and a person reviewing the judge's first fifty scores. Run it before the next prompt changes ships, then grow it from what it misses. Within a month, you will trust your release decisions far more than any dashboard could make you.
If your team is about to change models, rewrite prompts or launch an LLM feature without a test set you trust, Codiste can build one from your real data and failure cases before the release goes out.
FAQs
What are the most important LLM evaluation metrics?
It depends on the use case. For RAG assistants, faithfulness, context recall and answer relevancy matter most. For classification, accuracy and per-class precision and recall. For extraction, field-level F1 and format validity. Every production system should also track latency, cost per request and safety checks such as data leakage.
What is the difference between LLM evaluation and LLM benchmarking?
Benchmarking compares base models on public test sets such as MMLU or HumanEval to show general capability. LLM evaluation tests your own application on your tasks, data and rules. A model can lead benchmarks and still fail your use case, so production teams rely on application evaluation for release decisions.
Is LLM-as-a-judge reliable?
It is reliable enough to use when calibrated. Research on MT-Bench found a strong judge agreed with human preferences over 80% of the time, similar to agreement between people. Judges also show position, length and self-preference biases, so teams should check agreement with human labels and control those biases.
How do you measure hallucination in an LLM?
For systems that answer from documents, split each answer into individual claims and check whether the retrieved context supports each one. The share of supported claims is the faithfulness score, and the share of answers with any unsupported claim is the hallucination rate. Without source documents, compare answers against verified references.
What is an LLM evaluation framework?
An LLM evaluation framework is the system a team uses to decide whether a change can ship. It includes a test set built from real inputs, metrics chosen for specific failures, graders such as code checks and calibrated judges, release thresholds, and routines for running it on every change and on live traffic.
How many test cases are enough to evaluate an LLM?
Start with 20 to 50 real cases and grow from production failures. For reliable comparisons, aim for 200 or more on important flows. In 100 cases, a pass rate of 80% carries a margin of about plus or minus 8 points, so small differences between versions are often noise.
Nishant builds where ambitious ideas meet production reality. As CTO & Co-Founder at Codiste, he has helped ship 150+ AI systems. Through The CTO Story, he gets technical leaders talking about what rarely makes the slide deck: the trade-offs, failures, and decisions behind what actually ships.
Every great partnership begins with a conversation. Whether you're exploring possibilities or ready to scale, our team of specialists will help you navigate the journey.
