An LLM app can look impressive in a demo and still fail in production. It may invent facts, ignore instructions, leak data, or give different answers to the same question. LLM evaluation is how you find out before your users do. This guide explains what to measure, which methods to use, how to evaluate RAG systems and agents, and how to build a process you can run before every launch.
TL;DR
30-second summary
What is LLM evaluation and how do you test an LLM app before launch?
LLM evaluation is the process of measuring how accurately, safely, and consistently a large language model or LLM-powered application performs against defined criteria, using test datasets, automated metrics, LLM-as-a-judge scoring, and human review.
- Evaluation works at two levels, and application-level evaluation protects a launch. Model evaluation compares base models on general benchmarks to shortlist one. Application evaluation tests the whole system, including prompts, retrieval, tools, guardrails, and the model, on your users’ real tasks.
- Choose metrics that map to launch decisions, and report each one separately. Common dimensions are correctness, faithfulness, hallucination rate, relevance, safety and bias, robustness, consistency, and latency and cost. A single averaged score can hide a failing dimension, so set a pass threshold for each metric.
- Layer several methods, because none covers everything. Use golden datasets, automated metrics, LLM-as-a-judge scoring, human review, and red teaming. A practical split is cheap automated checks on every output, an LLM judge on a large sample, and human review on high-stakes cases.
- Measure hallucinations by claim, and score RAG and agents in parts. Split answers into claims, test with unanswerable questions, and repeat key prompts. For RAG, score retrieval and generation separately, and for agents, evaluate task success, tool-call correctness, efficiency, permissions, and recovery.
- Gate releases in CI and keep evaluating after launch. Define pass thresholds, build a golden dataset, run tests under controlled conditions, red-team the app, and block releases that miss thresholds. Then sample live conversations, track drift, and add every confirmed failure to the dataset.
Bottom line: A good demo shows what your LLM app can do, and evaluation shows what it does under real conditions. Define what good looks like, score with layered methods, gate releases on thresholds, and start with one use case and a small dataset.
What is LLM evaluation?
LLM evaluation is the process of measuring how accurately, safely, and consistently a large language model or LLM-powered application performs against defined criteria, using test datasets, automated metrics, LLM-as-a-judge scoring, and human review.
It works at two levels:
- Model evaluation compares base models on general benchmarks, so you can shortlist one for your use case.
- Application evaluation tests your whole system, including prompts, retrieval, tools, guardrails, and the model, on your users' real tasks.
Benchmarks help you pick a model, but they cannot tell you whether your support bot answers your customers' questions correctly. Application-level evaluation is what protects a launch.
How it differs from traditional testing
A traditional test checks that a given input produces a fixed output. LLM outputs are non-deterministic and quality depends on context, so you score outputs against criteria instead of exact matches, and you run enough cases to see patterns instead of single results.
People often use "LLM testing" and "LLM evaluation" interchangeably. A useful distinction is that evaluation measures quality on a scale, and testing adds pass thresholds that decide whether a release can go out. For a deeper look at AI-specific failure modes, see our AI testing services.
What to measure in LLM evaluation
Choose metrics that map to decisions your team has to make, such as whether a feature can launch. These are the dimensions most LLM apps need.
| Metric | What it checks | Example check |
|---|---|---|
| Correctness | Whether the answer is factually right and complete | Compare against a human-verified reference answer |
| Faithfulness | Whether claims are supported by the source material provided | Check each claim against the retrieved context |
| Hallucination rate | Share of answers with fabricated or unsupported claims | Count unsupported claims across a test set |
| Relevance and intent | Whether the answer addresses what the user asked | Score the answer against the question and expected intent |
| Safety and bias | Toxicity, harmful content, and unequal treatment of user groups | Run the same prompts across groups and sensitive topics |
| Robustness | Behavior under prompt injection, typos, and adversarial input | Replay red-team prompts and check the guardrails hold |
| Consistency | Whether similar inputs get similar quality | Run the same prompt several times and compare |
| Latency and cost | Time to a complete response and tokens used per request | Measure from prompt to full output |
Not every app needs every metric. A medical information bot weighs faithfulness and safety above all, while a drafting tool may care more about style and consistency. Define what "good" means for your use case first, and write it down as pass thresholds.
Report each metric on its own. A single averaged score can hide a failing dimension, such as a high-quality assistant that leaks sensitive data one time in fifty.
How to measure hallucinations
A hallucination is output that sounds confident but is false, invented, or unsupported by the source material your app was given. Track three types separately, because each has a different fix:
- Factual. The answer states something false about the world.
- Contextual. The answer is not supported by the documents or data you provided, such as a retrieved policy.
- Consistency. The app contradicts itself across answers or within one conversation.
To measure them:
- Score claims, not just answers. Split each answer into individual claims and check each one against a trusted source or the retrieved context. Report both the share of answers with at least one unsupported claim and the share of claims that are unsupported.
- Test with unanswerable questions. Include questions about features, policies, or entities that do not exist. The right behavior is to say the information is not available, so every invented answer counts as a failure.
- Repeat key prompts. Ask the same question several times and compare the answers. Contradictions point to instability even when no single answer can be checked.
- Review a sample by hand. Humans confirm that your automated check finds real hallucinations without flagging correct answers, and they rate severity. A wrong refund policy matters more than a wrong example.
Set stricter thresholds for high-stakes content, such as medical, legal, and financial answers, and track the rates over time so you can see whether prompt, model, or retrieval changes help or hurt.
If your team also uses AI inside the QA process itself, our guides to spotting and preventing AI hallucinations in software testing and what every QA should know about AI hallucinations cover that side.
LLM evaluation methods
No single method covers everything. Strong evaluation programs layer several.
Golden datasets
A golden dataset is a set of realistic inputs paired with human-verified reference answers or scoring criteria. Build it from real user questions where privacy rules allow, and include typical requests, edge cases, and adversarial prompts. Version the dataset, and add every production failure you find, so the same mistake cannot return unnoticed.
Automated metrics
Code-based checks are fast and cheap, so run them on everything. Examples include format validation, such as valid JSON and required fields, keyword and regex checks, and semantic similarity to a reference answer. Classic text-overlap metrics such as BLEU and ROUGE only suit narrow tasks, because they miss meaning.
LLM-as-a-judge
In this method, a second model scores outputs against a written rubric. It scales to thousands of cases that humans could not review, which makes it useful for screening and regression testing. Judges have documented biases, including a preference for the first answer shown, for longer answers, and for outputs from their own model family. To keep a judge reliable:
- Write criteria-based rubrics instead of asking for a vague 1 to 10 score.
- Randomize answer order in pairwise comparisons.
- Use a judge from a different model family than the model under test.
- Compare the judge's scores with human labels on a sample, and recheck whenever you change the judge.
Human review
Domain experts review a sample of outputs, especially ambiguous or high-stakes ones. Their labels also calibrate your LLM judge. Give reviewers the same rubric, and have a second reviewer settle disagreements.
Red teaming
Red teaming means attacking your own app on purpose to see what breaks. Probe for prompt injection, leakage of sensitive data, jailbreaks, harmful content, and agents taking actions they should not. The OWASP Top 10 for LLM Applications is a useful checklist, and our security testing services cover this kind of adversarial work.
A practical split is cheap automated checks on every output, an LLM judge on a large sample, and human review on a small set of high-stakes cases.
Evaluating RAG systems
Retrieval-augmented generation (RAG) answers questions using documents your system retrieves. Two things can go wrong. Retrieval can return the wrong or incomplete context, and generation can ignore or distort the context it received. Score the two separately, or you cannot tell which layer to fix.
| Layer | Metric | Question it answers |
|---|---|---|
| Retrieval | Context precision | Are the retrieved chunks relevant, and are the best ones ranked first? |
| Retrieval | Context recall | Did retrieval find everything needed to answer? |
| Generation | Faithfulness | Are the answer's claims supported by the retrieved context? |
| Generation | Answer relevance | Does the answer address the question that was asked? |
Read the scores together. Good retrieval with low faithfulness points to the generator, while low recall points to chunking, embeddings, or search settings. Open-source frameworks such as Ragas define these metrics, so you do not have to invent them.
Evaluating AI agents
Agents plan, call tools, and take multi-step actions, so checking the final answer is not enough. Evaluate the whole workflow:
- Task success. Did the agent complete the goal?
- Tool-call correctness. Did it pick the right tool, with the right arguments, in the right order?
- Efficiency. Did it avoid loops and unnecessary steps that add cost and delay?
- Permissions and safety. Does it stay within allowed actions and ask for confirmation before risky ones?
- Recovery. How does it handle tool errors and ambiguous requests?
Test full conversations and workflows instead of single prompts, and log every step so failures can be traced. Our AI agent benchmarking and AI chatbot testing services cover both.
Test your AI agent on full workflows, not single prompts
TestDevLab evaluates agents on task success, tool-call correctness, and safety across multi-step workflows. You see where they fail before they take a wrong action in production.
How to test an LLM app before launch
These six steps turn the methods above into a process your team can repeat.
1. Define what good looks like
Write down the quality, safety, and performance criteria for your use case, with a pass threshold for each metric. These thresholds become your go or no-go decision, so agree on them with product and engineering before testing starts.
2. Build the golden dataset
Collect realistic scenarios, edge cases, and adversarial prompts, and pair them with human-verified reference answers or scoring criteria. Cover the conditions where your app is most likely to fail, not only the easy ones.
3. Run tests under controlled conditions
Send the same inputs to your app every time, and record the model version, prompt version, retrieval settings, and temperature with each run. Without that record, you cannot tell whether a score changed because of your fix or because something else moved.
4. Score with layered methods
Clean the outputs first, so formatting noise does not skew scores. Then apply automated checks, an LLM judge on a large sample, and human review on high-stakes cases. Check how closely the judge agrees with your human labels before you trust it.
5. Red-team the app
Attack it with prompt injection, requests for sensitive data, jailbreak attempts, and out-of-scope tasks. Fix what you find, and add each successful attack to the dataset.
6. Set release gates and run them in CI
Block a release when a metric falls below its threshold. Rerun the evaluation suite whenever a prompt, model, retrieval setting, or tool changes, with a fast subset on every pull request and the full suite before release. This is shift-left testing applied to AI, and our CI/CD services help teams wire it into their pipelines.
Monitoring LLM quality in production
Pre-launch evaluation cannot cover everything. Real users ask questions you did not anticipate, your data changes, and model providers update their models. Quality can erode quietly, because LLM failures rarely throw errors.
Keep evaluating after release:
- Sample live conversations and score them with the same rubric and judge you used before launch.
- Track user signals, such as thumbs-down ratings, rephrased questions, and escalations to human agents.
- Watch for drift in quality scores, latency, and cost, and alert on safety incidents.
- Add every confirmed failure to your golden dataset.
Redact personal data before you store or review production logs.
Where QA engineers fit
QA brings test design, traceability, and release discipline to a field where many teams still rely on spot checks. QA engineers define acceptance criteria and thresholds, build and maintain datasets, design adversarial tests, run the evaluation pipeline, and report results in terms product teams can act on.
Independence also matters. The team that built an AI feature tends to test the cases it already expects, so an outside evaluation catches blind spots from proximity. If you want a partner, our AI testing services cover LLM, RAG, and agent evaluation, and our guide to the best AI application testing companies compares providers.
Final thoughts
A good demo shows what your LLM app can do, and evaluation shows what it does under real conditions. Define what good looks like, build a golden dataset, score with layered methods, gate releases on thresholds, and keep watching after launch. Start with one use case and a small dataset, then expand as you learn where your app fails.
FAQ
Most common questions
What is LLM evaluation?
LLM evaluation is the process of measuring how accurately, safely, and consistently a large language model or LLM-powered application performs against defined criteria. It uses test datasets, automated metrics, LLM-as-a-judge scoring, and human review. It works at two levels: model evaluation compares base models on general benchmarks, while application evaluation tests your whole system, including prompts, retrieval, tools, and guardrails, on real user tasks. Benchmarks help you pick a model, but application-level evaluation is what protects a launch.
How is LLM evaluation different from traditional software testing?
A traditional test checks that a given input produces a fixed output. LLM outputs are non-deterministic and quality depends on context, so you score outputs against criteria instead of exact matches and run enough cases to see patterns instead of single results. A useful distinction is that evaluation measures quality on a scale, while testing adds pass thresholds that decide whether a release can go out. Teams often use the two terms interchangeably.
What metrics should you use to evaluate an LLM app?
Most LLM apps need correctness, faithfulness, hallucination rate, relevance and intent, safety and bias, robustness, consistency, and latency and cost. Not every app needs every metric. A medical information bot weighs faithfulness and safety above all, while a drafting tool may care more about style and consistency. Report each metric on its own, because a single averaged score can hide a failing dimension, and write down a pass threshold for each one.
How do you measure hallucinations in an LLM app?
Track factual, contextual, and consistency hallucinations separately, since each has a different fix. Split each answer into individual claims and check each against a trusted source or the retrieved context, reporting both the share of answers with an unsupported claim and the share of unsupported claims. Add unanswerable questions about things that don’t exist, repeat key prompts to catch contradictions, and have humans review a sample. Set stricter thresholds for medical, legal, and financial content.
How do you evaluate a RAG system?
Score retrieval and generation separately, or you can’t tell which layer to fix. For retrieval, measure context precision, which checks whether retrieved chunks are relevant and ranked well, and context recall, which checks whether everything needed was found. For generation, measure faithfulness, meaning the answer’s claims are supported by the retrieved context, and answer relevance. Good retrieval with low faithfulness points to the generator, while low recall points to chunking, embeddings, or search settings. Frameworks like Ragas define these metrics.
Turn evaluation into a release gate your team can rely on
TestDevLab's ISTQB-certified engineers design the evaluation framework, build the ground truth, and report the metrics your team needs for go or no-go decisions. You launch with evidence instead of impressions from a demo.
