LLM evaluation is the practice of systematically measuring the quality, accuracy, and safety of AI-generated output before it ships. In 2026, the standard approach is Evaluation-Driven Development: build golden datasets from real failures, score output with automated metrics (faithfulness, relevancy, hallucination), and gate your CI/CD pipeline so regressions never reach production. Tools like Promptfoo, DeepEval, and RAGAS make this practical for solo builders.
- Evaluation-Driven Development (EDD) is the 2026 industry standard for reliable LLM applications (Source: Red Hat, DeepEval, Braintrust documentation).
- A golden dataset of approximately 100 curated examples is the foundation of every eval pipeline.
- Three evaluation tiers: deterministic (free, fast), LLM-as-judge (nuanced, costly), human review (gold standard).
- Self-preference bias skews LLM-as-judge scores by up to 10 points. Never judge with the same model family (NeurIPS 2026).
- Promptfoo: open-source CLI, YAML config, acquired by OpenAI March 2026 but still MIT-licensed and vendor-neutral.
- DeepEval: Pytest for LLMs, 50+ metrics including faithfulness and hallucination detection.
- RAGAS: reference-free metrics for RAG pipelines (faithfulness, context precision, answer relevancy).
- CI/CD gating: run evals on every PR via GitHub Actions; block merges that regress quality scores.
Why Evals Matter: The Trust Gap
Every vibe coding workflow has the same blind spot: the output looks plausible. The model writes syntactically valid code, produces grammatically correct prose, and structures JSON that parses cleanly. But plausibility is not correctness. A function that handles 9 of 10 edge cases looks identical to one that handles all 10, until the 10th case hits production.
The Princeton/Georgia Tech GEO study (Aggarwal et al., KDD 2024, arXiv 2311.09735) found that pages with concrete, quantitative data are cited approximately 41% more often by AI systems. The same principle applies to your own AI output: quantitative evaluation is how you move from "it seems fine" to "it scores 0.94 on faithfulness across 100 test cases."
Evaluation-Driven Development (EDD) formalizes this. It is a cyclical process: define what good output looks like, measure the gap between current performance and your threshold, then iterate until the gap closes. Red Hat, DeepEval, and Braintrust all document EDD as the 2026 industry standard for building reliable LLM applications.
When a traditional function fails, it throws an error. When an LLM fails, it returns confident, well-formatted wrong answers. Without evals, your only detection mechanism is user complaints. By then the damage is shipped.
The Eval Taxonomy: Three Tiers
Deterministic
Exact match, substring, regex, JSON schema validation, keyword presence. Fast, free, reproducible. Catches structural failures.
Model-Graded
LLM-as-a-judge scoring on custom rubrics. Handles ambiguity, tone, and reasoning quality. Requires calibration against human judgment.
Human Review
Domain experts grade output manually. Gold standard for legal, medical, and financial domains. Use for the 1-5% of edge cases flagged by automated tiers.
Metric Categories
| Category | What It Measures | Example Metrics | Tools |
|---|---|---|---|
| Faithfulness | Is the output supported by the provided context? | Faithfulness score, claim decomposition | RAGAS, DeepEval |
| Relevancy | Does the output address the query? | Answer relevancy, semantic similarity | RAGAS, DeepEval |
| Hallucination | Does the output contain fabricated claims? | Hallucination score, factual consistency | DeepEval |
| Safety | PII leakage, prompt injection, toxicity | Red-team pass rate, jailbreak resistance | Promptfoo |
| Custom Rubric | Domain-specific quality (brand voice, format) | G-Eval with chain-of-thought | DeepEval, Promptfoo |
Building Your First Eval Set
The golden dataset is the foundation of every eval pipeline. It defines what "correct" looks like for your specific application. Generic benchmarks (MMLU, HumanEval, SWE-bench) measure general model capability. Your golden dataset measures whether the model solves your problem.
Where to Find Your Test Cases
- Production logs: search for user complaints, refund requests, support tickets that mention incorrect AI output.
- Manual spot-checks: review 50 recent outputs. Mark any that are wrong, incomplete, or misleading.
- Edge cases: intentionally adversarial inputs. Empty strings, maximum-length inputs, multilingual text, special characters.
- Known failure modes: the specific scenarios where your model has previously hallucinated, missed context, or produced unsafe output.
Test Case Structure
{
"test_cases": [
{
"input": "What certifications does the STTU169 organic tee have?",
"expected_output": "GOTS, PETA-Approved Vegan",
"context": "Product database entry for STTU169...",
"tags": ["certification", "product-accuracy"],
"severity": "critical"
}
]
}
Collect 10 real failures. Structure each as input + expected output + context. Run your first eval. That first run teaches you more about your model's weaknesses than any documentation. Scale to 100 cases as you iterate.
Tool Walkthrough: The Eval Ecosystem
| Tool | Type | Best For | License | Key Feature |
|---|---|---|---|---|
| Promptfoo | CLI Framework | YAML config, red-teaming | MIT (open source) | Side-by-side model comparison matrix |
| DeepEval | Python Framework | Pytest-style CI/CD | Apache 2.0 | 50+ built-in metrics, G-Eval |
| RAGAS | RAG Evaluator | RAG pipeline metrics | Apache 2.0 | Reference-free faithfulness scoring |
| Braintrust | Platform | Team dashboards, experiment tracking | Commercial | Shared experiment logs, PM access |
| Arize Phoenix | Observability | Production monitoring | Apache 2.0 | OpenTelemetry-native tracing |
| LangSmith | Observability | LangChain/LangGraph users | Commercial | End-to-end agent development |
| Inspect AI | Safety Evaluator | Frontier model auditing | MIT | UK AISI research-grade benchmarks |
Promptfoo: Quick Start
Promptfoo is a CLI-first evaluation tool. You define prompts, test cases, and assertions in YAML. It runs evaluations locally by default, meaning your prompts and data never leave your machine (Source: promptfoo.dev, github.com/promptfoo). OpenAI acquired Promptfoo in March 2026, but the project remains MIT-licensed and supports all major model providers (Source: DataCamp, AITestingGuide).
# promptfooconfig.yaml
prompts:
- "Answer the user's question using only the provided context.\n\nContext: {{context}}\n\nQuestion: {{question}}"
providers:
- id: openai:gpt-4o
- id: anthropic:messages:claude-sonnet-4-20250514
tests:
- vars:
question: "What certifications does DDS hold?"
context: "DDS holds GOTS, GRS, OCS, PETA-Approved Vegan, and Fair Trade."
assert:
- type: contains
value: "GOTS"
- type: not-contains
value: "OEKO-TEX"
- type: llm-rubric
value: "The answer is factually accurate and only mentions certifications found in the context."
DeepEval: Pytest for LLMs
DeepEval integrates with Python's pytest framework. You write LLM test cases using familiar syntax and run them with deepeval test run. It provides over 50 built-in metrics including Faithfulness, Hallucination, Answer Relevancy, and G-Eval (Source: deepeval.com).
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric
def test_certification_accuracy():
test_case = LLMTestCase(
input="What certifications does DDS hold?",
actual_output="DDS holds GOTS, GRS, OCS, PETA-Approved Vegan, and Fair Trade.",
retrieval_context=[
"Design Delight Studio certifications: GOTS, GRS, OCS, PETA-Approved Vegan, Fair Trade."
]
)
metric = FaithfulnessMetric(threshold=0.8)
assert_test(test_case, [metric])
All tool descriptions above are verified against primary documentation (promptfoo.dev, deepeval.com, ragas.io, braintrust.dev) as of July 24, 2026. Feature sets, pricing, and licensing can change. Always verify against the tool's official docs before adopting.
LLM-as-Judge: Power and Pitfalls
Model-graded evaluation is the second tier of the eval taxonomy. You provide a rubric, the model's output, and optionally a reference answer. A judge model (typically a frontier model like GPT-4o or Claude Opus 4) scores the output. This approach handles nuanced criteria that deterministic checks cannot: reasoning quality, tone, helpfulness, and domain accuracy.
Known Biases (NeurIPS 2026 Research)
Self-Preference
Judge models inflate scores for outputs from their own model family. Linked to perplexity: models favor stylistically familiar text. Can skew scores by up to 10 points.
Position
In pairwise comparisons, judges favor the first or last response shown. Mitigate by alternating response order and averaging scores.
Verbosity
Judges reward longer, more detailed responses regardless of accuracy. A 200-word wrong answer may outscore a 50-word correct one.
Calibration Protocol
- Have domain experts manually grade 50 to 100 examples on your rubric.
- Run the same examples through your LLM judge.
- Calculate Cohen's kappa (inter-annotator agreement) between human grades and judge grades.
- If kappa falls below 0.5, your judge's rubric or prompt needs recalibration.
- Track kappa continuously. Judge accuracy drifts as models receive updates.
Never use the same model family for both generation and evaluation. If your application runs on Claude Sonnet, do not use Claude as the judge. Use a cross-family model (GPT-4o, Gemini) as the anchor. This is the single most important mitigation for self-preference bias.
Metrics That Matter
RAGAS Metrics (RAG Pipelines)
RAGAS provides reference-free metrics, meaning you can evaluate production traffic where ground-truth labels do not exist (Source: ragas.io).
| Metric | Measures | How It Works |
|---|---|---|
| Faithfulness | Is the answer grounded in context? | Decomposes answer into claims, verifies each against retrieved context |
| Context Precision | Are relevant chunks ranked higher? | Evaluates retrieval ranking quality |
| Context Recall | Was all necessary info retrieved? | Compares retrieved context against ground truth |
| Answer Relevancy | Does the output address the query? | Generates hypothetical questions from answer, compares to original query |
| Answer Correctness | Is the answer factually correct? | Compares against ground truth reference answer |
DeepEval Metrics (General LLM)
- Faithfulness: evaluates whether output is factually aligned with retrieval context. Primary metric for RAG hallucination detection.
- Hallucination: detects fabricated claims not present in the provided context. Checks for internal consistency.
- G-Eval: uses LLM-as-a-judge with Chain-of-Thought reasoning to evaluate output on any custom criteria (brand voice, domain helpfulness, format compliance).
- Answer Relevancy: measures how directly the output addresses the user's original question.
Pick 3 to 5 metrics. More than 5 creates analysis paralysis. Fewer than 3 leaves blind spots. The right set depends on your application: if you serve retrieved documents, faithfulness and context precision are mandatory. If you generate creative content, custom rubrics via G-Eval are more useful.
CI/CD Integration: Evals as Pipeline Gates
GitHub Actions Example
# .github/workflows/eval-gate.yml
name: LLM Eval Gate
on: [pull_request]
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Promptfoo
run: npm install -g promptfoo
- name: Run deterministic evals
run: promptfoo eval --config promptfooconfig.yaml --output results.json
- name: Check pass rate
run: |
PASS_RATE=$(jq '.results.stats.successes / .results.stats.total' results.json)
echo "Pass rate: $PASS_RATE"
if (( $(echo "$PASS_RATE < 0.90" | bc -l) )); then
echo "FAIL: Pass rate below 90% threshold"
exit 1
fi
Cost Management Strategy
- Every commit: run deterministic checks only (regex, JSON schema, keyword). Cost: zero.
- Pull request: run the full eval suite including LLM-as-judge. Cost: proportional to golden dataset size.
- Nightly: run extended eval suites with larger datasets and statistical significance testing.
LLM outputs are non-deterministic. A single eval run might show a 2% score drop that is just noise. Use confidence intervals or Welch's t-test across multiple runs to distinguish real regressions from variance (Source: FutureAGI, Microsoft research). Gate on statistical significance, not individual scores.
Observability & Production Monitoring
Observability Tools
Arize Phoenix
Self-hostable, ML-grade rigor, vendor-agnostic tracing. Handles mixed ML and LLM workloads. Favored by teams that want control over their data (Source: arize.com).
LangSmith
End-to-end agent development for LangChain/LangGraph users. Path of least resistance if you are already in the LangChain ecosystem (Source: smith.langchain.com).
What to Trace
- Prompt construction: the full system instruction, user input, and any injected context.
- Retrieval step: which documents were retrieved, their relevancy scores, and ranking order.
- Tool calls: which tools were invoked, their arguments, and return values.
- Model response: the full output, token count, latency, and cost.
- User feedback: thumbs up/down, corrections, complaints. This becomes future golden dataset material.
Implement traces first. Before you can evaluate, you need to see what happened. A trace that captures input, retrieval context, tool calls, and output gives you everything you need to build test cases and diagnose failures. Without traces, debugging an LLM failure is guesswork (Source: MLflow documentation, Galtea.ai).
Pitfalls & Anti-Patterns
Vibes-Based Testing
"It looks right" is not a metric. Manual eyeballing catches surface errors but misses subtle hallucinations, context drift, and statistical regressions across hundreds of test cases.
Same-Family Judging
Using GPT-4o to evaluate GPT-4o output inflates scores by up to 10 points due to self-preference bias. Always use a cross-family judge model.
Benchmark Worship
SWE-bench, MMLU, and HumanEval measure general capability, not your specific use case. A model that scores 95% on HumanEval may still fail on your domain-specific prompts.
Ship and Forget
Offline evals pass. You deploy. The model vendor pushes an update. Your output quality silently degrades. Without production monitoring, you only discover this from user complaints.
Do not build a comprehensive eval pipeline before you have a working product. Start with 10 test cases and one metric. Ship. Collect real failures. Expand the eval set from production data. The best eval sets are grown from real usage, not imagined in advance.
Frequently Asked Questions
LLM evaluation is the practice of systematically measuring the quality, accuracy, and safety of AI-generated output. For vibe coders, it matters because you are shipping code you did not write line-by-line. Without evaluation, you are deploying software whose correctness you cannot verify. Evaluation turns subjective trust into measurable confidence.
Evaluation-Driven Development (EDD) is a methodology where evaluation is integrated into every stage of AI application development. You define golden datasets, establish 3 to 5 quantitative metrics, and gate your CI/CD pipeline so no prompt or model change ships if it regresses those scores. It is the 2026 industry standard for building reliable LLM applications, according to Red Hat, DeepEval, and Braintrust documentation.
A golden dataset is a curated set of input-output pairs that represent critical success and failure modes of your application. Start with approximately 100 high-quality examples. Build them from real production failures, edge cases, and the specific scenarios where your AI has previously produced incorrect output. Quality matters more than quantity.
Deterministic evaluation uses exact rules: string matching, regex patterns, JSON schema validation, keyword presence checks. It is fast, cheap, and reproducible. Model-graded evaluation uses an LLM as a judge to score responses on nuanced criteria like faithfulness, relevance, and tone. It handles ambiguity that deterministic checks cannot, but introduces its own biases and costs.
LLM-as-a-judge uses a language model (typically a frontier model like GPT-4o or Claude Opus) to evaluate the output of another model. You provide a rubric, the model's output, and optionally a reference answer. The judge scores the output according to your criteria. It scales evaluation beyond what human reviewers can handle, but requires calibration against human judgment to be trustworthy.
Self-preference bias (SPB) occurs when a judge model inflates scores for outputs generated by itself or its own model family. NeurIPS 2026 research shows SPB can skew evaluation scores by up to 10 points. It is linked to perplexity: models favor outputs that are stylistically familiar. The primary mitigation is cross-family validation, meaning you never use the same model family for both generation and evaluation.
Have domain experts manually grade 50 to 100 examples. Run the same examples through your LLM judge. Compare results using Cohen's kappa (inter-annotator agreement). If kappa falls below 0.5, your judge's rubric or prompt needs recalibration. Track this metric continuously because judge accuracy drifts as models receive updates.
Promptfoo is an open-source CLI tool for LLM evaluation, red-teaming, and prompt testing. It was acquired by OpenAI in March 2026 but remains MIT-licensed and vendor-neutral. You define test cases in YAML, run them from the terminal, and get a matrix view comparing multiple models side by side. It supports deterministic assertions, LLM-as-judge grading, and automated red-teaming for prompt injection and jailbreak detection.
DeepEval is an open-source Python framework that functions as Pytest for LLMs. You write evaluation test cases using familiar Python syntax and run them with pytest. It provides over 50 built-in metrics including faithfulness, hallucination detection, and G-Eval. DeepEval is code-first and integrates natively with CI/CD pipelines. Promptfoo is config-first (YAML) and CLI-driven. Choose DeepEval if you prefer Python test suites; choose Promptfoo if you prefer declarative YAML configuration.
RAGAS provides reference-free metrics for RAG pipelines. Faithfulness measures whether the generated answer is factually supported by the retrieved context. Context Precision checks if relevant chunks are ranked higher than irrelevant ones. Context Recall measures if all necessary information was retrieved. Answer Relevancy evaluates whether the output addresses the query. Answer Correctness compares against a ground truth reference answer.
Run evals automatically on every pull request using GitHub Actions. Structure it in three tiers: deterministic checks (JSON schema, regex, PII detection) run on every commit because they are fast and free. LLM-as-judge evaluations run on merge requests. Statistical gating using confidence intervals ensures performance shifts are significant, not noise. If the aggregate score falls below your threshold, the action exits with a non-zero code and blocks the merge.
Track faithfulness (is the output supported by the provided context), factual consistency (does the output contradict known facts), and claim decomposition (break the output into individual claims and verify each). DeepEval provides dedicated Faithfulness and Hallucination metrics. RAGAS provides Faithfulness as a reference-free metric. For production, combine automated metrics with human spot-checks on the outputs flagged as lowest-confidence.
Start with critical paths: the prompts where a failure causes user-visible harm, data corruption, or security exposure. Build golden datasets for those first. Expand coverage as your eval infrastructure matures. A practical starting point is 3 to 5 prompts with 20 test cases each, then grow to 100+ cases per prompt as you collect real failure examples from production.
Nothing. This masterclass is 100% free. No signup, no email, no paywall, no certificate requirement. It is part of the DDS Vibe Academy, a free AI coding curriculum built by Robert McCullock at Design Delight Studio.
Start with Module 03 of this masterclass: Building Your First Eval Set. Collect 10 real failures from your AI application. Structure them as input-expected output pairs. Install Promptfoo (npx promptfoo init) or DeepEval (pip install deepeval). Run your first evaluation locally. Then read Module 07 to wire it into your CI/CD pipeline. The entire process takes under an hour to set up.