Evaluating LLM Applications: Building Your Own Eval Suite

“It seemed to work when I tried it” is not an evaluation strategy, but it’s how a lot of LLM features ship. The problem surfaces later, usually after a prompt tweak, a model upgrade, or a new edge case quietly regresses something that used to work fine. Traditional software has unit tests and CI to catch this. LLM applications need the same discipline, adapted for outputs that are graded on quality rather than exact equality.

This is about building that discipline for your own application, not about running published academic benchmarks — the difference is that your eval suite should be testing your prompts, your retrieval pipeline, and your failure modes, not a generic leaderboard task.

Start With a Golden Dataset, Not a Framework

Before picking any evaluation library, you need a golden dataset: a curated set of realistic inputs paired with either a known-correct output, a rubric for what a correct output looks like, or both. This is the single highest-leverage thing you can build, and it is entirely application-specific.

A good golden dataset is assembled from three sources, roughly in order of value:

1. Real production inputs, especially ones that previously caused a bug, a user complaint, or a support ticket. These are the cases you can least afford to regress.
2. Deliberately adversarial or edge-case inputs you write yourself — ambiguous phrasing, missing context, conflicting instructions, inputs in a different language than expected.
3. Synthetic variations generated by prompting an LLM to paraphrase or perturb your real examples, useful for volume once you’ve exhausted the first two sources.

# golden_dataset.py
GOLDEN_CASES = [
    {
        "id": "refund-policy-conflicting-dates",
        "input": "I bought this on Jan 3rd but the receipt says Jan 30th, can I get a refund?",
        "expected_behavior": "Ask the user to clarify the discrepancy before making a refund decision; must not silently pick one date.",
        "category": "ambiguity_handling",
    },
    {
        "id": "empty-retrieval-result",
        "input": "What's our policy on left-handed scissors?",
        "expected_behavior": "State that no relevant policy was found; must not fabricate a policy.",
        "category": "hallucination_guard",
    },
]

Note that expected_behavior here is a rubric, not a fixed string. Most real LLM application outputs are not single correct strings — they are one of many acceptable phrasings that satisfy some criteria. This is exactly the gap LLM-as-judge grading is meant to fill.

LLM-as-Judge: What It Buys You, and Its Biases

For free-text outputs, having a human read every response on every run doesn’t scale, and exact string matching is far too strict for anything beyond structured extraction. LLM-as-judge uses a separate model call to grade the system-under-test’s output against your rubric.

JUDGE_PROMPT = """You are grading an AI customer support response.

User input: {input}
Expected behavior: {expected_behavior}
Actual response: {actual_response}

Does the actual response satisfy the expected behavior? Answer with exactly one word: PASS or FAIL.
Then, on a new line, give a one-sentence reason.
"""

def judge(client, case, actual_response):
    prompt = JUDGE_PROMPT.format(
        input=case["input"],
        expected_behavior=case["expected_behavior"],
        actual_response=actual_response,
    )
    result = client.messages.create(
        model="claude-judge-model",
        max_tokens=100,
        temperature=0,
        messages=[{"role": "user", "content": prompt}],
    )
    verdict_line = result.content[0].text.strip().split("\n")[0]
    return verdict_line.upper().startswith("PASS")

This works, but LLM judges have well-documented biases you should design around rather than ignore:

• Position bias. When a judge compares two candidate outputs side by side, it tends to favor whichever one appears first, independent of quality. If you’re doing pairwise comparison (A/B-ing two prompt versions), run the comparison both ways (A-then-B and B-then-A) and require agreement, or randomize order across your dataset.
• Verbosity bias. Judges systematically rate longer answers as better, even when the extra length adds no information. If you’re comparing outputs of very different lengths, say so explicitly in the judge prompt (“length alone is not a quality signal”) and spot-check with human review.
• Self-preference bias. A judge model tends to rate outputs written in its own family’s style more favorably. If your system-under-test and your judge are the same model (or same family), consider using a different model as judge, or at minimum treat borderline scores with more suspicion.
• Rubric leakage / sycophancy. A judge given a rubric that sounds like it’s describing what the response actually says will pattern-match on surface similarity rather than truly verifying the criteria. Ask the judge for a short justification (as above) so you can spot-check reasoning, not just the verdict.

None of this means LLM-as-judge is unusable — it means you should validate it against human judgment on a sample before trusting it, and re-validate whenever you change the judge model or prompt.

def validate_judge_against_humans(judge_fn, human_labeled_cases):
    agreements = 0
    for case in human_labeled_cases:
        judge_verdict = judge_fn(case["actual_response"], case)
        if judge_verdict == case["human_label"]:
            agreements += 1
    agreement_rate = agreements / len(human_labeled_cases)
    return agreement_rate  # track this over time; investigate drops

Offline vs Online Evaluation

These two modes answer different questions and neither substitutes for the other.

	Offline evaluation	Online evaluation
Runs against	Golden dataset, fixed inputs	Live production traffic
When	Before every deploy, in CI	Continuously, after deploy
Measures	Regression vs known cases and rubrics	Real-world distribution shift, actual user satisfaction
Feedback signal	LLM-as-judge, exact match, rule-based checks	Thumbs up/down, retry rate, escalation rate, task completion
Catches	”This prompt change broke case X we already knew about"	"Users are asking things our golden dataset never anticipated”
Speed	Fast, deterministic-ish, runs in minutes	Slow, statistical, needs volume to be meaningful

Offline evals are what let you ship confidently; online evals are what tell you your offline suite is missing something. A healthy loop feeds interesting online failures (a bad thumbs-down, an escalated conversation) back into the golden dataset as new offline test cases — this is how the suite grows in the direction that actually matters, rather than in the direction you happened to think of upfront.

Wiring Regression Evals Into CI

The point of an eval suite is worthless if it only runs manually before a demo. Wire it into CI the same way you would a test suite, with an explicit pass threshold rather than requiring 100% (LLM-as-judge grading has noise, and a single flaky case shouldn’t block every deploy):

# .github/workflows/eval.yml
name: LLM Eval Suite

on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/llm/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - name: Run eval suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: python run_evals.py --threshold 0.90 --report eval_report.json
      - name: Upload report
        uses: actions/upload-artifact@v4
        with:
          name: eval-report
          path: eval_report.json

# run_evals.py
import json
import sys

def run_suite(golden_cases, system_under_test, judge_fn):
    results = []
    for case in golden_cases:
        actual = system_under_test(case["input"])
        passed = judge_fn(case, actual)
        results.append({"id": case["id"], "passed": passed, "category": case["category"]})
    return results

def main(threshold, report_path):
    results = run_suite(GOLDEN_CASES, my_system_under_test, judge)
    pass_rate = sum(r["passed"] for r in results) / len(results)
    with open(report_path, "w") as f:
        json.dump({"pass_rate": pass_rate, "results": results}, f, indent=2)
    print(f"Pass rate: {pass_rate:.2%} (threshold: {threshold:.2%})")
    if pass_rate < threshold:
        sys.exit(1)

if __name__ == "__main__":
    main(threshold=0.90, report_path="eval_report.json")

Breaking results down by category (as in the JSON above) matters more than the aggregate pass rate — an aggregate can stay flat while one category (say, hallucination_guard) quietly regresses because it’s a small fraction of the total dataset. Track category-level pass rates over time, not just the overall number, and alert on category regressions specifically.

Practical Guidance for a Small Team

You don’t need a large eval framework to get most of the value. A pragmatic starting point:

• Start with 20-50 golden cases weighted toward real production failures, not synthetic ones. Grow this over time; don’t wait to have hundreds before you start.
• Use exact-match or rule-based checks wherever you can (structured output, required fields present, forbidden phrases absent) — they’re free of judge bias and worth using before reaching for an LLM judge.
• Reserve LLM-as-judge for genuinely free-text quality questions, and validate it against a small human-labeled sample whenever you change the judge model or prompt.
• Set a pass threshold below 100% and review failures rather than treating every red run as a hard blocker — eval noise is real, but a consistently dropping pass rate is a signal, not noise.
• Feed real online failures back into the golden dataset continuously; this is what keeps the offline suite relevant instead of stale.

Key Takeaways

• A golden dataset built from real production failures, deliberate edge cases, and synthetic variations is the highest-leverage asset in an eval suite, and it is entirely application-specific.
• LLM-as-judge grading scales free-text evaluation past what manual review can cover, but carries position bias, verbosity bias, and self-preference bias that you should design around, not ignore.
• Validate your judge against human-labeled samples whenever you change the judge model or prompt, and track judge-human agreement over time.
• Offline evaluation (golden dataset, run in CI) catches known regressions fast; online evaluation (production signals) catches real-world cases your golden dataset never anticipated — you need both, and they should feed each other.
• Wire the eval suite into CI with an explicit pass threshold below 100%, since judge-based grading has inherent noise.
• Track pass rates per category, not just in aggregate — a stable overall number can hide a real regression in a small but critical category.
• Prefer deterministic, rule-based checks over LLM-as-judge wherever the output has verifiable structure; reserve the judge for genuinely subjective quality questions.