LLM Evaluation in Python: Test Your AI App with pytest (2026)

Here is the uncomfortable thing about shipping an LLM feature: your test suite stays green while the product gets worse. You change a prompt on Tuesday, the unit tests still pass because they only assert status_code == 200, and on Friday support tickets show the model started refusing a whole category of requests. Nothing broke — there was just nothing measuring whether the answer was good.

This guide builds the missing layer: an evaluation suite that runs in pytest, catches regressions in output quality, and costs a few cents per run instead of a few dollars. We will go from a plain JSONL dataset to deterministic checks, similarity scoring, LLM-as-judge, and a CI setup you can actually leave switched on.

1. Why Normal Tests Fail on LLM Features

Three properties of language models break the assumptions unit tests are built on:

• Non-determinism. Even at temperature 0, output is not byte-stable across model versions, and providers silently update models behind stable aliases. assert "yes" in output passes today and fails after a provider rollout.
• Many valid answers. "Summarize this ticket" has thousands of acceptable outputs. Exact-match assertions only work for a narrow slice of tasks — classification, extraction, routing.
• Failures are graded, not binary. A response can be 80% right. Traditional tests have no vocabulary for that, so teams end up with a test that either never fails or fails constantly and gets deleted.

The fix is to stop writing assertions against exact strings and start writing assertions against properties: does it parse, does it contain the required fields, is it semantically close to a reference answer, does it stay grounded in the retrieved context. If you are new to pytest itself, our pytest tutorial for beginners covers fixtures and parametrize.

2. Pick the Right Eval for the Job

Most wasted effort in LLM testing comes from using a heavyweight eval where a cheap one would do. Work down this table and stop at the first row that genuinely measures your feature:

The ordering matters. A schema check catches most real regressions for free; an LLM judge is the last resort, not the default. Teams that start with a judge end up with a slow, flaky suite they stop running.

3. Build a Golden Dataset First

Evals are only as good as the cases they run against. Store them as JSONL — one line per case, easy to diff in review, easy to append to when a user reports a bad output:

{"id": "billing-refund", "input": "I was charged twice for the same order, order #8812.",
 "expected_label": "billing", "reference": "Apologize, confirm the duplicate charge on order 8812, and state a refund is being issued within 3-5 business days.",
 "must_mention": ["refund", "8812"]}
{"id": "cancel-angry", "input": "This is ridiculous. Cancel my account right now.",
 "expected_label": "cancellation", "reference": "Acknowledge the frustration, confirm the cancellation, and ask if they want a confirmation email.",
 "must_mention": ["cancel"]}

Three rules that make the dataset pay off. First, every case you add comes from a real failure — a support ticket, a bug report, a weird input you noticed. Synthetic cases you invented while feeling productive rarely match production traffic. Second, include the cases you are tempted to skip: empty input, input in the wrong language, a 4,000-token input, prompt injection attempts. Third, keep it small and honest. Thirty well-chosen cases beat three hundred generated ones; you will run this on every pull request.

Loading it in pytest is one fixture:

import json, pathlib

import pytest

CASES = [
    json.loads(line)
    for line in pathlib.Path("evals/cases.jsonl").read_text().splitlines()
    if line.strip()
]

@pytest.fixture(params=CASES, ids=[c["id"] for c in CASES])
def case(request):
    return request.param

Using ids= means a failure prints as test_label[billing-refund] instead of test_label[case3], which is the difference between a test report you act on and one you ignore.

4. Wire It Into pytest

Start with the cheapest check that has any signal. For a classifier, that is an exact label match:

def test_route_label(case):
    result = classify(case["input"])          # your LLM call
    assert result.label == case["expected_label"], (
        f"Expected {case['expected_label']!r}, got {result.label!r}"
    )

For open-ended generation, assert on properties instead of text. The must_mention field from the dataset gives you a check that is strict where it matters and tolerant where it does not:

def test_response_covers_key_facts(case):
    answer = generate_reply(case["input"])
    lowered = answer.lower()
    missing = [t for t in case["must_mention"] if t.lower() not in lowered]
    assert not missing, f"Answer omitted {missing} for {case['id']}"
    assert len(answer) < 1200, "Answer ran long — check max_tokens"

That second assertion looks trivial and catches a real class of regression: someone bumps max_tokens or switches model, and replies quietly triple in length and cost.

5. Structure and Schema Evals

If your feature returns structured data, schema validation is a free, high-signal eval. Reuse the same Pydantic model your application uses — if you do not have one yet, the pattern is in our structured LLM output with Pydantic guide:

from pydantic import ValidationError

def test_extraction_matches_schema(case):
    raw = extract(case["input"])
    try:
        obj = Extraction.model_validate_json(raw)
    except ValidationError as e:
        pytest.fail(f"{case['id']}: invalid output — {e.errors()[:2]}")
    assert obj.items, f"{case['id']}: returned zero items"

Notice the second assertion. A schema can be perfectly valid and completely empty, and "the model returned an empty list" is one of the most common silent production failures. Always assert on content, not just shape.

6. Semantic Similarity Scoring

When there are many valid answers, compare meaning instead of text. Embed both the generated answer and the reference, then take cosine similarity:

from openai import OpenAI
import numpy as np

client = OpenAI()

def embed(texts: list[str]) -> np.ndarray:
    resp = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return np.array([d.embedding for d in resp.data])

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

def test_answer_matches_reference(case):
    v = embed([generate_reply(case["input"]), case["reference"]])
    score = cosine(v[0], v[1])
    assert score >= 0.80, f"{case['id']}: similarity {score:.2f} below 0.80"

Pick the threshold by measuring, not by guessing. Run the suite, print every score, and look at the actual distribution — you are looking for a clear gap between "obviously fine" and "obviously wrong", and you set the line inside that gap. In our testing on short support replies, correct answers cluster above 0.85 and incorrect ones fall below 0.72, which makes 0.80 a defensible cut. Your numbers will differ; derive them from your own data.

Embeddings are cheap but not free. Batch the calls, reuse the reference embeddings across runs (compute once, cache), and never embed inside a loop over every test case.

7. LLM-as-Judge With a Rubric

For genuinely open-ended quality — tone, helpfulness, whether the answer actually answered the question — a stronger model scoring a weaker one's output is the practical option. The failure mode is a vague prompt like "rate this answer 1-10", which produces noise. Give the judge a rubric and demand structured output:

from pydantic import BaseModel, Field

class Score(BaseModel):
    relevance: int = Field(ge=1, le=5)
    groundedness: int = Field(ge=1, le=5)
    tone: int = Field(ge=1, le=5)
    reason: str = Field(max_length=200)

JUDGE = """Score the assistant reply against the user request.
1 = unusable, 3 = acceptable with flaws, 5 = excellent.
Judge only the reply. Ignore style you personally prefer."""

def judge(user_input: str, reply: str) -> Score:
    resp = client.beta.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": JUDGE},
            {"role": "user", "content": f"Request: {user_input}\nReply: {reply}"},
        ],
        response_format=Score,
        temperature=0,
    )
    return resp.choices[0].message.parsed

def test_judge_scores_acceptable(case):
    s = judge(case["input"], generate_reply(case["input"]))
    assert s.relevance >= 3, f"{case['id']}: relevance {s.relevance} — {s.reason}"
    assert s.groundedness >= 3, f"{case['id']}: {s.reason}"

Two things make this trustworthy. Temperature 0 plus a structured schema keeps scoring stable run to run. And assert on individual dimensions, not an average — an answer can score 4 on average while being a 1 on groundedness, and that is exactly the regression you care about.

Validate the judge before trusting it: hand-score twenty cases yourself and compare. If your agreement is below roughly 80%, the rubric is ambiguous — split the criteria into sharper questions rather than tuning thresholds.

8. Evaluating RAG Pipelines

RAG has a specific failure that no generic eval catches: the answer is fluent, plausible, and unsupported by anything you retrieved. Evaluate retrieval and generation separately, or you will not know which one broke:

• Retrieval. Did the expected source chunk appear in the top-k? Assert on chunk IDs, not text.
• Groundedness. Does every claim in the answer trace to retrieved context? This is the best use of an LLM judge — pass the context and ask whether each sentence is supported.
• Answer relevance. Does it address the question, regardless of source?

def test_retrieval_recall(case):
    hits = retrieve(case["input"], k=5)
    ids = [h.chunk_id for h in hits]
    assert case["expected_chunk_id"] in ids, (
        f"{case['id']}: expected chunk missing, got {ids}"
    )

Retrieval recall is free and fast, so run it on every commit. Groundedness needs a judge, so run it nightly. Our RAG pipeline with LlamaIndex guide covers the ingestion side; libraries like ragas and deepeval bundle these metrics if you would rather not hand-roll them.

9. Running Evals in CI Without Burning Budget

A suite that costs $4 per pull request will be switched off within a month. Keep CI cheap and put the expensive checks on a schedule:

# pyproject.toml / pytest.ini
[pytest]
markers =
    eval: LLM evaluation tests (may call a paid API)

# CI: fast, free checks on every push
pytest -m "not eval"

# Nightly job: full eval suite
pytest -m eval

Three practices keep it sustainable. Sample in CI — run the full deterministic suite but only a random subset of judged cases on pull requests, using pytest -k or a --eval-sample=0.3 flag you implement in a fixture. Set a hard spend cap in the eval fixture that fails loudly when cumulative tokens exceed a budget. And flake-guard the judge by running each judged case twice on failure before reporting, because a single bad score is not a regression.

Track the score over time, not just pass/fail. Appending {"date": ..., "similarity_avg": 0.87, "judge_avg": 4.1} to a file each night gives you the chart that actually answers "did this prompt change help?"

10. Common Mistakes

Mistake 1: Asserting on Exact Strings

Non-deterministic output plus exact matching equals a suite that fails randomly. Match on properties, labels, or similarity. Reserve exact matching for genuinely enumerable outputs.

Mistake 2: No Golden Dataset, Just Vibes

Testing against three inputs you typed into a REPL is not evaluation. Commit a dataset file and grow it with every real failure.

Mistake 3: Using an LLM Judge for Everything

It is slow, costs money, and drifts. A schema check or a substring check catches the same regression for free roughly half the time. Start cheap, escalate only where cheap has no signal.

Mistake 4: Testing Only the Happy Path

Your model will receive empty strings, 10k-token inputs, and "ignore previous instructions". Add those cases now. They are also the ones that reveal cost explosions.

Mistake 5: Ignoring Cost and Latency

Quality is one axis. Assert max_tokens stay bounded and measure wall-clock per case — a prompt change that doubles latency is a regression even if every answer is perfect.

Mistake 6: Never Re-checking the Reference Answers

Reference answers rot. When a support policy changes, the "correct" answer in your dataset becomes wrong and the suite starts failing legitimate outputs. Review the dataset quarterly and date every entry.

Bottom Line

Evaluation is what turns an LLM demo into a maintained feature. The sequence is short: commit a JSONL dataset built from real failures, add the cheapest assertion that has signal, escalate to similarity and then to a judge only where cheaper checks are blind, and keep the expensive parts on a nightly schedule.

Start today with sections 3 and 4 — a dataset file and one parametrized test. That takes under an hour and already catches the regressions that currently reach production. Add the judge when you have a specific quality question nothing else can answer.