Build a regression test for an AI release classifier
Score a prompt on labeled examples before changing models or quantization. The test runner uses only Python.

In this article
Define the output contract
Suppose a news desk classifies source notes as release, preview or other. Write one sentence defining each label. “Release” means an artifact readers can obtain now. Reserve “preview” for a proposed or prerelease artifact. Treat plans without an artifact as “other.”
Collect independent inputs
Start with the three synthetic fixtures below. Give the model only each note and the label definitions, then save its responses as a JSON array of strings in predictions.json in the same order. Keep the answers in the scorer separate from the model prompt.
1. Stable version 2.0 is available for download.
2. The team published 3.0-rc1 for testing.
3. A roadmap outlines work for next year.Score failures before retuning
Save the scorer below as score.py and run python3 score.py. Its exit status is nonzero on a mismatch. Inspect the individual mistakes. A missing response or an unexpected label is a failed case.
import json
from pathlib import Path
expected = ["release", "preview", "other"]
actual = json.loads(Path("predictions.json").read_text())
if not isinstance(actual, list) or len(actual) != len(expected):
raise SystemExit("Expected three answers in order")
for i, (want, got) in enumerate(zip(expected, actual), 1):
print(i, "PASS" if got == want else "FAIL", repr(got), "expected", want)
raise SystemExit(int(actual != expected))Make the test worth keeping
Before comparing a hosted model with a local quant, add held-out real examples, including ambiguous release candidates. Freeze the prompt and dataset. Record model version, retries, latency and cost alongside accuracy. Repeat variable runs and inspect cases where a cheaper model silently changes its answer.
An accuracy number needs a denominator
A run with two correct predictions out of three earns 67 percent. A single changed answer therefore moves the score by about 33 percentage points. Use the three examples to prove the harness works. Build the decision set from real cases such as a stable tag, a candidate announced before download, a project that calls a beta “available,” a skipped response and a roadmap disguised as release notes.
Suppose a model marks two prereleases as stable while keeping its overall score high. A newsletter could announce those builds as ready for production. Count errors by expected and actual label so you can see that direction of failure. Consider a misclassified stable release and a misclassified preview separate editorial costs. You could initially gate releases through human review until that error rate meets your threshold.
Put a candidate in front of the same evidence
Record the old model, prompt, source passage, expected label and exact output before testing a replacement. Keep the release candidate prompts out of training examples used to improve the prompt. Run both candidates through the same scorer. For malformed output, track the parse failure alongside label errors. If retries turn malformed outputs into correct responses, count the retries as latency and cost.
Before changing a score threshold, inspect whether the expected label is well defined. If two editors disagree about a public beta, update the rubric and record the reason. An evaluation should represent what the editorial process means by success.
Primary sources
OpenAI · Evaluation best practicesRead next Local or cloud? Follow the work.