November 14, 2026 · The CompletionKit team

AI quality assurance: best practices for testing prompts and outputs

AI features are easy to demo and hard to trust. A prompt produces ten convincing answers in the playground, then meets real customer input and does something nobody saw. A one-line wording change fixes the case that prompted it and quietly breaks three others. A model upgrade shifts the tone and the formatting without throwing a single error. None of that shows up in a unit test, because the code is fine. The output is what moved.

AI quality assurance is how a team checks that the prompts and outputs behind an AI feature are good enough to ship, and that they stay that way as prompts, models and inputs change. You write down what a good output looks like, collect a fixed set of real inputs, grade each output with code checks and an LLM judge, and compare each proposed change against the version you already ship. Ordinary QA still covers the rest of the app (the API responds, the button works, the data lands where it should). AI quality assurance covers the part that is new: an output that can be different every time and still be right, or wrong.

Why quality assurance for AI needs a different approach

Manufacturing hit the same problem a century ago. In May 1924 Walter Shewhart, an engineer at Western Electric, sent his boss a memo about a page long, a third of it taken up by a sketch of what became the control chart. His point was that every process varies. Some variation is chance and some has an assignable cause, and the job of quality control is to tell the two apart rather than to pretend the variation away. You stop inspecting one part at a time and start measuring the process.

LLM output is a process in exactly that sense. Send the same input through the same prompt twice and you can get two different answers, both fine. So an exact-match assertion is the wrong instrument for most of what matters, and a single good example proves very little. What works is the Shewhart move: measure across many inputs, know how much the score moves on its own, and treat a drop beyond that as a cause worth finding. Everything below is a way of doing that for prompts.

Write down what good means before you test anything

You cannot test a prompt until you know what a passing output looks like, and this is the step teams skip most. "This one feels better" is fine while exploring. It is not a standard anyone else can apply, and it is not one a judge can apply either.

Turn each product expectation into something observable. "Good customer support" is hard to score. "The reply answers the customer's actual question, gives one clear next step, and does not promise anything the company cannot do" is easy to score, by a person or a model. Depending on the feature, the criteria might cover accuracy, completeness, faithfulness to a source, tone, length, instruction following, required fields in structured output, or claims the output must never make.

For criteria a judge will score, anchor each point of the scale. In CompletionKit a judge metric scores 1–5, with 5 the best, and each band carries its own description. A 5 that means "quotes the order number verbatim" and a 3 that means "acknowledges an order exists" give the judge, and the next engineer, far less room to guess than one line asking whether the reply is good.

Test every change against a fixed dataset of real inputs

A handful of examples typed into a playground is how you develop a prompt. It is not how you check one. For that you need a dataset that stays put, so the score next week means the same thing as the score today. The guide to building a prompt eval dataset covers sourcing and sizing in detail; the short version is to draw on four kinds of row.

• Representative inputs. The ordinary requests that make up most of your traffic.
• Edge cases. The ambiguous, very long, badly formatted or adversarial inputs.
• Known failures. Every input that has produced a bad output before.
• High-stakes cases. The requests where a wrong answer costs you a customer, a refund or a compliance problem.

Then hold the dataset fixed while you compare prompt versions. If the prompt and the rows both change between runs, a moved score could be either, and you have learned nothing.

Pair deterministic checks with an LLM judge

Some requirements have a right answer and some need judgment, and each kind deserves its own instrument. If code can decide it, let code decide it: a check is instant, costs nothing to run and never drifts. Save the judge for questions that need reading comprehension. Run both over the same rows and you get one picture instead of two half-pictures.

A check scores each row pass or fail. A judge scores it 1–5 and writes a rationale saying why. The LLM-as-a-judge guide covers how to set the judge up so its number holds.

Set a baseline, then compare every change to it

An improvement needs something to improve on. Before the next edit, run the prompt you ship today over the dataset and keep the result, row by row. That run is the baseline, and every later change gets the same treatment: the system prompt, the few-shot examples, the model or its version, the provider, temperature, retrieval, the output format. Each one is a new run on the same rows, compared with the baseline.

Model upgrades belong on that list as much as prompt edits do. A model that does better on public benchmarks can still follow your formatting rules differently, run longer, or handle your borderline cases another way. The question is not whether the new model is better in general. It is whether it is better on your inputs, with your prompt, against your rubric, and a run answers that directly. Repeat that comparison for every change and you are doing prompt regression testing, which is the core of AI quality assurance.

Read the rows, not just the average

An average is a summary, and summaries hide things. A new version can lift the overall score while the rows that matter most get worse: a few dozen easy cases improve a little, five critical ones go from passing to failing, and the headline number still goes up. So read beneath it. Look for rows that used to pass and now fail, large drops on single rows, failures that cluster around one kind of input, and two metrics moving in opposite directions.

The rows also tell you what to fix. In our own workspace we ran a generic two-sentence support prompt, the kind that asks for a short, friendly answer to a shop customer, over eight customer messages that each contained an order number. A judge metric called acknowledges-order-number scored the replies. The prompt averaged 3.88. The average alone says "decent, could be better". The lowest row says much more: it scored 1, and the judge's rationale was that the reply never mentioned the customer's order or its number. We added one sentence to the prompt ("Acknowledge their specific order.") and reran it on the same eight rows with the same judge. It scored 5.0.

(Real runs in our workspace on June 12, 2026, with replies generated by gpt-4.1-mini and scored by claude-sonnet-4-6.)

Turn production failures into test cases

Testing before launch tells you whether a change looks safe given what you already know. Live traffic will always find inputs your dataset missed, and that is expected. What matters is that each one only gets to surprise you once.

Sample real outputs on a schedule and score them against the same rubric you use before release. In CompletionKit that is a judge-only run: put the outputs you already have in a column and the judge scores them, with no generation step. When something fails, add the input to the dataset, write down what should have happened, and every future version gets tested against it. Give it a year and the dataset is a list of every mistake the feature is no longer allowed to make.

Version prompts so every score has an identity

A score is only useful if you know what produced it. When a prompt lives as a string pasted across code, docs and someone's notes, "did anyone change the prompt last week?" has no good answer. Treat each revision as its own version, so every run ties a specific prompt version to a dataset, a model and a set of metrics. When a regression shows up, you can diff the version that scored well against the one that didn't, and roll back to it. Prompt versioning goes into the mechanics. In CompletionKit, editing a published prompt creates a new version, and your app pulls a published version by its URL at runtime, which means what you tested and what you serve are the same text.

Keep your tuning examples apart from your test set

Tune a prompt long enough against the same few examples and it gets very good at those examples. That is not the same as getting better. It may just have learned the test.

Machine learning settled this long ago with the held-out test set, and the same rule applies to prompts. Keep the rows you look at while editing separate from the rows you use to decide whether to ship. For a high-stakes feature, keep a second dataset you don't open until a version looks ready, and run the candidate against it once. If the score holds there, the improvement is real. If it drops, you tuned to your examples.

Gate releases in CI/CD

An evaluation that has to be remembered gets skipped the week it matters. If prompts are part of your production system, they need a release process like the rest of it. The CI/CD for LLM applications post walks through a GitHub Actions job that does this over the REST API. The shape is simple.

1. Someone proposes a new prompt version.
2. CI runs it over the evaluation dataset.
3. Every output is scored by the checks and the judge.
4. The scores are compared with the current production baseline.
5. The build fails if a gate is crossed, and someone reads the rows that moved.

Set the gates from what the feature actually needs, not from a wish for perfect scores. A safety or format check that must always pass is a hard gate. A judge score that wanders a little from run to run on its own needs a tolerance wider than that wander, or CI will fail green builds and people will learn to ignore it.

Check the judge as well

An LLM judge is a model too. It can favour longer answers, read a vague rubric differently from your team, and score the same output differently from another judge model. If you take its score on trust, your quality assurance rests on an instrument nobody has checked.

So check it. Have people rate a sample of the same rows the judge scored, and look hard at the disagreements. When your team keeps calling an output good and the judge keeps calling it poor, the cause is usually an unclear rubric, missing context, the wrong judge model, or a scale whose bands are hard to tell apart. In CompletionKit you mark each judged row agree, disagree or borderline, give a corrected score when you disagree, and see where the judge and your team part ways. When you rewrite a rubric, you can compare the two versions' agreement with your verdicts side by side before publishing the new one. Letting a coding agent improve the judge shows how to run that loop over MCP.

The goal is not perfect agreement on every subjective case; people disagree with each other too. It is enough agreement that when a score moves, the movement means something.

Start with an AI quality assurance loop small enough to run

None of this needs thousands of rows or a dozen metrics on day one, and building too much up front is a good way to end up running none of it. Start with a fixed dataset of a few dozen real inputs, two or three metrics with written rubrics, checks for the hard rules, one baseline run, and a comparison after every change. Add the rest when the product gives you a reason: a row because something failed, a metric because a requirement appeared, a CI gate because a regression cost you. The best AI quality assurance process is the one that actually runs before a change reaches users.

Where CompletionKit fits

CompletionKit is a prompt evaluation tool built around that loop. Point a prompt at your inputs and a judge grades every output against the rubric you wrote, next to whatever code checks you add. You then test the judge against your own verdicts, and your app pulls the winning version at runtime. It runs prompts on Anthropic, OpenAI, OpenRouter, Azure AI Foundry or self-hosted Ollama models with keys you supply, and you can drive it from the app, the REST API, or the MCP server for coding agents.

Other good tools cover much of the same ground; the 2026 roundup is an honest read on who each one suits. CompletionKit's case rests on its focus on prompts and outputs, the whole loop in one place, and pricing that charges a team one flat price rather than a fee per seat, so the people who review outputs don't each need a licence. It is also independent of the model labs whose models it grades. We host it for you, or you can run the engine on your own servers; it is source-available under BSL 1.1.

FAQ

Test your prompts free   Prompt regression testing

Built by Homemade Software. Disagree with any of this? support@completionkit.com or r/completionkit.

← All posts