November 14, 2026 · The CompletionKit team

LLM application requirements for production-ready AI apps

The LLM application requirements that decide whether an AI feature survives production are mostly about its outputs. You need a written definition of a good answer, broken into criteria precise enough to fail; a set of real inputs to test against; a scorer for each criterion; and a habit of re-running that test every time the prompt or the model changes. Rate limits, retries and dashboards matter as well, but they keep the feature running. They do not tell you whether what it says is right.

Getting a prompt to produce one impressive answer in a playground takes an afternoon. Keeping it producing good answers through a dozen prompt edits, two model upgrades and six months of product change is the actual job, and it is where most teams find out their requirements were never written down.

Why an LLM application needs different requirements

Ordinary code is close to deterministic. The same function with the same input returns the same value, so a unit test can assert on it. A language model gives you no such promise. Two differently worded answers can both be correct. A one-line prompt edit can fix the case you were looking at and quietly break three you were not. A model upgrade can sharpen the reasoning and change the tone, the formatting, the length and the refusal behaviour all at once.

So the questions a production team has to answer are not the ones a unit test answers. Did this prompt change make the outputs better? Which kinds of input got worse? Does the output still follow the format and the policy? Is the new model worth what it costs? Each of those needs evidence, and the evidence only exists if the requirement behind it was stated in a form you can measure.

Write each requirement so it can fail

The software world settled this a long time ago. IEEE 830, the 1998 recommended practice for software requirements specifications, listed "verifiable" among the qualities of a good requirement: if there is no finite, cost-effective way to check whether the product meets it, it is not really a requirement. LLM features break that rule constantly. "Produce a good customer support reply" goes into the spec, and nobody can say whether a given reply meets it.

Rewrite it as four things that can each be wrong on their own: the reply answers the question the customer asked; it follows the refund policy; it makes no claim the policy and the ticket do not support; it stays polite without grovelling. Now each line is a test. A reply can be polite and wrong about the refund window, and you will see exactly that, rather than a vague sense that the bot got worse.

The usual candidates are accuracy, relevance, completeness, tone, format, policy compliance, valid structured output, correct tool choice, citations that point at real sources, no invented facts, and refusing what it should refuse. You will not need all of them. You need the ones that would embarrass you if they slipped, written specifically enough that two people reading the same output would agree whether it passed.

Match each requirement to a check or a rubric

Once a requirement is written down, the next question is how to score it. Some requirements have one right answer and can be checked exactly, which is fast, free and never argues with you. Others are open-ended, and need a judge reading the output against a rubric. Using the cheap tool wherever it fits leaves the judge for the questions that actually need judgement.

The rubric carries the weight on the bottom half of that table. "Rate the tone from 1 to 5" gets you a noisy number. A rubric that says what a 5 looks like, what a 3 looks like and what drags a reply down to a 1 gets you a score two judges would roughly agree on. The LLM-as-a-judge guide covers how to write one.

Test against inputs your users actually send

A requirement is only as good as the inputs you test it on. Twenty clean examples someone typed up on a Tuesday will pass almost any prompt. The inputs that find problems come from production logs, support tickets, the bug reports where the feature misfired, and the strange cases QA stumbled into: the email with three timezones in it, the question in two languages, the user who pastes their whole contract into the box.

Give the hard cases more room than their share of real traffic. If one request in twenty is ambiguous, a dataset that is one-twentieth ambiguous will hide a regression on exactly the inputs that cause complaints. Synthetic examples can fill a specific gap, but they should not be the whole set. How to build a prompt evaluation dataset goes through sourcing and sizing in detail.

Hold every change to the same bar

The requirements, the scorers and the dataset together make a fixed bar. The point of having one is that every change gets measured against it, the same way, before it ships.

1. Score the live prompt on the dataset and keep those numbers as the baseline.
2. Make the change: a prompt edit, a new model, a different temperature.
3. Run the same dataset with the same metrics.
4. Compare row by row, not just the averages, and read the rows that moved.
5. Ship when the evidence supports it, and not before.

That loop needs the prompt to be versioned, so you know exactly which text and which model produced each set of scores, and it is what prompt regression testing means in practice. Wire it into your pipeline and a drop in scores fails the build the way a broken unit test does; CI/CD for LLM applications has a worked GitHub Actions job.

Model changes deserve the same treatment as prompt changes. Treat the model as a dependency of the application, not its identity. When a new model lands, run it over your dataset with your metrics and look at quality next to latency and cost. A bigger model that scores a little higher may not be worth twice the wait; a smaller one that holds its scores on a narrow task may be the better choice. A public leaderboard cannot tell you which, because it was not scored on your inputs or your rubric.

Keep the metrics separate

A single "quality" number hides the regressions you most need to see. Suppose a prompt edit makes replies clearer and friendlier while getting the refund window wrong more often. Averaged together, the score barely moves. The product got worse.

Score each requirement as its own metric and the trade-off is visible: tone up, accuracy down, and you can decide whether that is acceptable instead of shipping it by accident. A useful test for any metric is whether someone on the team can say what goes wrong for users when it drops. If nobody can, it probably does not belong in the release gate.

The judge needs requirements too

Once a judge is scoring half your requirements, the judge itself becomes something that can be wrong. Zheng and colleagues' 2023 paper on judging chatbots with MT-Bench and Chatbot Arena, which popularised the term LLM-as-a-judge, also documented the ways judges fail: favouring the first answer they read, preferring longer answers, and rating their own model's outputs more kindly. A rubric that reads well can still produce scores your team would not give.

The requirement here is agreement with people. Have someone on the team read a sample of the judge's scores and rationales and mark each one agree or disagree. If the disagreements cluster, the rubric is underspecified, and the cases where you disagreed are the material for tightening it. A judge you have checked against human verdicts is one you can let gate a release; one you have not checked is a second opinion of unknown quality.

The operational requirements, briefly

A production LLM application has ordinary engineering requirements as well. Providers rate-limit and time out, so requests need queuing, retries with backoff and sensible timeouts. Structured output occasionally breaks, so it needs validating before anything acts on it. Some features need a fallback model or a human handoff when the first answer fails. Long jobs belong off the request path. Production needs logging of which prompt version and model served each request, with token counts, latency and errors, so a bad answer a user reports can be reconstructed and added to the dataset as a new test case.

Those belong to your infrastructure and your observability tooling, and they are well covered elsewhere. They keep the feature up. The requirements above are the ones that keep it right.

A checklist of LLM application requirements

Before calling an AI feature production-ready, you should be able to say yes to most of these:

• Good output is defined in writing, as separate criteria that can each fail.
• Each criterion has a scorer: a deterministic check where there is one right answer, a rubric where there is not.
• The dataset comes from real inputs and over-represents the hard cases and past failures.
• Prompts are versioned, and every score can be traced to the prompt text and model that produced it.
• Prompt and model changes are re-run against the same dataset and compared row by row before they ship.
• The comparison runs automatically and can block a release.
• The judge has been checked against human verdicts on a sample of its scores.
• Quality, latency and cost targets are written down, so a model swap has something to be measured against.
• A bad answer reported in production becomes a new row in the dataset.

If several of those are no, the feature may still work. Changing it safely is what gets harder every month.

Where CompletionKit fits

CompletionKit covers the output half of that checklist as one loop. You run a prompt over a dataset of real inputs; an LLM judge scores each output against your rubric on a 1 to 5 scale, alongside deterministic checks that pass or fail; you mark the judge's scores agree, disagree or borderline, with a corrected score whenever you disagree, and the trust panel on each metric shows how closely the judge matches you; you compare runs row by row; and your app fetches the prompt from a stable URL at runtime, so publishing the version that won the comparison is all it takes to serve it. The same runs are available over the REST API for CI and over an MCP server for coding agents.

Braintrust, Langfuse and Promptfoo cover much of the same ground, and the tool comparison is honest about where each one wins. CompletionKit's case is focus on prompt and output evaluation, flat pricing for the whole team with no per-seat charge, a source-available engine you can self-host under BSL 1.1, and no ownership tie to a model lab. The docs walk through a first dataset, metric and run.

FAQ

Test your prompts free   What is prompt evaluation?

Built by Homemade Software. Disagree with any of this? support@completionkit.com or r/completionkit.

← All posts