November 14, 2026 · The CompletionKit team

How to reduce AI hallucinations through better prompt testing

You reduce AI hallucinations by testing the prompt, not by asking the model nicely. Run the prompt over a fixed dataset that includes the inputs where it is likely to make things up, score every output for whether the source it was given supports it, and re-run that test on every change. A clearer instruction, better context and a line telling the model not to invent details are all worth doing. None of them tells you whether the model got more reliable. The test does.

Picture a support assistant that tells a customer they qualify for a refund. The refund policy says they don't. The reply is polite, well written and wrong. If the only test was asking a few questions and reading the answers, nobody caught it, because a hallucination reads exactly like a correct answer until you check it against the source.

What AI hallucinations are

An AI hallucination is an output that presents something as fact when nothing the model was given backs it up. Computer vision used the word first, but it caught on for language models through a 2018 Google paper by Katherine Lee and colleagues, Hallucinations in Neural Machine Translation, which used it for translations "completely untethered from the source material". Two years later Joshua Maynez and colleagues at Google Research gave it the split most people still use. In On Faithfulness and Factuality in Abstractive Summarization (ACL 2020) they sorted hallucinations into intrinsic ones, which distort what the source says, and extrinsic ones, which add content the source never contained. Their annotators found one or the other in more than 70% of the single-sentence summaries they read.

That split is still the most practical way to think about what you are testing for, because the two kinds fail in different places.

• Intrinsic: the model gets the source wrong. The quarterly report says revenue rose 8%; the summary says 12%. A name is swapped, a date moves, a "may" becomes a "will".
• Extrinsic: the model adds what the source doesn't have. It invents a policy exception, a product feature, a statistic, or a citation to a page that doesn't exist.
• False certainty, the extrinsic case that matters most in apps. The model gives a definite answer to a question its context can't answer, instead of saying so.

Underneath all three is the same mechanism. A language model writes the most likely next words given what it has seen. It does not check each claim before it makes it, so an answer can be fluent, detailed and wrong at the same time.

Why a better prompt alone won't prevent AI hallucinations

The usual first fix is one more line in the system prompt: "Only state facts from the documentation. If you're not sure, say you don't know." That might help. It might do nothing. An instruction is not a guarantee, and a model can follow a new rule on five questions and ignore it on the sixth. It can also overcorrect, and start refusing questions it could answer perfectly well.

"Don't hallucinate" also leaves the real decision open. When the answer is missing, should the model refuse, ask a follow-up question, or give a partial answer? Until the prompt says, the model decides on its own, and even once it says, you still have to check that the model does it.

Adding context has the same gap. Retrieval (RAG) gives the model the documents it needs, which cuts down on answers from memory, but the retrieved page can be stale or beside the point, and the model can still misread a number or blend two sources. More context is not the same as checked output.

And the habit that ties them together, editing the prompt, trying four examples and shipping because they look better, tells you nothing about the inputs you didn't try. Those are where the hallucinations live.

How to reduce AI hallucinations with prompt testing

Reducing hallucinations is an evaluation problem as much as a prompting one. You are not hunting for the one instruction that makes a model accurate. You are finding the inputs where it invents things, putting a number on how often, and making sure each change lowers that number without raising another.

1. Build a dataset out of the cases that fail. Start from real traffic: production logs, support tickets, the summary a customer complained about. For a support assistant, include questions the docs answer plainly, questions about policies that aren't documented, questions built on a false premise, and every past conversation where the assistant made something up. Give each row the source context the model sees, and an expected answer where one exists. The guide to building a prompt eval dataset covers where to find these rows and how many you need.
2. Score the current prompt before you change it. That run is your baseline. Without it you can't tell a real improvement from a prompt that merely reads better. Keep the dataset fixed while you compare versions; if you change the rows and the prompt at once, you have moved the ruler along with the thing you are measuring.
3. Test whether the model knows when not to answer. Give the assistant a policy that says refunds are available within 30 days, and ask about a purchase from 75 days ago. A good reply says the purchase is outside the window; it doesn't promise an exception. Then take the policy out of the context and ask again. Now the right answer is that it can't tell, and a reply that quotes a refund window anyway is a hallucination. Keep both rows, because you want to catch the assistant that guesses and the one that refuses everything.
4. Score groundedness against the source the model was given. The question is narrow: is every claim in this output supported by the context in this row? You don't need to know the product to see that a summary which says "every plan" contradicts a document that says "Enterprise plan". A fair paraphrase passes; a price that appears nowhere in the source fails.
5. Check the judge before you trust it. An LLM judge can miss a subtle error, reward confident prose, or score the same output differently twice. Review a sample of its scores by hand and see how often it agrees with you before you let its number gate anything. Where it disagrees, fix the rubric; a coding agent can do that rewrite against your reviews.
6. Re-run the test on every change, and read the rows that broke. A fix for one hallucination can break cases that already worked. The average can go up while one critical row goes from right to invented.

A groundedness rubric for the judge

Some hallucinations a script can catch. If the source says $49 and the output says $59, a check that the output contains "$49" fails, every time, with no model involved. Use deterministic checks for anything like that: numbers, dates, IDs, a required citation marker, a phrase the output must never contain.

The rest needs reading. A support reply can have four correct sentences and one invented one; a summary can keep the whole document and add a conclusion nobody wrote. That is the job for an LLM judge: a second model that reads the input, the source context and the output, and scores the output against a rubric you wrote. Here is a groundedness rubric, scored 1–5 with 5 the best.

Two cautions. Groundedness says the output matches its source, not that the source is true; a model that faithfully repeats an outdated help page scores a 5. And a judge asked to rule on factual accuracy without the evidence in front of it is just another model guessing, so make sure the source context travels with every row.

Regressions, and model swaps

Here is the trap the tightened prompt sets. The assistant sometimes invents policy exceptions, so you add "Never answer unless every detail is stated in the documentation." The invented exceptions stop. So do the answers to simple questions that need two documented policies read together. The new prompt is more cautious, not better, and only a per-row comparison against the previous version shows you that. Running that comparison on every change is prompt regression testing, and it works best when every prompt version is tied to the run that scored it, so you can go back to the one that worked.

A model upgrade is a change of the same kind. A newer or cheaper model can handle missing information differently from the one you tested on, and a public benchmark score tells you nothing about how it treats your refund policy. Run the same dataset and the same metrics against the candidate model and read the rows that moved. Once all of this is a fixed dataset and a baseline, it belongs in your build: the CI/CD for LLM applications guide shows a job that fails when a critical check fails or scores drop past a threshold you set.

How to tell whether hallucinations are going down

No single number covers every hallucination, but you can count the ones you care about. The plainest is a hallucination rate: the share of outputs in the run with at least one unsupported claim. It only means something against the same dataset, though. A set packed with hard cases on purpose will always look worse than a sample of ordinary traffic, so compare versions on the same rows, never across datasets. Track a few numbers together, because each one hides something the others show.

Weigh the failures, too. An invented detail in a casual writing assistant is not the same as an invented account balance. The average tells you whether something moved; the failing rows tell you what to fix.

What prompt testing won't fix

Prompt testing reduces AI hallucinations. It doesn't end them. A well-tested prompt can still fail on an input nobody thought of. A grounded answer can repeat a wrong source. Retrieval can hand the model the wrong document before the prompt ever runs, and a judge can miss a hallucination it should have caught. Depending on what's at stake you may also need retrieval tests, structured output validation, outside verification and a human in the loop for high-risk answers.

What testing gives you is a repeatable way to improve the part you control. When a new hallucination turns up in production, put the input that caused it in the dataset, fix the prompt, re-run, and check that nothing else broke. The system gets more reliable because the same mistake gets harder to repeat, not because someone wrote the perfect prompt.

Testing for hallucinations in CompletionKit

In CompletionKit, a run takes a prompt version and a dataset, generates an output for every row, and scores each one. There is a Correctness starter metric that asks the judge whether the output is factually right and free of made-up information, and you can write your own groundedness rubric like the one above. The judge sees each row's input data, so the source context goes with it. Exact facts go to checks, which pass or fail: equals or numeric equals against the row's expected value, contains, not contains, and regex. When you compare two runs, each row is marked fixed, broke or same, so the regression is on the page instead of buried in an average.

You can review the judge's scores, record your own, and see how often it agrees with you before you rely on it; your corrections go back to the judge as examples. If the outputs already exist, in production logs or from another system, a judge-only run scores them without generating anything. Your CI job can start the same runs through the REST API, and a coding agent can start them through the MCP server. The code is source-available (BSL 1.1) and self-hostable, no model lab owns the company, and you pay per team, not per seat (see pricing). The docs walk you through a first dataset and run.

FAQ

Test for hallucinations free   What is prompt evaluation?

Built by Homemade Software. Disagree with any of this? support@completionkit.com or r/completionkit.

← All posts