Skip to content

OpenAI Evals shutdown: move prompts and evals by November 30

OpenAI shuts down its hosted Evals product and its prompts API on November 30, 2026. If your app fetches a stored prompt from OpenAI by its id, or your tests live in the Evals dashboard, you have until then to copy them out and replace them. OpenAI announced both on its deprecations page on June 3. The open-source openai/evals repo on GitHub is a separate project and is not on that list.

What stops working on November 30

On November 30 OpenAI turns off /v1/prompts and every stored prompt, plus the Evals dashboard, the Evals API and its graders. The prompts shutdown is the one that can break your live app. A typical support app makes a call like this, with the prompt text stored on OpenAI's side:

response = client.responses.create(
    prompt={
        "id": "pmpt_abc123",
        "version": "4",
        "variables": {"customer_name": name, "issue": issue},
    },
)

OpenAI hasn't said what that call returns after November 30, so plan for it to fail. Search your code, config and environment variables for pmpt_, and search your code for Responses calls that pass a prompt argument.

The Evals shutdown won't break your app, but you lose the evals, graders and run results stored there. Existing evals went read-only on October 31, according to OpenAI's Evals guide, so you can't change them, but you can still read them through the API until November 30.

What to save before November 30

For each stored prompt, copy these into your own repo or prompt store:

  • The text of the version your app pins. In the example that is version 4, even if someone has since saved a version 5. If your call leaves out version, your app gets the current version, so copy that one.
  • The model, temperature or reasoning effort, tools and response format.
  • The variable names, customer_name and issue in the example.

OpenAI has no export tool for evals. Its own migration guide to Promptfoo has you recreate test cases and graders by hand, so pull the data through the API now. GET /v1/evals lists your evals. For each one, save the three things below.

  • The dataset. It is usually a JSONL file you uploaded through the Files API. The run's data_source names the file, and GET /v1/files/{file_id}/content returns it.
  • The grader prompt, word for word. GET /v1/evals/{eval_id} returns it under testing_criteria.
  • The scored results of your last good run, from GET /v1/evals/{eval_id}/runs/{run_id}/output_items.

Keep those results. They are the only way to check that your new grader agrees with the old one on the same rows.

Where to move your prompts and evals

For prompts, OpenAI's advice is to move the prompt text into your application code and pass it as input messages. That works if only engineers change prompts. If people outside engineering edit them, you need a prompt store your app fetches from, so their edits don't wait on a deploy. The prompt versioning post covers how that works.

For evals, OpenAI points to Promptfoo. Promptfoo is a good open-source CLI and runs evals against most providers' models. OpenAI agreed to buy Promptfoo in March 2026, so moving there keeps your evals with OpenAI. If you test models from more than one vendor, or don't want your eval tool tied to the vendor you're testing, look at the alternatives. We make the longer case in your eval tool shouldn't work for the model it's grading, and the 2026 roundup of prompt evaluation tools covers the options.

CompletionKit is one of them. You edit and test the prompt in CompletionKit, publish it, and your app picks up the new version without a deploy. Here is the call from above after the move:

ck = requests.get(
    "https://completionkit.com/orgs/acme/api/v1/prompts/support-reply",
    headers={"Authorization": f"Bearer {CK_API_KEY}"},
).json()

text = ck["template"].replace("{{customer_name}}", name).replace("{{issue}}", issue)

response = client.responses.create(model=ck["llm_model"], input=text)

To bring an eval across, convert the JSONL dataset to CSV with one column per variable, upload it, and recreate the grader as a metric. You can host CompletionKit yourself; the code is source-available under BSL 1.1. CompletionKit vs Promptfoo compares the two side by side.

Try CompletionKit free

Built by Homemade Software. Disagree with any of this? support@completionkit.com or r/completionkit.

← All posts