API reference: CompletionKit

MCP Server

Connect Claude Code, Cursor, or any MCP client to manage prompts, runs, datasets, and metrics conversationally. 54 tools over streamable HTTP.

claude mcp add --transport http completion-kit \
  https://completionkit.com/orgs/your-org-slug-goes-here/mcp \
  --header "Authorization: Bearer YOUR_TOKEN"

{
  "mcpServers": {
    "completion-kit": {
      "url": "https://completionkit.com/orgs/your-org-slug-goes-here/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_TOKEN"
      }
    }
  }
}

Available tools

prompts 7

prompts_list List all prompts

prompts_get Get a prompt by ID

prompts_create Create a prompt

prompts_update Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing — promote it with prompts_publish — so an agent's edits don't go live without a gate. If it has no runs, it is updated in place.

prompts_delete Delete a prompt

prompts_publish Publish a prompt version, making it the current version

prompts_suggest_improvement Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).

runs 9

runs_list List all runs

runs_get Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prompt down without listing responses.

runs_create Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.

runs_update Update a run

runs_delete Delete a run

runs_generate Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.

runs_regrade Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.

runs_rerun Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.

runs_retry_failures Re-run only the failed responses of a run, optionally limited to specific response ids via "only".

responses 2

responses_list List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low scorers, and sort "score_asc" to read the worst rows first. For per-metric averages of the whole run use runs_get instead of aggregating here.

responses_get Get a specific response

datasets 6

datasets_list List all datasets

datasets_get Get a dataset by ID

datasets_create Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and "actual_output" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also available to the prompt as a variable.

datasets_update Update a dataset

datasets_delete Delete a dataset

datasets_create_from_url Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the "expected_output" (answer key) and "actual_output" (pre-made output) columns are recognized specially, overridable per run.

metrics 6

metrics_list List all metrics

metrics_get Get a metric by ID

metrics_create Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.

metrics_update Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.

metrics_delete Delete a metric

metrics_suggest_variants Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.

metric 8

metric_groups_list List all metric groups

metric_groups_get Get a metric group by ID

metric_groups_create Create a metric group

metric_groups_update Update a metric group

metric_groups_delete Delete a metric group

metric_versions_list List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.

metric_versions_publish Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.

metric_versions_dismiss Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused — to demote a published version, publish a different one as current instead.

provider 5

provider_credentials_list List all provider credentials (API keys are not exposed)

provider_credentials_get Get a provider credential by ID (API key is not exposed)

provider_credentials_create Create a provider credential

provider_credentials_update Update a provider credential

provider_credentials_delete Delete a provider credential

tags 5

tags_list List all tags

tags_get Get a tag by ID

tags_create Create a tag. Color is auto-assigned.

tags_update Rename a tag.

tags_delete Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.

agreements 2

agreements_list List agreements. Filter by run_id, response_id, metric_id, or created_by.

agreements_create Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.

judges 2

judges_replay Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.

judges_compare Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.

promptfoo 1

promptfoo_import Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary of what mapped and what was skipped and why; nothing is dropped silently.

usage 1

usage_get Get this organization's plan usage and limits for the current billing period: runs and prompt fetches used, their limits, how many remain, and when the period resets. Call this to pre-check quota before starting runs. Runs are hard-blocked once the run limit is reached (with a small grace band), so a run over the limit will fail with run_limit_reached.

Prompts

Create, version, and manage LLM prompt templates.

GET /api/v1/prompts

List all prompts, ordered by most recent. Supports pagination (limit, offset) and tag[] filtering.

curl https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/prompts \
  -H "Authorization: Bearer YOUR_TOKEN"

POST /api/v1/prompts

Create a new prompt.

Required: name, template, llm_model Optional: description

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/prompts \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "summarizer", "template": "Summarize: {{text}}", "llm_model": "gpt-4.1"}'

GET /api/v1/prompts/:id

Get a single prompt by numeric ID or slug.

PATCH /api/v1/prompts/:id

Update a prompt. Accepts same params as create. If the prompt has no runs it is updated in place; if it already has runs, a new published version is created and returned (with a new id), leaving the original unchanged.

curl -X PATCH https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/prompts/1 \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"template": "Summarize concisely: {{text}}"}'

DELETE /api/v1/prompts/:id

Delete a prompt. Returns 204 No Content.

POST /api/v1/prompts/:id/publish

Publish a prompt version, making it the current version in its family.

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/prompts/1/publish \
  -H "Authorization: Bearer YOUR_TOKEN"

Runs

Create runs, generate LLM responses, and judge them with metrics.

GET /api/v1/runs

List runs with response counts and average scores. Supports pagination (limit, offset) and the following filters.

Optional filters: status (pending, running, completed, failed), prompt_id, dataset_id, tag[]

curl "https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs?status=completed&prompt_id=1&tag[]=real%20estate" \
  -H "Authorization: Bearer YOUR_TOKEN"

POST /api/v1/runs

Create a new run.

Optional: name, prompt_id, dataset_id, metric_ids, judge_model, temperature, judge_temperature (defaults to 0 so re-judging is reproducible; above 0 the same output can score differently), max_tokens (cap on generated tokens per row; leave unset for the provider default, set it to match production when your prompt's output is long enough to truncate), output_column (score existing outputs: omit prompt_id and grade a dataset column instead, default actual_output), expected_column (dataset column holding each row's answer key, given to the judge and to checks that compare against the row's expected value, default expected_output)

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"prompt_id": 1, "dataset_id": 1, "metric_ids": [1, 2]}'

GET /api/v1/runs/:id

Get a run with status, progress, response count, and average score.

Also returns: metric_averages, a per-metric breakdown with each metric's average score (pass_rate for checks), the number of rows it graded, and low_count, how many scored below the medium quality threshold

curl https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1 \
  -H "Authorization: Bearer YOUR_TOKEN"

POST /api/v1/runs/:id/generate

Start generating responses. Returns 202 Accepted. Poll the run to check progress.

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1/generate \
  -H "Authorization: Bearer YOUR_TOKEN"

POST /api/v1/runs/:id/retry_failures

Re-queue responses that failed during generation. Returns 202 Accepted, or 409 Conflict if the judge version has drifted since the run executed (use /rerun instead to avoid mixing versions in one run).

Optional: only (array of failed response IDs, to retry just a subset)

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1/retry_failures \
  -H "Authorization: Bearer YOUR_TOKEN"

POST /api/v1/runs/:id/rerun

Clone the run and start generating responses on the copy against the current prompt and metric versions. Returns the new run with 202 Accepted. Useful for capturing a fresh baseline after metric edits.

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1/rerun \
  -H "Authorization: Bearer YOUR_TOKEN"

POST /api/v1/runs/:id/regrade

Re-judge the existing successful responses against the current metric versions without regenerating model output. Returns 202 Accepted, or 422 if no responses are eligible.

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1/regrade \
  -H "Authorization: Bearer YOUR_TOKEN"

GET /api/v1/runs/:id/compare?with=:other_id

Side-by-side comparison against another run. Returns {left_run_id, right_run_id, metric_ids: [...], rows: [...]} with one row per input case, per-metric scores on both sides, and the delta. Rows are keyed by the left run's input cases: a case present only in the left run is returned with the right side nulled out, and a case present only in the right run is omitted.

curl "https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1/compare?with=2" \
  -H "Authorization: Bearer YOUR_TOKEN"

PATCH /api/v1/runs/:id

Update a run. Accepts same params as create.

curl -X PATCH https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1 \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "baseline v2", "metric_ids": [1, 2, 3]}'

DELETE /api/v1/runs/:id

Delete a run and all its responses. Returns 204 No Content.

Responses

Read-only access to generated responses and their review scores. Nested under runs.

GET /api/v1/runs/:run_id/responses

List responses for a run, including nested review scores.

Optional filters: status (pending, retrying, succeeded, failed), min_score, max_score, sort (id, score_asc, score_desc), plus limit and offset

Trim the payload: fields takes a comma-separated list of response keys and always returns id. Prefix with reviews. to trim each nested review, e.g. fields=score,reviews.metric_name,reviews.ai_score

curl "https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1/responses?sort=score_asc&limit=10&fields=id,score,reviews.metric_name,reviews.ai_score" \
  -H "Authorization: Bearer YOUR_TOKEN"

GET /api/v1/runs/:run_id/responses/:id

Get a single response with its review scores and feedback.

Datasets

Data used as input for runs.

GET /api/v1/datasets

List all datasets. Supports pagination (limit, offset) and tag[] filtering.

POST /api/v1/datasets

Create a dataset from inline CSV or an uploaded CSV file.

Required: name, and either csv_data (inline CSV) or a multipart file (CSV upload, preferred for large datasets)

First row is the header. Two column names are recognized specially: expected_output is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and actual_output is a pre-made output to score in a prompt-less run. Both are overridable per run via expected_column and output_column.

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/datasets \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "tickets", "csv_data": "text,expected_output\\nHello,Hi"}'

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/datasets \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -F "name=tickets" \
  -F "file=@tickets.csv"

GET PATCH DELETE /api/v1/datasets/:id

Get, update, or delete a dataset.

curl -X PATCH https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/datasets/1 \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "tickets-q3"}'

Metrics

Scoring dimensions used by the judge model.

GET /api/v1/metrics

List all metrics. Supports pagination (limit, offset) and tag[] filtering.

POST /api/v1/metrics

Create a metric. Defaults to an LLM judge; pass metric_type check with a check_config for a deterministic pass/fail check.

Required: name Optional: metric_type (llm_judge default, or check), instruction and rubric_bands (array of {stars, description}) for judges, check_config (check_kind, target, plus per-kind keys) for checks

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/metrics \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "relevance", "instruction": "Is the response relevant?"}'

GET PATCH DELETE /api/v1/metrics/:id

Get, update, or delete a metric.

curl -X PATCH https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/metrics/1 \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"instruction": "Is the response factually accurate?", "rubric_bands": [{"stars": 5, "description": "fully accurate"}]}'

Agreement loop

Drive metric improvement from disagree-flagged agreements: ask the model to rewrite the instruction and rubric into a new draft version.

POST /api/v1/metrics/:id/suggest_variants

Generate draft metric versions from the current disagreements. Only rubric-style (llm_judge) metrics can suggest variants. Returns 201 with the new draft versions, or 422 if the metric is a check (checks are exact and have no variants to suggest), if no disagreements exist, or if the model produced nothing usable.

Optional: count, model

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/metrics/1/suggest_variants \
  -H "Authorization: Bearer YOUR_TOKEN"

Metric versions

Every metric carries a history of versions (the current published one, prior published ones, and any draft suggestions). Reviews and agreements record the version they ran against, so the API can surface stale state and let you revert.

GET /api/v1/metrics/:metric_id/metric_versions

List every version for the metric, newest version_number first.

GET /api/v1/metrics/:metric_id/metric_versions/:id

Get a single version with its instruction and rubric bands (or check_config for checks), state, and source.

POST /api/v1/metrics/:metric_id/metric_versions/:id/publish

Publish the version as current. Works for a draft (promote) or a superseded published version (revert). Copies the version's instruction and rubric (or check_config for checks) back onto the metric.

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/metrics/1/metric_versions/5/publish \
  -H "Authorization: Bearer YOUR_TOKEN"

DELETE /api/v1/metrics/:metric_id/metric_versions/:id

Dismiss a draft version. Returns 204 No Content, or 409 Conflict if the version is published (published versions are immutable history).

Metric Groups

Named groups of metrics you can apply to a run as a set.

GET /api/v1/metric_groups

List all metric groups with their metric IDs. Supports pagination (limit, offset) and tag[] filtering.

POST /api/v1/metric_groups

Create a metric group.

Required: name Optional: description, metric_ids (array), tag_names (array)

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/metric_groups \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "core quality", "metric_ids": [1, 2, 3]}'

GET PATCH DELETE /api/v1/metric_groups/:id

Get, update, or delete a metric group. PATCH with metric_ids replaces all metric associations.

curl -X PATCH https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/metric_groups/1 \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"metric_ids": [4, 5]}'

Agreements

Per-verdict feedback events on a response/metric pair: agree, disagree (with a corrected score and note), or borderline. Agreements capture the metric version that was current when the verdict was cast, which is what drives the trust signal and the "stale" indicators across the rest of the API.

All agreement endpoints require the judge_agreement_enabled configuration (on by default). When it is disabled every agreement action returns 404 with {"error": "Agreement disabled"}.

GET /api/v1/agreements

List agreements across all runs. Supports filtering by any combination of the query params below.

Optional filters: run_id, response_id, metric_id, metric_version_id, created_by, verdict (agree, disagree, or borderline)

curl "https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/agreements?metric_id=1&verdict=disagree" \
  -H "Authorization: Bearer YOUR_TOKEN"

POST /api/v1/runs/:run_id/responses/:response_id/metrics/:metric_id/agreements

Cast an agreement on a specific response/metric pair. The metric version on the record is set automatically from the run's review. Idempotent per (response, metric, created_by): the first cast creates the agreement (201 Created), and re-casting for the same created_by updates it in place (200 OK) rather than creating a duplicate.

Required: verdict (agree, disagree, or borderline); corrected_score (1–5) is required when verdict is disagree Optional: corrected_score (for agree/borderline), note, created_by (defaults to "api")

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1/responses/42/metrics/3/agreements \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"verdict": "disagree", "corrected_score": 3, "note": "too generous", "created_by": "alice"}'

GET /api/v1/runs/:run_id/responses/:response_id/metrics/:metric_id/agreements

List agreements scoped to a specific run, response, and metric — the nested equivalent of the top-level index filtered by those three ids.

DELETE /api/v1/agreements/:id

Delete an agreement. Returns 204 No Content.

Provider Credentials

LLM provider API keys. The api_key field is write-only and never returned in responses.

GET /api/v1/provider_credentials

List all provider credentials (api_key excluded).

POST /api/v1/provider_credentials

Create a provider credential. Each provider may have only one credential; creating a second for an already-configured provider returns 422.

Required: provider (openai, anthropic, ollama, openrouter, azure_foundry) Optional: api_key (not enforced by the API, but required in practice for hosted providers; local providers like ollama can omit it), api_endpoint (required when provider is azure_foundry), api_version

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/provider_credentials \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"provider": "openai", "api_key": "sk-..."}'

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/provider_credentials \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"provider": "azure_foundry", "api_key": "...", "api_endpoint": "https://my-resource.openai.azure.com", "api_version": "2024-10-21"}'

GET PATCH DELETE /api/v1/provider_credentials/:id

Get, update, or delete a provider credential. DELETE returns 422 (not 204) when the provider is still in use by prompts or judge runs; remove those references first.

curl -X PATCH https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/provider_credentials/1 \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"api_key": "sk-new..."}'

Imports

Bring an existing promptfoo config into CompletionKit in one call. Prompts, the test dataset, and assert-based metrics are created where they map cleanly and skipped with a reason where they don't. Providers are matched against your already-configured credentials; any referenced in the config but not yet configured are reported so you can add them.

POST /api/v1/imports/promptfoo

Import a promptfooconfig.yaml. Send the YAML as a config param, or POST the raw YAML as the request body. Returns 201 with a mapping summary, or 422 if the YAML cannot be parsed.

Request: config (the YAML text) or a raw YAML request body

Response 201: prompts and metrics each list what was created and what was skipped (with a reason); dataset is {created, rows, columns} (or {skipped}); providers is {matched, unmatched} (provider strings from the config)

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/imports/promptfoo \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/x-yaml" \
  --data-binary @promptfooconfig.yaml

Plan quotas

Each plan includes a monthly allowance of runs and prompt fetches. Usage counts per billing period and resets when the period rolls over. The same allowance applies whether you go through the app, the REST API, or the MCP server. Deleting a run does not return quota: the period counter only moves up, so a deleted-then-recreated run still counts twice.

Pre-check your usage

Both surfaces expose the same read so you can see what's left before starting work: GET /api/v1/usage over REST, or the usage_get MCP tool. Either returns both counters for the current period. Unlimited plans report limit and remaining as null.

GET /api/v1/usage

Read-only. The MCP usage_get tool returns this same body.

{
  "runs":           { "used": 42, "limit": 100, "remaining": 58, "percent_used": 42, "over_limit": false, "blocked": false },
  "prompt_fetches": { "used": 10, "limit": 1000, "remaining": 990, "percent_used": 1, "over_limit": false, "blocked": false },
  "period":         { "start": "2026-07-01T00:00:00Z", "end": "2026-07-31T23:59:59Z", "resets_at": "2026-08-01T00:00:00Z" }
}

Over the limit

Runs are hard-blocked once the run limit is reached (a small grace band is allowed first), surfaced with structured fields on both surfaces. Prompt fetches are never blocked for being over quota: your app depends on them at runtime, so once over the limit they keep serving and only warn (owner email plus a heads-up to your team). We count them so you can see the overage, but we won't brick a live app.

REST 402 Payment Required

POST /api/v1/runs returns 402 once the run allowance is spent. (Prompt fetches never return 402 for being over quota.)

{
  "error": "quota_exceeded",
  "scope": "runs",
  "message": "Run limit reached for this billing period.",
  "used": 100, "limit": 100,
  "resets_at": "2026-08-01T00:00:00Z",
  "upgrade_url": "https://completionkit.com/pricing"
}

MCP runs_create error

The tool result is an error (isError: true) whose structuredContent marks it a hard quota. retryable: false means retrying will not help and will only spend more quota, so an agent should stop and surface the upgrade link rather than loop.

{
  "error": "quota_exceeded",
  "code": "run_limit_reached",
  "scope": "runs",
  "retryable": false,
  "used": 100, "limit": 100,
  "resets_at": "2026-08-01T00:00:00Z",
  "upgrade_url": "https://completionkit.com/pricing"
}

Approaching the limit

Once usage crosses 80% of the run limit, a successful runs_create still succeeds but carries a non-fatal usage_warning in its structuredContent, so an agent can slow down ahead of the block instead of discovering it by hitting it.

{
  "usage_warning": {
    "code": "run_limit_approaching",
    "used": 85, "limit": 100, "remaining": 15,
    "resets_at": "2026-08-01T00:00:00Z"
  }
}

API reference