CompletionKit CompletionKit

API reference

REST JSON API and MCP server. Authenticate with a bearer key in the Authorization header — create one under Settings → API once you've made an organization. Swap your-org-slug-goes-here and YOUR_TOKEN below for your own.

MCP Server

Connect Claude Code, Cursor, or any MCP client to manage prompts, runs, datasets, and metrics conversationally. 35 tools over streamable HTTP.

▶ Claude Code 
claude mcp add completion-kit \
  --transport http \
  --url https://completionkit.com/orgs/your-org-slug-goes-here/mcp \
  --header "Authorization: Bearer YOUR_TOKEN"
{} Cursor / Generic 
{
  "mcpServers": {
    "completion-kit": {
      "url": "https://completionkit.com/orgs/your-org-slug-goes-here/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_TOKEN"
      }
    }
  }
}

Available tools

prompts 6

prompts_list List all prompts
prompts_get Get a prompt by ID
prompts_create Create a prompt
prompts_update Update a prompt
prompts_delete Delete a prompt
prompts_publish Publish a prompt version, making it the current version

runs 6

runs_list List all runs
runs_get Get a run by ID
runs_create Create a run
runs_update Update a run
runs_delete Delete a run
runs_generate Generate responses for a run using its prompt and dataset

responses 2

responses_list List responses for a run
responses_get Get a specific response

datasets 5

datasets_list List all datasets
datasets_get Get a dataset by ID
datasets_create Create a dataset with CSV data
datasets_update Update a dataset
datasets_delete Delete a dataset

metrics 5

metrics_list List all metrics
metrics_get Get a metric by ID
metrics_create Create a metric with evaluation criteria
metrics_update Update a metric
metrics_delete Delete a metric

metric 5

metric_groups_list List all metric groups
metric_groups_get Get a metric group by ID
metric_groups_create Create a metric group
metric_groups_update Update a metric group
metric_groups_delete Delete a metric group

provider 5

provider_credentials_list List all provider credentials (API keys are not exposed)
provider_credentials_get Get a provider credential by ID (API key is not exposed)
provider_credentials_create Create a provider credential
provider_credentials_update Update a provider credential
provider_credentials_delete Delete a provider credential

tags 5

tags_list List all tags
tags_get Get a tag by ID
tags_create Create a tag. Color is auto-assigned.
tags_update Rename a tag.
tags_delete Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.

Prompts

Create, version, and manage LLM prompt templates.

GET /api/v1/prompts

List all prompts, ordered by most recent.

curl https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/prompts \
  -H "Authorization: Bearer YOUR_TOKEN"

POST /api/v1/prompts

Create a new prompt.

Required: name, template, llm_model Optional: description

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/prompts \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "summarizer", "template": "Summarize: {{text}}", "llm_model": "gpt-4.1"}'

GET /api/v1/prompts/:id

Get a single prompt by ID.

PATCH /api/v1/prompts/:id

Update a prompt. Accepts same params as create.

DELETE /api/v1/prompts/:id

Delete a prompt. Returns 204 No Content.

POST /api/v1/prompts/:id/publish

Publish a prompt version, making it the current version in its family.

Runs

Create runs, generate LLM responses, and judge them with metrics.

GET /api/v1/runs

List all runs with response counts and average scores.

POST /api/v1/runs

Create a new run.

Required: prompt_id Optional: name, dataset_id, metric_ids, judge_model

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"prompt_id": 1, "dataset_id": 1, "metric_ids": [1, 2]}'

GET /api/v1/runs/:id

Get a run with status, progress, response count, and average score.

curl https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1 \
  -H "Authorization: Bearer YOUR_TOKEN"

POST /api/v1/runs/:id/generate

Start generating responses. Returns 202 Accepted. Poll the run to check progress.

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1/generate \
  -H "Authorization: Bearer YOUR_TOKEN"

PATCH /api/v1/runs/:id

Update a run. Accepts same params as create.

DELETE /api/v1/runs/:id

Delete a run and all its responses. Returns 204 No Content.

Responses

Read-only access to generated responses and their review scores. Nested under runs.

GET /api/v1/runs/:run_id/responses

List all responses for a run, including nested review scores.

curl https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/runs/1/responses \
  -H "Authorization: Bearer YOUR_TOKEN"

GET /api/v1/runs/:run_id/responses/:id

Get a single response with its review scores and feedback.

Datasets

Data used as input for runs.

GET /api/v1/datasets

List all datasets.

POST /api/v1/datasets

Create a dataset.

Required: name, csv_data

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/datasets \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "tickets", "csv_data": "text,expected_output\\nHello,Hi"}'

GET PATCH DELETE /api/v1/datasets/:id

Get, update, or delete a dataset.

Metrics

Scoring dimensions used by the judge model.

GET /api/v1/metrics

List all metrics.

POST /api/v1/metrics

Create a metric.

Required: name Optional: instruction, rubric_bands (array of {stars, description})

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/metrics \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "relevance", "instruction": "Is the response relevant?"}'

GET PATCH DELETE /api/v1/metrics/:id

Get, update, or delete a metric.

Metric Groups

Named groups of metrics you can apply to a run as a set.

GET /api/v1/metric_groups

List all metric groups with their metric IDs.

POST /api/v1/metric_groups

Create a metric group.

Required: name Optional: description, metric_ids (array)

GET PATCH DELETE /api/v1/metric_groups/:id

Get, update, or delete a metric group. PATCH with metric_ids replaces all metric associations.

Tags

Domain labels you can attach to metrics, prompts, runs, and datasets. Tags are auto-assigned a color from a 10-color palette. Each index page can be filtered by one or more tags using ?tag[]=name query params (OR semantics).

GET /api/v1/tags

List all tags with name and color.

curl https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/tags \
  -H "Authorization: Bearer YOUR_TOKEN"

POST /api/v1/tags

Create a tag.

Required: name

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/tags \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "real estate"}'

GET PATCH DELETE /api/v1/tags/:id

Get, update, or delete a tag. PATCH accepts name. DELETE returns 204 No Content and removes all taggings for this tag.

Tagging resources

Metrics, prompts, runs, and datasets accept a tag_names array on their create and update endpoints. Passing a name that does not yet exist silently creates the tag. On PATCH, the list replaces all existing tags for that record (omit the field to leave tags unchanged).

curl -X POST https://completionkit.com/orgs/your-org-slug-goes-here/api/v1/metrics \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "Accuracy", "tag_names": ["real estate"]}'

MCP tools

tags_listList all tags
tags_getGet a tag by ID
tags_createCreate a tag (name required)
tags_updateUpdate a tag's name
tags_deleteDelete a tag and remove all its taggings

The existing metrics_create, metrics_update, prompts_create, prompts_update, runs_create, runs_update, datasets_create, and datasets_update tools all accept a tag_names parameter with the same auto-create and replace semantics as the REST API.

Provider Credentials

LLM provider API keys. The api_key field is write-only and never returned in responses.

GET /api/v1/provider_credentials

List all provider credentials (api_key excluded).

POST /api/v1/provider_credentials

Create a provider credential.

Required: provider (openai, anthropic, ollama, openrouter), api_key Optional: api_endpoint

GET PATCH DELETE /api/v1/provider_credentials/:id

Get, update, or delete a provider credential.