Changelog

Notable changes to CompletionKit Cloud. The self-hosted engine has its own changelog in the completion-kit repository.

2026-08

Engine 0.28.40–0.28.42: the retry button says what it will retry, and the dashboard cards read straight. The button on a run header read "Retry N failed cases" whatever had gone wrong, which implied regeneration even when the response was fine and only its judge had timed out; it now reads "Retry scoring on 1 response" when only reviews failed, "Retry 1 failed response" when only generation did, and the combined count when both did. It also drops its bespoke red styling for the same secondary outline as Edit and Compare, so red belongs to the parts of the panel that report state rather than to the action that clears them. On the dashboard, the Checks and Activity cards no longer draw fourteen dead bars for a window with nothing in it (an empty card is not a broken chart, so the sparkline is dropped entirely and the headline and footnote remain), and the four stat cards now line their values up with each other instead of each floating at a different height.
Engine 0.28.39: retrying a run's failed cases now covers judge failures, and a row whose scoring failed no longer claims it's Done. A response can generate perfectly well and still have its review blow up, which the run header counts among the failed cases. "Retry failed cases" only ever looked at responses whose own status was failed, so when the failure was a judge it matched nothing, enqueued nothing, and still flipped the run back to running, leaving it stuck there with no way back to completed. Retry now re-queues each failed review on a succeeded response against its own metric, judge or check, and leaves the run's status alone when there is genuinely nothing to retry. The row chip agrees with the header too: a response whose review failed shows Failed once the run has finished, instead of reading Done.
Engine 0.28.33–0.28.38: judge temperature is now honest, plus a dashboard Checks-card fix. When a reasoning judge refused temperature=0, the request was quietly re-sent without it and the provider applied its own (~1.0) default, yet the run still reported judge_temperature: 0 and claimed its scores were reproducible, most confident exactly when it was least deterministic. Runs now detect the refusal (across every provider and the phrasings in use), record it in a judge_temperature_ignored flag, and the run page and MCP payload distinguish a temperature you set high from one the model refused. A run can also now send no temperature at all ("leave it to the model"), which is the only request some current models accept. On the dashboard, the Checks card no longer lists the same failing metric twice (it groups by metric with a count), and a fresh workspace's seed now demonstrates the deterministic check metric type.
Going over your prompt-fetch quota never takes your production app down. Your app fetches its prompt from CompletionKit at runtime, so a hard block would break it. Over the limit, prompt fetches keep serving (metered, so the overage is visible) instead of returning a 402, and serving a specific prompt now counts, not just listing. A billing lapse that just needs a card update keeps serving too; only a full cancellation stops it. Runs are unchanged (still capped over quota, since evals are discretionary). Owners now also get an 80%-of-limit heads-up on both runs and prompt fetches.
Engine 0.28.31: see how often each prompt is served in production. CompletionKit already serves your published prompts over the API; now it counts those fetches and surfaces them, so you can tell which prompts are actually live and how heavily they're used. The prompt page shows a served count over the last 7 and 30 days, and the dashboard highlights the most-served prompts (#158).
Engine 0.28.30: starting a run over MCP no longer times out. runs_generate used to hold the request open while it created and enqueued every row, which surfaced as a timeout on large runs even though the run had already started. It now returns the moment the run is claimed and does the work asynchronously, so the call comes back immediately (#150).
Engine 0.28.32: clearer dashboard cards and a fixed run banner. The failing-checks card is hidden on workspaces with no check metrics (and reads as a pass-rate trend where checks do exist), the failures card names the run that failed instead of a truncated judge name, the trend-card headers no longer wrap, and the stale-metrics banner on a run now recommends Re-grade (the cheap, correct action) rather than Re-run, with each button's cost spelled out inline (#166).
Engine 0.28.25–0.28.29: judging and MCP controls. Judges now score at temperature 0 by default, separate from the generation temperature (#155); runs accept a generation max_tokens and a settable temperature over MCP (#159); runs_get returns per-metric averages and responses_list gains field selection, score filters and paging (#160); a run shows how often a human agreed with its judge and says "Unverified" when nobody has checked (#154); a run whose scores cluster at the top now warns instead of reading as a clean success (#153); an LLM-judge metric can no longer be saved with a blank rubric (#156); and a run where every response failed is marked failed instead of showing a blank score (#157).

2026-07

Engine 0.28.23: editing a prompt's name, description, or tags no longer creates a new version. A prompt with runs used to version on every save, so renaming it or changing its tags (metadata that doesn't affect how the prompt renders or scores) minted a new version and cluttered the history. It now matches how metrics already behave: a new version is created only when a behavioral attribute changes (the template or the model); name, description, and tags update in place on the current version.
Agents get a clear signal when you hit your plan's run limit, and can check usage before starting. A new usage read (GET /api/v1/usage over REST, or the usage_get MCP tool) reports how many runs and prompt fetches you've used this billing period, their limits, and what's left, so you or a coding agent can pre-check before kicking off work. When you're over the run limit, runs_create now returns a machine-readable error (run_limit_reached, retryable: false) instead of looking like a transient failure, so an agent stops and surfaces the upgrade path rather than retrying into more quota; a successful run past 80% of the limit carries a heads-up warning. The same quota surfaces as a 402 on the REST API. See the API reference for the exact shapes.
Engine 0.28.22: starting a run over MCP is less likely to time out. Kicking off a run enqueued one background job per row (generation) or per row × metric (judging), each its own database insert, so a 35-row judged run made ~140 individual enqueues while the request stayed open; over MCP that synchronous cost is what surfaced as a runs_generate timeout even though the run had already started. The jobs now enqueue in a single bulk call, roughly halving both the enqueue time and the database writes (partial #150; the durable fix, returning immediately and enqueuing the run asynchronously, is still to come).
Engine 0.28.20–0.28.21: MCP run tools fixed, and the runs list no longer times out. Over MCP, runs_create/runs_update now accept metric_group_id (it was silently dropped, producing a run that judged nothing yet reported success), and any run with no metrics attached now carries a clear warning. New MCP verbs runs_regrade, runs_rerun, and runs_retry_failures mean re-judging an already-generated run no longer requires deleting and recreating it (#147, #148). And the runs index, which loaded every response and review into memory to compute per-run scores, now uses a handful of grouped queries, fixing the gateway timeouts on orgs with many runs (#151).
Engine 0.28.18–0.28.19: starting a run over a large dataset no longer risks stalling the web server. A run's responses were inserted in one giant SQL statement (several megabytes on a 20,000-row dataset), which could spin the CPU in per-query SQL processing long enough to block the single web worker and trip a health-check restart. The insert is now chunked into 1,000-row batches, keeping each statement small regardless of dataset size (#149). Also a small styling fix: the suggestion Version column now uses the standard version chip.
Engine 0.28.16–0.28.17: the suggestion result panel is rewritten to be legible to a first-time user. It now leads with a plain verdict ("We re-ran this wording on 12 of the run's responses. It scored 4.1 now, 3.3 before" with a Better/Same/Worse tally, or "We couldn't test this rewrite" with a short explanation of why) instead of internal jargon (held-out, re-scored, validated, net negative). The "Apply anyway" button carries its warning inline above it rather than only in the confirm dialog (#146), the redundant full-prompt dump is gone (the diff already shows the new prompt), and applying is spelled out plainly (it publishes the wording as a new version and leaves the current one intact). The prompt page's suggestions list is retitled "Suggested improvements," renders reasoning as clean text instead of raw ** and backticks, adds a Version column, and eager-loads to avoid per-row queries.
Engine 0.28.14–0.28.15: the prompt page loads fast again, plus a prompts-list layout fix. Opening a prompt fired ~47 database queries (about 9.5 seconds) even with only a handful of runs, because per-run scores were recomputed in several places and the eager-loaded data was thrown away and re-queried each time; it now reuses the loaded data and groups the work into a few queries regardless of how many runs and versions the prompt has (#145). Also fixed a long model ID (like a dated model snapshot) overlapping the score badge in the prompts list (#144).
Engine 0.28.12–0.28.13: the monospace font loads again, uploads hardened, lighter dependencies. JetBrains Mono was 404ing on every authenticated page (its asset paths didn't resolve under the Propshaft pipeline), so the UI fell back to the system monospace; the paths are fixed and it loads (#143). File uploads (the promptfoo import and dataset upload) are now size-capped before they're read, and the YAML import's alias-expansion vector is closed, hardening both against memory-exhaustion (no code-execution risk ever existed) (#142). The engine also went Propshaft-native and dropped unused sassc/bootstrap/jquery, which removed Sprockets from this app's bundle entirely.
Engine 0.28.11: run pages open quickly on large runs. Opening a run recomputed its summary panel (average score, per-metric scores, pass rates) by loading every response's reviews one query at a time, so a run with a couple thousand responses took ~19 seconds and ~2,000 queries to open, no matter how short each response was; those summaries now load in a single query (#135). The page also parsed the entire dataset CSV on every load to show a 50-row preview, so a 20,000-row dataset cost hundreds of milliseconds of rendering; it now reads just the rows it shows (#136). With the earlier responses-table work (0.28.9/0.28.10) and a 25-a-page default, a big run that opened in ~19 seconds now opens in well under one.
Engine 0.28.9: big runs load faster, and cleaner pagination. The run page was pulling every response's full body just to show a 600-character preview; on runs with long outputs (16 KB and up per response) that meant fetching well over a megabyte per page to display a fraction of it. It now loads a short database-side preview instead, so the responses table loads far quicker on large runs, and the response pagination control is restyled (centered, with a divider).
Engine 0.28.8: accessibility pass over the product (WCAG 2.1 AA, part 1). An axe-core sweep and manual audit across the signed-in surfaces (prompts, datasets, metrics, runs, tags, providers) fixed tag and chip contrast, added keyboard focus rings, gave metric pips, status dots, and icon-only controls proper screen-reader names, gave every page a distinct title, and labelled dialogs and form errors. Part 2 (spoken run and discovery progress) is still to come.
Engine 0.28.7: MCP tool discovery works without a token. Registries and clients can now list the MCP server's toolbox (initialize, tools/list) without credentials, so a directory shows what the server does instead of an auth wall. Anything that reads or writes your org data (tools/call) still requires the token and stays scoped to your organization. The hosted /mcp endpoint serves the discovery methods unauthenticated and keeps data calls gated.
Engine 0.28.6: the MCP endpoint answers a GET with 405, not 404. Registries and health checkers (such as Glama) probe the MCP endpoint with a GET; it now returns a proper 405 Method Not Allowed (with an Allow header, no token required) instead of a 404, so the hosted MCP server reads as present and healthy rather than missing. The shared /mcp endpoint answers the same way.
Engine 0.28.5: provider API keys stay out of error diagnostics. When model discovery failed against a bad key, the provider's response (which for some providers echoes the key you submitted) could reach error tracking and the stored error detail. Provider error messages are now redacted (the submitted key, sk-… keys, and bearer tokens), and an expected bad-key failure is treated as a handled result rather than reported as an application error. A malformed run-page URL (?page[]=) now falls back to the first page instead of erroring.
Invitations check the email address before sending. Inviting a teammate now validates that the address is well-formed, so a typo shows an error on the invite form instead of silently failing to deliver.
Engine 0.28.0: large runs no longer time out or bog down the run page. Starting a run over a big dataset now creates all its rows in one batched insert instead of one at a time, so a two-thousand-row run kicks off without hanging the request, and the run page shows responses 100 at a time with pagination instead of rendering every row at once.
Engine 0.27.6–0.27.9: fuller, more accurate API reference. The REST API reference now has copy-paste curl examples on every create, update, and action endpoint (the provider-credentials POST included, with an Azure AI Foundry variant), and its parameter lists are corrected: the provider set lists Azure AI Foundry, api_version is documented, and required-versus-optional labels match what the API actually accepts.
Engine 0.27.5: fix a false "metric versions no longer live" warning. Judging a brand-new metric's first run could, under concurrency, create duplicate copies of its version, which made the run wrongly warn that its scores were against out-of-date versions. Version creation is now race-safe and the database enforces one row per version number, so the duplicate (and the false warning) can't recur.
Engine 0.27.4: a provider error no longer sinks the whole run. When a model provider returns an error mid-run (insufficient credits, a rate limit, a bad request), that row or judge review is now marked Failed with the provider's message and the run finishes, instead of the job erroring out and retrying. Failed judge reviews get a Failed badge.
Engine 0.27.3: deleting a metric used by a run no longer errors. Removing a metric that a past run referenced could fail with a server error; it now deletes cleanly, and the run keeps the rest of its history.
Engine 0.27.2: clearer errors instead of server failures on a few edge cases. Editing a prompt to use a model that can't generate, viewing a failed response that has no provider detail, entering a tag name that isn't allowed, and importing a promptfoo config with a validation problem now each show a clear message instead of a server error.
Engine 0.27.0: Azure Foundry catalog count, and a guard against non-generating models. Azure AI Foundry providers now show how many models are available to deploy in your catalog, with a refresh button on the models list. And choosing a model that can't generate responses (for example a judge-only model) as a prompt's model is now rejected up front with a clear error, instead of failing when you run it.
Engine 0.26.8: see why a response failed. A failed response in a run is now clickable, and its page shows the provider error that caused it (which provider, the HTTP status, and the message) instead of an empty response body. Quicker to tell a bad key from a rate limit from a malformed prompt.
Engine 0.26.7: discover a provider's models on demand. A provider with no discovered models yet now shows a "Discover models" button on its page, so you can start or retry model discovery without re-saving the credential.
Engine 0.26.6: broader Azure AI Foundry support. Azure AI Foundry project endpoints (the /api/projects/ URLs) now work for both model discovery and running prompts, and models that require max_completion_tokens instead of max_tokens (some newer Azure and reasoning models) are handled automatically.
Engine 0.26.5: model discovery survives duplicate model IDs. When a provider's model list returned the same model more than once, discovery could error out before finishing. It now de-duplicates the list first, so discovery completes cleanly for those providers.
Engine 0.26.4: in-app confirmation dialog and a tidier provider form. Deleting a dataset, prompt, metric, metric group, tag, provider, or run now asks you to confirm in a styled in-app dialog instead of the browser's plain popup, with the destructive action shown in red. A provider that can't be deleted yet, like one still used by a prompt or run, shows its delete control disabled with the reason beside it. The provider form also reveals the Azure API-version field only when you pick Azure AI Foundry, and that field is now optional: leave it blank to use Azure's v1 API.
Engine 0.26.1: add Azure AI Foundry as a provider, delete unused providers, and a fuller model list. You can now connect an Azure AI Foundry (Azure OpenAI) endpoint as a first-class provider, with its own API-version field, instead of forcing it through the generic OpenAI-compatible option where model discovery failed. Provider credentials can also be removed now: delete one that nothing references from its edit page, with a guard that refuses while a prompt or run still uses one of its models and cleans up that provider's discovered models on delete. Two discovery fixes ride along: every OpenRouter model now resolves to a definite judge state instead of showing "?" depending on when its catalog was first discovered, and a failed lookup on a custom endpoint now says what to check (and points Azure endpoints at the Foundry provider) instead of a raw "ollama ... 404".
Undo an accidental upgrade in the same billing cycle. If you upgrade a paid plan and switch back before the next invoice, the change now reverts right away and drops the not-yet-billed upgrade charge, instead of keeping you on the higher plan (and its pending cost) until the period end. Switching down after you've actually been billed for the higher plan still waits for the period you paid for.
Engine 0.25.3: broken links on record pages fixed. Lists shown on a record's own page (like the runs on a dataset page) were building malformed URLs such as /datasets/4.4 that 404'd. Those links now resolve correctly, and the dataset CSV-download link is fixed the same way.
Engine 0.25.0: choose your answer-key column, plus two fixes. Grading an output against each row's expected value no longer requires naming the column exactly expected_output: pick any column as the answer key on the dataset or run, and the forms now surface it. Two fixes rode along: a provider's usage ("last used", judge count) no longer reads "Never used" once its runs age past your retention window, and a score-existing-outputs run no longer hangs on a blank cell in the graded column, it fails that row cleanly instead.
Engine 0.22.0: check a field against each row's expected answer. Deterministic checks can now compare an output field against a per-row expected value (an answer-key column), not only a fixed value you type. Switch a contains, equals, or does-not-contain check to "compare against each row's expected value", optionally point it at a field inside the answer key, and every row is graded against its own ground truth, with no LLM judge and no external script. Built for extraction and classification, where each input has its own correct answer.
Change plans in-app, and downgrades wait for the period you paid for. Upgrading, downgrading, and cancelling no longer bounce you to Stripe. An upgrade applies right away (prorated). A downgrade to a cheaper paid plan now takes effect at the end of the period you already paid for, so you keep the higher plan's limits until then instead of being cut off mid-cycle, and you can cancel a pending downgrade with one click. Downgrading to Free cancels at period end. Downgrades settle as account credit, never a surprise second charge.
Engine 0.21.0: a clearer check-metric editor. Setting up a deterministic check now uses plain-language labels ("Contains a phrase", "Is valid JSON", "A JSON field equals a value") in place of the internal keys, with styled dropdowns that match the rest of the app. The checks run exactly as before; only the editor is easier to read.

2026-06

Engine 0.20.2: deterministic check metrics, promptfoo import, and retention polish. Metrics can now be deterministic pass/fail checks (valid JSON, no refusal, contains a token, regex, length, JSON path) that run with no LLM call and so are unmetered, alongside the 1-5 judge. Run comparisons and the dashboard are check-aware, including a failing-checks card. You can import a promptfooconfig.yaml into prompts, datasets, and metrics from the Metrics page. Runs past your plan's retention window are now hidden consistently across the dashboard and lists, with a clear "older runs are hidden" state instead of "no runs yet" once everything has aged out.
Subscription emails. Every verified org owner now gets an email when a paid plan starts, when the plan changes, and when a subscription is cancelled, alongside the existing internal Slack alerts. Previously a plan change produced no customer-facing confirmation at all.
Signup is no longer silently dropped by the bot filter. The invisible-captcha timing and spinner checks were rejecting legitimate signups, anything submitted within about 2 seconds of the page loading, and sending them to a "check your email" page while creating no account and no verification email. Those two gates are removed; the hidden honeypot field stays, and a tripped honeypot is now logged and reported to error tracking instead of failing silently.
Engine 0.16.1: starting a run over MCP no longer reports a false failure. On the hosted deployment, an MCP agent calling runs_generate could get an error back for a run that had in fact started, inviting a retry and a double-generate. The underlying route-warmup error is fixed and run-start UI broadcasts are now best-effort, so a cosmetic broadcast problem can never fail a started run.
Engine 0.16.0: validated prompt suggestions, a properly scoped judge, and no more OpenAI-by-default. A prompt-improvement suggestion now re-runs the candidate prompt against a held-out slice of the run's reviewed responses and shows a before/after scoreboard (improved, held, regressed) before you publish; a rewrite that scored net-negative asks for confirmation instead of applying in one click. The judge no longer reads the full prompt when scoring metrics outside its scope, so a prompt's banned-word list can't move an unrelated correctness score (output-quality scores may shift as that bias disappears). And the engine stopped assuming OpenAI: each organization's default judge model is now resolved from its own discovered models, so an Anthropic-only org gets a Claude judge for metric suggestions instead of a failing call to a hardcoded gpt-4.1, and the onboarding sample prompt picks a model you actually have.
Engine 0.15.1: Anthropic model discovery no longer false-negatives on thinking-by-default models. The capability probe read the first content block of the Anthropic response, but recent Claude models think by default and return a thinking block first, so the probe saw empty text and wrongly flagged the model as unable to generate or judge. It now scans for the text block past any leading thinking block; a response with no text block at all is still correctly treated as a failed probe.
Engine 0.15.0: provider edit-page Refresh works again, and MCP prompts_update drafts instead of auto-publishing. Hitting Refresh on a provider's edit page now shows the spinner and live progress; the polling guard used to bail when the completed models card had no discovery-status element, which was exactly the state you click Refresh from. Separately, prompts_update over MCP, on a prompt that already has runs, now creates the new version as a draft and leaves promotion to prompts_publish, so an agent's edits get a review gate instead of going straight to current (a prompt with no runs is still updated in place). The MCP blog FAQ was corrected to match.
Engine 0.14.0: create datasets from a URL or an uploaded CSV file. The REST POST /api/v1/datasets endpoint now also accepts a multipart file upload, not just an inline csv_data string. A new MCP tool datasets_create_from_url downloads a CSV from a public http(s) URL on the server (SSRF-checked, capped at 10MB) so a large dataset never has to be inlined into the tool-call arguments. Both routes land on the same authenticated, tenant-scoped dataset create path.
Engine 0.13.0: MCP tool judges_suggest renamed to metrics_suggest_variants, plus a new prompts_suggest_improvement tool. The metric-variant suggestion tool moved into the metrics namespace to sit beside the REST metrics#suggest_variants. There is no alias kept, so any MCP client that calls it by a hardcoded name needs updating; clients that discover tools through tools/list pick up the new name automatically. The new prompts_suggest_improvement tool suggests an improved prompt template grounded in a run's test results and judge feedback (the same engine behind the web "Suggest improvements"), returning the reasoning and rewritten template and persisting a suggestion. It takes a run_id and rejects judge-only runs that have no prompt.
Engine 0.12.4: the MCP server runs statelessly, so coding agents stop getting disconnected. Previously every MCP call after the first had to carry a live session id, and once a session hit its idle timeout the next request was rejected with a "Session not initialized" error, forcing the client to reconnect mid-task. The server now serves tool calls without requiring a prior initialize or a valid session id, per the Streamable HTTP transport spec. initialize still issues a session id for clients that use one, and DELETE still ends a session.
Engine 0.12.0 to 0.12.3: provider model discovery updates live again. Adding a provider used to leave the Providers page stuck on "Looking up models…" with no progress, and the Refresh button on the prompt and run forms did nothing, because the engine's live-update calls were pinned to its standalone mount path and 404'd under Cloud's per-tenant /orgs/:slug mount. The engine now drives discovery progress from a polled status endpoint and renders the poll and refresh URLs from route helpers, so they follow whatever mount Cloud uses. The model capability probe also stopped over-asking for output tokens, which had wrongly flagged Anthropic models like Haiku 4.5 and the Opus line as unable to generate, and Refresh now re-checks models that previously failed the probe.
Email verification takes a click now. Following the link in the verification email lands on a confirmation page with a button instead of verifying the instant the link is opened. Corporate mail-security scanners (Microsoft Safe Links, Proofpoint, Mimecast and the like) auto-fetch every link in inbound mail, which was silently marking addresses verified with no human involved. The state change now happens on the button's POST, which scanners don't submit, so a verified address means a person actually clicked through.
More member seats; seats are no longer an upgrade lever. Free now includes 3 members (was 1), Starter 10 (was 3), and Team is unlimited (was 10). The tiers differ on runs, prompt-fetches, history retention, SSO and audit log, and support, not on seats.
Engine 0.9.0 to 0.12.0: wider MCP surface, judge-suggest over MCP, calibration table renamed. The engine's MCP server now exposes the full toolset (datasets, metrics, prompts, responses, provider credentials, metric groups, tags) plus judges_suggest, which rewrites the judge instruction into draft metric-version variants targeted at recent disagreements, so a coding agent can drive the whole evaluate-improve loop, including drafting the next revision, over MCP. Reviews now require a metric_version. The completion_kit_calibrations table was renamed to completion_kit_agreements; the rename carried cloud's organization_id, foreign key, and RLS tenant_isolation policy across intact, and the seed sets each review's version via MetricVersion.ensure_current_for.

2026-05

Engine 0.9.0: calibration is measure + rewrite only; few-shot pinning removed. The engine drops the few_shot_examples column from completion_kit_metrics and removes the add_few_shot / remove_few_shot routes from both web and REST API namespaces. The calibration loop is now: review verdicts, watch the trust panel (Wilson interval + kappa), use the auto-suggest improvement to draft a rubric rewrite. Cases-to-learn-from feeds the rewrite, not a pinned-examples list. Plus a broad copy / UX polish pass on the metric show page (Versions sit above Calibration, suggestion banner moves to the edit form with icon buttons, publish modal clarifies draft-until-publish). Cloud-side scrubbed all marketing references to few-shot pinning from /docs Trust + Concepts, /glossary, and the why-we-built FAQ.
Engine 0.8.0: canonical REST API error shape + starter-pack page fix. Every engine REST API error response now uses one shape: a top-level error string, plus a details field map on 422 validation failures. Replaces three prior shapes ({error: "..."} for auth/business rules, {errors: ActiveModel::Errors} for validation, {errors: ["..."]} for run start/rerun failures). Cloud's existing Billing::QuotaResponse and EngineApiAuthentication already used the canonical shape, so no cloud-side changes were needed. Also fixes the /orgs/:slug/metrics zero-state UI (the engine added a real .ck-title--sm rule and switched .ck-starter-grid to five-across so the empty-state heading stops outshouting the page H1 and the fifth starter card no longer orphans).
Mailer: discard permanent SMTP failures. A bot signup with an SMTP-rejecting recipient hit Net::SMTPSyntaxError inside ActionMailer::MailDeliveryJob; default Solid Queue retries kept firing Honeybadger. Subclassed MailDeliveryJob to discard on Net::SMTPSyntaxError and Net::SMTPFatalError (any 5xx permanent failure), log the bouncing recipient address, and send exactly one Honeybadger event with the recipient in context.
Engine 0.5.44 → 0.7.0: metric versioning, regrade-only runs, REST/MCP parity. The internal "judge version" concept is now consistently named "metric version" everywhere (model class, table completion_kit_judge_versions → completion_kit_metric_versions, FKs, indexes), and 0.6.0 dropped the backward-compat aliases. Reviews now carry the metric_version_id they were judged under, so the response detail page and run show page surface which review rows are stale (judged by an older version) and a "Re-run with current judge" button on the run. The improvement loop and retry_failures flow refuse to act on stale reviews. A new regrade-only run re-judges a run's existing responses against the current metric version without re-generating any model output. Reverting to an older metric version now writes a revert audit row. Cross-run comparison view shows side-by-side per-case score deltas with version chips. The REST API gains pagination, tag filter, and resource-specific filters across every index endpoint, a flat /api/v1/calibrations with filters + destroy, and MCP tools for metric-version management. Turbo broadcasts moved into Response and Review after_save_commit callbacks; cloud added ResponseBroadcastSafety / ReviewBroadcastSafety mixins (matching the existing ProviderCredentialBroadcastSafety pattern) so broadcasts swallow URL-helper errors in non-request contexts like seeds and background jobs.
Signup bot defences. The signup form now ships with an invisible honeypot (random hidden field + form-load timestamp threshold) via the invisible_captcha gem. The Slack signup notification fires on email verification rather than on user create, so bots that haven't validated a real inbox no longer ping the team channel. A daily UnverifiedUserPruningJob deletes accounts that never verified within 7 days, taking their sessions with them via the existing dependent: :destroy chain. Honeypot tripping silently redirects to the same "check your email" success page, so spam-runners can't tell the form rejected them.
Engine 0.5.43: judge versioning surface + calibration scoped to the live version. Metrics now carry a proper versions table mirroring prompt versioning (per-metric version_number, published_at, source chips for "AI suggestion" vs hand edits). The trust panel, the disagreements list, and the auto-suggest improvement loop all scope to the metric's current judge version by default, so reverting or republishing the rubric no longer mixes signal across versions. The few-shot examples surface gets a real "forget this" action and the cards are now actually fed back into the judge prompt. Rubric edits show a per-band side-by-side diff (unchanged bands dimmed, changed bands word-diffed) on both the AI-suggested and hand-edited drafts. Response detail page surfaces what other operators on the org said about the row. Schema-only follow-up: the new version_number / published_at columns sit on the already-tenanted completion_kit_judge_versions table, no new tables to wire.
Engine 0.5.42: starter metrics pack. The Metrics index now leads with five preconfigured one-click cards (Correctness, Instruction following, Format compliance, Tone, Conciseness) when an org has no metrics yet, and shows whichever haven't been adopted as an "Add a starter metric" row when the index is non-empty. Each card opens a preview modal with the full 1-to-5 rubric the judge will use before anything gets created. Per-org "Don't show this one" dismissals persist. Cloud added cascade-deleting organization_id to the new completion_kit_starter_metric_dismissals table, reworked its uniqueness to be per-org, applied RLS, and shipped isolation specs. The day-seven activation drip email now points at the pack rather than abstract rubric advice.
Engine 0.5.40 – 0.5.41: calibration UX cleanup. The "Improve the metric" flow is now a single inline suggestion card on the metric page (no more /improvements page, no banner, no queue), gated on having at least one Disagree verdict so the model has signal to work from. Save feedback is the Save button itself flashing green for 1.4s. The disagreements section becomes "Cases to learn from" with one card per case (no overflowing table at medium widths) and the verdict button reads "Remember this" instead of "Teach the judge". Onboarding panel's redundant divider hairline is gone. Response JSON output gets a light syntax highlight and scrolls horizontally instead of wrapping. No new tables: purely UI and copy.
Topbar polish at medium widths: the authenticated topbar now collapses to the hamburger up to 1024px (was 640px) so Dashboard / Prompts / Datasets / Metrics / Runs / Settings / Upgrade / Avatar stop wrapping onto two or three rows on small laptops and split-screen windows. The brand wordmark hides at the same breakpoint so the puzzle logo alone marks the spot. Settings-menu dividers also softened to a near-invisible line so the dropdown reads as one continuous list rather than three boxed groups.
Engine 0.5.33 – 0.5.39: privacy/a11y audit follow-through and judge calibration. 0.5.33 self-hosts JetBrains Mono (no more Google Fonts call), tightens functional cookie flags, raises the --ck-dim contrast token, ships PII and data-flow privacy docs, and lands a broad accessibility pass on the engine's authenticated pages. 0.5.34 hardens provider-endpoint SSRF re-validation and adds keyboard-accessible index tables. 0.5.35 – 0.5.38 add judge calibration: per-row agree/disagree/borderline verdicts, a trust panel with Wilson interval and quadratic-weighted κ, a disagreements list that promotes rows into the judge's few-shot examples, and an auto-suggest button that runs the judge model against its own recent disagreements to draft rubric rewrites. 0.5.39 ships the engine stylesheet as plain CSS (no ERB) for Propshaft hosts, and tightens the disagree-pill flow so clicking it without a score reveals the slider instead of producing a validation error. Cloud added cascade-deleting organization_id FKs and RLS to the new completion_kit_judge_versions and completion_kit_calibrations tables.
Settings menu: cog icon and "Providers". The Settings trigger in the authenticated topbar is now the same cog-6-tooth heroicon the engine's standalone app uses, instead of the word "Settings". The Provider-credentials menu item and page title shorten to "Providers" to match. Less text in the header, same destinations.
30-day organization purge. A daily job hard-deletes soft-deleted organizations after 30 days, matching the DPA's commitment. Cloud-owned tables cascade via the existing dependent: :destroy chain; engine-owned tables cascade through the on_delete: :cascade FK constraints the cloud-side migrations set when wiring each engine table for tenancy.
Self-service account deletion. The account page now has a Delete account section that destroys the user, their sessions, email preferences, and any invitations they've sent. Solo-owned organizations with no other members are deleted along with them; organizations where the user is the sole owner with other members still attached are listed and the deletion is blocked until ownership is transferred. The privacy policy points at this control instead of asking users to email.
Cookie consent banner with Google Consent Mode v2: visitors now see an opt-in banner before any GA4 cookies are set. We default consent to denied on every page load, the GA4 script still loads (so it can pick up the granted signal on the same pageview) but analytics_storage only flips to granted after the visitor clicks Accept. The choice is persisted in a first-party ck_cookie_consent cookie. Do Not Track and Global Privacy Control still suppress GA4 entirely, banner included.
Privacy: cookie disclosure tightened. Privacy policy now lists every functional cookie we set (name, purpose, type, expiry) instead of claiming a single session cookie. The ck_run_history_notice_dismissed cookie's expiry drops from 5 years to 1 year. Session and functional cookie writes set secure: Rails.env.production?, same_site: :lax explicitly on every path.
Privacy: data export broadened. The GDPR export now includes email-subscription state and invitations sent/received, on top of the user, organizations, and sessions it already covered.
Privacy: idle sessions purged. A daily job deletes Session rows past the 30-day idle timeout so IP and User-Agent history doesn't accumulate forever.
Accessibility statement at /legal/accessibility, linked from the marketing footer alongside the other legal pages. States our WCAG 2.1 AA target, known limitations, and a contact route at support@completionkit.com.
Accessibility pass: every layout now declares lang="en" and exposes a Skip-to-content link plus a <main> landmark. Flash messages live in a persistent ARIA live region so screen readers reliably hear them after Turbo navigations. Dropdown menus and the feedback widget drop half-applied role="menu" / role="dialog" and rely on native <details> semantics. The pricing period toggle uses aria-current instead of fake tab roles. Form inputs have a visible focus ring. The docs and pricing pages lead with a proper <h1>. Logo images carry explicit width and height to prevent layout shift. The invitation form requires the email field, and the accept form drops placeholders that duplicated the label.
Editable organization slug: owners can now change an organization's slug from settings, behind a confirmation modal that spells out the consequences. The old URL keeps redirecting for 45 days; if it is still getting traffic after 15 days we email the owner a reminder before it stops working.
Workspace dashboard. Finishing or skipping onboarding now lands you on a real dashboard instead of the bare prompts list: workspace totals, a 14-day run-activity sparkline, your worst-scoring metric, failed reviews, and version-over-version prompt score changes. It's linked from the topbar. Onboarding steps also now carry a one-line definition of each concept.
Landing page for signed-in users: visiting the home page while signed in now shows the landing page with a Dashboard button instead of bouncing straight to your prompts.
Free-plan prompt-fetch limit: serving prompts through the API is now metered. Free includes 1,000 fetches a month, Starter 10,000, Team 100,000, Business unlimited. Building and evaluating a prompt stays free; running real production traffic through it is a paid use case.
Usage metering. Per-organization run counters with billing-anchored periods, soft warnings at 80%, hard blocks at 100%, and automatic reset each period.
Quota gating across all surfaces: the web UI, the REST API (402 Payment Required with a structured body), and the MCP server (isError tool result) all enforce run limits.
Usage notification emails: owners get a heads-up at 80%, a "runs blocked" email at 100%, and an "unblocked" note when the new period starts.
In-app feedback widget. A small panel on every page that emails support directly, with replies threaded back to you.
GA4 + Honeybadger Insights: the activation funnel is tracked in both, plus engine-side usage events in Insights.
Legal pages: Terms, Privacy Policy, DPA, responsible-disclosure policy. GDPR data export on the account page. /.well-known/security.txt.
Onboarding checklist: new organizations land on a getting-started dashboard with a four-step checklist, plus an optional sample dataset & prompt to poke around.
Disaster-recovery runbook: documented restore procedures and a quarterly drill cadence.
Drip email scaffold: a four-step activation series for newly-verified users, with one-click unsubscribe.
Robot avatars: profile avatars are now a set of distinctive robots; pick one in account settings or upload your own.
Branded error pages: 404 / 500 / 4xx pages now match the app's dark, terminal-flavoured look instead of Rails' defaults.
No trial period: upgrading to a paid plan now starts billing right away. The free tier is there for anyone who wants to try first.
Plans tidied up: API and MCP access on every tier (free included). Unlimited prompts and datasets on every tier. The per-plan member limit is now enforced. SSO and audit logs are labelled "coming soon" on Team rather than implied as shipped.
Run-history retention: Free keeps the last 30 days of run history, Starter the last 365; Team and Business are unlimited. Older runs are hidden, never deleted: upgrade to an unlimited-history plan and they reappear. A dismissable banner appears when runs are being hidden.
Engine 0.5.7: prompt version history with per-version diff modals; the prompts list shows each prompt's API endpoint with a copy button; new runs inherit the previous run's tags; model pickers and the provider-models table are grouped by OpenAI family / OpenRouter vendor; fixed a spurious "no worker running" banner; cleaner flash messages.
API page: one place under Settings → API: your bearer keys in a table (create with a named-and-validated form, copy, revoke), then the full REST + MCP reference with copy-paste examples wired to your live key. Replaces the separate "API tokens" page.
Engine 0.5.8 – 0.5.9: your prompts' endpoints now sit inside the Prompts section of the API reference; one font size for every code example there (curl + MCP install snippets); fixed the Metric Groups / Tags / Providers reference tabs rendering blank; renaming or retagging a run with results no longer forks a new version; fixed "now ago" relative timestamps.
Public API docs: the full REST + MCP reference is now at /docs/api, crawlable and linked from /docs, with generic YOUR_ORG/YOUR_TOKEN placeholders. The in-app version under Settings → API is the same reference, wired to your own keys.
Landing redesigned: new hero with an animated "anatomy of a run" visual, refreshed sections (Problem · How a run works · The unlock · Compared · Three ways), dual hero CTAs (free Cloud signup + GitHub with a live star count), footer follow-through.
Three ways to run it: Cloud (hosted), the bundled standalone Rails app (self-host), or the source-available Rails engine (embed in your app). Landing, docs, and pricing-page tagline all use the same framing now.
Pricing + Docs on-brand: both pages restyled to match the redesigned landing. Docs page rewritten with an Overview (what / problem / how / Next level: agents run the loop), a nine-term concepts glossary, and clearer "pick a place to start" footer.
Free tier: 20 runs / month (down from 50). Same product, tighter ceiling on the free side.
Judge-only runs: grade an existing column in your dataset without re-running a prompt. Drop in production logs, hand-curated examples, or outputs from another system; the LLM judge scores them against your metrics. (Engine 0.5.11.)
Engine 0.5.10 – 0.5.13: new puzzle-piece brand mark; MCP sessions survive Puma restarts and slide their expiry on activity (no more "session not initialized" mid-conversation); prose font sizes normalized across the API reference; prompts-show page tightened. The Response detail page now caps Input / Response / Expected at 28rem and scrolls inside the block (big LLM-as-judge inputs no longer dwarf the page), JSON payloads pretty-print automatically, run-table names truncate so the metadata columns stay clustered, and scrollbars finally pick up dark color-scheme on Safari.
Engine 0.5.14 – 0.5.15: judges on reasoning models no longer silently 1-star: reasoning models (GPT-5 Pro family, o-series) charge their reasoning tokens against the chat-completion max_tokens cap. With the previous 1000-token default, ~40% of large judge prompts truncated and an unlucky share returned empty content that the parser coerced to 1 (the worst score on the 1-5 scale). The default cap is now 8192, and both OpenAI and OpenRouter clients raise on truncated or empty completions instead of returning a blank. 0.5.15 closes the remaining gap: the parser now raises JudgeParseError instead of returning {score: 1, feedback: "Could not parse…"}, so parse failures surface as distinct failed reviews (excluded from averages) rather than silent 1-star floors. If you've been seeing inexplicably-low 1-star scores on reasoning-model judges, this is the fix: they should re-judge correctly on the next run.
Dashboard recent-runs table matches the rest of the app: same pips, name truncation, judge-only label, and metadata layout as runs/index, dataset/show, and prompt/show.
Favicon halo: a thin #64748b ring around the puzzle piece for better legibility against pale browser chrome.
Engine 0.5.16: every results table (runs, prompts, datasets, metrics, tags, responses, prompt versions, suggestions, provider models) now uses table-layout: fixed with explicit column widths so long names can't push the table past the section's right edge. Tag pills also got a brightness bump so multi-colored tag rows don't read as "some are washed out."
Engine 0.5.17: the run-detail dataset row count is correct again, and long metric names wrap instead of overflowing.
Engine 0.5.18 – 0.5.22: the dashboard now surfaces what's actually happening: recent activity, your worst-scoring metric, failed reviews, and recent prompt changes. Alert cards can be dismissed, and quietly reappear if the score moves again. And the whole app is now responsive: a hamburger nav, horizontally-scrollable tables, and stacked headers on small screens.
Engine 0.5.23 – 0.5.26: fixed a worker thread race that could disrupt live updates; the dashboard's ignored-metrics flyout is now sorted worst-score-first with a tidier scrollbar; and a second mobile pass tightened the engine's own pages at phone widths: no more horizontal overflow or clipped text.
Engine 0.5.27 – 0.5.29: security hardening: SSRF protection on provider API endpoints, and rate limiting added across login, the REST API, the MCP server, and the web UI.

2026-04

Cloud launch: completionkit.com and www both serve the hosted app on Render, backed by Supabase Postgres.
Stripe billing: Free / Starter / Team plans, Stripe Checkout, Customer Portal, webhook-driven subscription sync, 14-day Team trial.
Multi-tenancy: per-organization data isolation via application-layer tenant scoping plus Postgres row-level security.
Authentication: sign-up, email verification, password reset, account management, revocable database-backed sessions.
Marketing site: landing and pricing pages moved into the cloud app from GitHub Pages.
Operations: Honeybadger error tracking, uptime monitoring with a public status page, log aggregation, Mission Control jobs dashboard.