> ## Documentation Index > Fetch the complete documentation index at: https://docs.usenexio.com/llms.txt > Use this file to discover all available pages before exploring further. # Annotations and evaluation > Collect ratings on turns, write eval scenarios, run suites on demand, and turn a bad answer into a regression test. Two feedback loops keep an assistant from getting worse. **Annotations** are ratings people leave on individual turns. **Eval scenarios** are scripted conversations with checks that decide pass or fail; the `gate` suite runs on every publish (see [Versions and publish](/conversations/versions-and-publish)), and the other suites run when you ask. **Promote** connects the two: one call turns a downrated turn into a gate scenario. ## Credentials | Action | Organization API key | Scoped key | | ------------------------------------------------------------------------------ | -------------------- | --------------------------------------- | | Create, list, retract annotations | Allowed | `conversations:use` | | List scenarios, list and read eval runs, read diffs, read a version's eval run | Allowed | `conversations:use` | | Author, correct, or retire scenarios; promote an annotation; start an eval run | Allowed | Refused (`403 insufficient_capability`) | Scenario authoring and on-demand runs are also available in the portal on the instance's **Evals** tab. Follower instances refuse authoring and runs with `409 instance_follows_canonical`; their scenarios live on the instance they follow. ## Annotations An annotation rates one turn: `good`, `bad`, or `neutral`, with an optional comment and reason. `POST .../conversations/{conversation_id}/annotations` | Field | Type | Required | Rules | | -------------- | ------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------- | | `end_user` | string | Yes | Must match the conversation. | | `turn_id` | UUID | Yes | A turn in this conversation (`turn_id` from the `conversation` frame or a message). Otherwise `404 turn_not_found`. | | `rating` | `good`, `bad`, `neutral` | Yes | | | `comment` | string | No | At most 4,000 bytes of UTF-8 text. | | `reason` | string | No | A short reason code of your choosing, at most 64 bytes of UTF-8 text. | | `target` | JSON object | No | Which part of the turn the rating is about, at most 4,096 bytes. | | `submitter_id` | string | No | A stable id for the person rating, from your system. | | `feedback_key` | string | No | At most 64 characters. Requires `submitter_id`. | With `submitter_id` and `feedback_key`, posting again for the same turn, submitter, and key updates the one annotation instead of adding another. Use it for a thumbs control that the user can change. ```bash curl theme={null} curl -X POST https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/conversations/9e4c7a12-5b3d-4f81-a6e0-2d8b1c4f7a93/annotations \ -H "Authorization: Bearer $NEXIO_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "end_user": "u_dana_ortiz", "turn_id": "5f2a8c61-3e7b-4d09-b1c4-8a6e2f9d0c75", "rating": "bad", "reason": "wrong_run_count", "comment": "It said three runs failed; there were four.", "submitter_id": "u_dana_ortiz", "feedback_key": "thumbs" }' ``` ```python Python theme={null} import os import requests resp = requests.post( "https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/conversations/9e4c7a12-5b3d-4f81-a6e0-2d8b1c4f7a93/annotations", headers={"Authorization": "Bearer " + os.environ["NEXIO_API_KEY"]}, json={ "end_user": "u_dana_ortiz", "turn_id": "5f2a8c61-3e7b-4d09-b1c4-8a6e2f9d0c75", "rating": "bad", "reason": "wrong_run_count", "comment": "It said three runs failed; there were four.", "submitter_id": "u_dana_ortiz", "feedback_key": "thumbs", }, ) annotation = resp.json() ``` ```typescript TypeScript theme={null} const annotation = await fetch( "https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/conversations/9e4c7a12-5b3d-4f81-a6e0-2d8b1c4f7a93/annotations", { method: "POST", headers: { Authorization: `Bearer ${process.env.NEXIO_API_KEY}`, "Content-Type": "application/json" }, body: JSON.stringify({ end_user: "u_dana_ortiz", turn_id: "5f2a8c61-3e7b-4d09-b1c4-8a6e2f9d0c75", rating: "bad", reason: "wrong_run_count", comment: "It said three runs failed; there were four.", submitter_id: "u_dana_ortiz", feedback_key: "thumbs", }), }, ).then((r) => r.json()) ``` Response `201`: ```json theme={null} { "id": "b4e17c90-2a5d-4f3b-9c81-6e0d2a7f4b35", "conversation_id": "9e4c7a12-5b3d-4f81-a6e0-2d8b1c4f7a93", "instance_id": "3b9d2f4e-8a1c-4e57-9f02-6c1d8e7a5b34", "turn_id": "5f2a8c61-3e7b-4d09-b1c4-8a6e2f9d0c75", "end_user": "u_dana_ortiz", "rating": "bad", "comment": "It said three runs failed; there were four.", "reason": "wrong_run_count", "submitter_id": "u_dana_ortiz", "feedback_key": "thumbs", "source": "api", "created_at": "2026-09-23T14:10:44.912305Z", "updated_at": "2026-09-23T14:10:44.912305Z" } ``` * **Restore after a reload:** `GET .../annotations?end_user=...&submitter_id=...` returns `{annotations, truncated}`: up to 500 of that submitter's API annotations on the conversation, newest first. Annotations made in the portal are never returned here. * **Retract:** `DELETE .../annotations/{annotation_id}` with a JSON body `{end_user, submitter_id}`. Answers `204`. A mismatch, a portal annotation, or one already retracted is `404 annotation_not_found`. Annotation errors: `400 invalid_turn_id`, `invalid_rating`, `invalid_comment`, `invalid_reason`, `invalid_target`, `invalid_feedback_key`, `invalid_request`; `404 conversation_not_found`, `turn_not_found`, `annotation_not_found`. In the portal, the instance's **Feedback** tab shows `good` and `bad` ratings from both the API and the portal, and your team can read them there. Nexio staff set a triage status on each one: new, acknowledged, in progress, fixed, won't fix, or duplicate. Changing it needs `review:moderate`, which only Nexio staff hold. ## Eval scenarios A scenario is a scripted conversation plus a rubric. | Field | Rules | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `name` | At most 200 bytes (UTF-8). | | `script` | 1 to 8 turns, at most 64 KiB. Each turn has `message` and optional `tool_results` and `confirmations`, both keyed by client tool **name**. | | `rubric` | At most 16 KiB, with at least one deterministic check. | | `suite` | `gate` (default), `workflows`, `adversarial`, or `smoke`. | No application is attached during an eval, so the script plays your application's part. A client tool handed off with no scripted result gets a scripted error. A confirmation with no scripted decision is denied. Platform read tools run for real against your org's data; platform write tools run as a dry run with no side effect (see [Platform tools](/conversations/platform-tools#writes-in-evaluation-runs)). Every scripted turn makes real model calls, metered to your org. ### Rubric checks Deterministic checks alone decide pass or fail. `judge` dimensions are scored by a model and recorded, and never fail a scenario. | Check | Passes when | | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `must_refuse: [rule ids]` | Each listed refusal rule fired. | | `must_escalate: [rule ids]` | Each listed escalation rule fired. | | `must_call_tools: [{name, input_contains?}]` | Each named tool was called; with `input_contains`, at least one call's input contains that text (case-sensitive). | | `must_emit_components: [{component}]` | Each named component was emitted. | | `must_cite: [refs]` | Each reference appears in a citation or in the final answer text. | | `must_not_refuse: true` | No refusal fired anywhere, and the final turn ended normally (not refused, escalated, withheld, or cut off by the token budget). | | `final_must_not_contain: [phrases]` | None of the phrases appears in the final answer, compared case-insensitively. | | `judge: [{name, criteria}]` | Never decides. Scored and recorded. | ```json theme={null} { "name": "summarizes failed runs with the right count", "suite": "workflows", "script": [ { "message": "How many runs failed today?" } ], "rubric": { "must_call_tools": [{ "name": "runs.list", "input_contains": "FAILED" }], "must_not_refuse": true, "final_must_not_contain": ["I cannot help with that"], "judge": [{ "name": "concise", "criteria": "Answers in two sentences or fewer." }] } } ``` ### Suites and caps | Suite | Runs | Active scenario cap | | ----------------------------------- | ------------------------------- | ------------------- | | `gate` | On every publish, and on demand | 40 | | `workflows`, `adversarial`, `smoke` | On demand only | 200 each | A scenario over the cap is refused with `409 conversation_eval_scenario_cap`. Editing or retiring a `gate` scenario while a publish's gate is running aborts that publish with `409 scenario_set_changed_during_publish`. ### Scenario routes * `GET .../eval-scenarios?suite=...` lists active scenarios in authoring order. * `POST .../eval-scenarios` creates one. `201` with the scenario. * `PATCH .../eval-scenarios/{scenario_id}` corrects one in place, keeping its id and its past results. Every field is optional; the merged scenario is validated as a whole. * `DELETE .../eval-scenarios/{scenario_id}` retires one. `204`. Past results stay readable. Later selections skip it; an on-demand run that already read its scenario list may still run it. Retiring one that is already retired is `404`. Scenario errors: `400 invalid_eval_scenario` (with `details`), `400 invalid_request`, `404 conversation_eval_scenario_not_found`, `409 conversation_eval_scenario_cap`, `409 instance_follows_canonical`. ## Promote an annotation `POST /api/v1/conversation-instances/{instance_slug}/annotations/{annotation_id}/promote` with `{rubric, name?}` drafts a `gate` scenario from the annotated conversation. The script replays the user messages on that turn's branch up to and including the annotated turn (at most the last 8 turns), with the recorded client tool results scripted in and those calls scripted as approved. The rubric is required and must carry a deterministic check. Answers `201` with the scenario, whose `origin` is `promoted_from_annotation` and whose `annotation_id` points back. ```json theme={null} { "name": "counts failed runs correctly", "rubric": { "must_call_tools": [{ "name": "runs.list" }], "final_must_not_contain": ["three runs failed"] } } ``` Errors: `400 invalid_request` (body is not JSON), `400 invalid_eval_scenario` (no `rubric` in the request), `404 annotation_not_found`, `409 conversation_eval_scenario_cap`, `409 instance_follows_canonical`, `422 annotation_not_promotable` (the annotated turn is not among the conversation's newest 500 messages, there are no user turns to script, or the drafted scenario is invalid, for example a rubric that is `{}` or has no checks). ## Run a suite on demand An eval run measures the instance's latest **published** version, not the draft. An instance with no published version is `409 instance_not_published`. `POST .../eval-runs` with `{"suite": "workflows"}`. Needs an organization API key. The response is `201` with the run at status `running`. ```bash curl theme={null} curl -X POST https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/eval-runs \ -H "Authorization: Bearer $NEXIO_API_KEY" \ -H "Content-Type: application/json" \ -d '{"suite": "workflows"}' ``` ```python Python theme={null} import os import requests run = requests.post( "https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/eval-runs", headers={"Authorization": "Bearer " + os.environ["NEXIO_API_KEY"]}, json={"suite": "workflows"}, ).json() ``` ```typescript TypeScript theme={null} const run = await fetch("https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/eval-runs", { method: "POST", headers: { Authorization: `Bearer ${process.env.NEXIO_API_KEY}`, "Content-Type": "application/json" }, body: JSON.stringify({ suite: "workflows" }), }).then((r) => r.json()) ``` ```json theme={null} { "id": "e81f4a2c-3b7d-4c95-a0e6-2f9b5d1c8a47", "suite": "workflows", "status": "running", "triggered_by": "manual", "version": "4", "pass_count": 0, "fail_count": 0, "created_at": "2026-09-23T15:30:02.448190Z" } ``` `GET .../eval-runs/{run_id}` returns `{run, results}`. Poll every 30 seconds or more; scenarios run one after another. ```json theme={null} { "run": { "id": "e81f4a2c-3b7d-4c95-a0e6-2f9b5d1c8a47", "suite": "workflows", "status": "failed", "triggered_by": "manual", "version": "4", "pass_count": 2, "fail_count": 1, "created_at": "2026-09-23T15:30:02.448190Z" }, "results": [ { "scenario_id": "2d7a9e31-5c4b-4f08-b6e2-8a1d3f7c9b52", "scenario_name": "summarizes failed runs with the right count", "passed": false, "scores": { "checks": [ { "check": "must_not_refuse", "passed": false, "detail": "final turn stopped at escalated" }, { "check": "must_call_tool:runs.list", "passed": true, "detail": "tool called as expected" }, { "check": "final_must_not_contain:I cannot help with that", "passed": true, "detail": "phrase absent from the final answer" } ], "judge": { "concise": 0.8 } } }, { "scenario_id": "6f1c3a85-0d2e-4b79-8a46-c5e9b2d7f013", "scenario_name": "lists engines", "passed": true, "scores": { "checks": [ { "check": "must_call_tool:engines.list", "passed": true, "detail": "tool called as expected" } ] } }, { "scenario_id": "a3d8e6b2-7c14-4f5a-9e0b-1d2f6c8a4b97", "scenario_name": "explains a webhook failure", "passed": true, "scores": { "checks": [ { "check": "must_call_tool:webhooks.list", "passed": true, "detail": "tool called as expected" } ] } } ] } ``` `GET .../eval-runs/{run_id}/diff` compares with the previous completed (`passed`, `failed`, or `waived`) run of the same suite, or with `?baseline={run_id}`. With no `baseline` and no earlier completed run, the answer is `404 conversation_eval_baseline_not_found`. ```json theme={null} { "run_id": "e81f4a2c-3b7d-4c95-a0e6-2f9b5d1c8a47", "baseline_run_id": "7c2b9d40-1e6f-4a83-95d2-3b8e0f4a6c19", "diff": { "newly_failing": ["summarizes failed runs with the right count"], "newly_passing": [], "still_failing": [], "still_passing": ["lists engines", "explains a webhook failure"], "added": [], "removed": [] } } ``` `GET .../eval-runs?suite=...&limit=...` lists runs newest first; `limit` is 1 to 200, default 50. ### Eval run states | Status | Meaning | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `running` | Queued or executing. The platform picks up runs every 30 seconds and holds each for up to 15 minutes per scenario before another worker may resume it. | | `passed` | On-demand run: every scenario passed. Publish gate run: no regression against the prior version's run. A first gate run with no prior run, or a gate run whose failures were already failing before, is also `passed`. Read `fail_count` and the results for scenario outcomes. | | `failed` | On-demand run: at least one scenario failed. Publish gate run: a regression, that is a scenario that newly fails or a new scenario that fails. | | `waived` | A publish gate run whose regression was waived. | | `error` | The run cannot complete. A permanent failure (the instance, its archived configuration, a stored scenario, or the eval executor is unavailable or unreadable) ends the run at once. A transient execution failure is retried, and the run ends here after 5 attempts. `error` says why; results already recorded stay. | `triggered_by` is `publish`, `manual`, `schedule`, or `event`. Starting a run in a deployment without the eval executor is `503 conversation_eval_unavailable`. ## Read a version's latest eval run `GET .../versions/{version}/eval-run` returns the most recent eval run recorded for that version, whatever started it. The publish gate run is the first, and a later on-demand run for the same version replaces it in this answer: `{version, config_version_hash, run, results, diff?}`. `version` is the integer as a string, and `diff` compares with the run of the version immediately before it (`prior_version`, `newly_failing`, `new_failing`, `newly_passing`); it is absent when that version has no recorded run. Errors: `400 instance_version_invalid_format`, `404 instance_version_not_found`, `404 conversation_eval_run_not_found`. How the gate suite decides a release. Rules that `must_refuse` and `must_escalate` test.