Annotations and evaluation

Two feedback loops keep an assistant from getting worse. Annotations are ratings people leave on individual turns. Eval scenarios are scripted conversations with checks that decide pass or fail; the gate suite runs on every publish (see Versions and publish), and the other suites run when you ask. Promote connects the two: one call turns a downrated turn into a gate scenario.

Credentials

Action	Organization API key	Scoped key
Create, list, retract annotations	Allowed	`conversations:use`
List scenarios, list and read eval runs, read diffs, read a version’s eval run	Allowed	`conversations:use`
Author, correct, or retire scenarios; promote an annotation; start an eval run	Allowed	Refused (`403 insufficient_capability`)

Scenario authoring and on-demand runs are also available in the portal on the instance’s Evals tab. Follower instances refuse authoring and runs with 409 instance_follows_canonical; their scenarios live on the instance they follow.

Annotations

An annotation rates one turn: good, bad, or neutral, with an optional comment and reason. POST .../conversations/{conversation_id}/annotations

Field	Type	Required	Rules
`end_user`	string	Yes	Must match the conversation.
`turn_id`	UUID	Yes	A turn in this conversation (`turn_id` from the `conversation` frame or a message). Otherwise `404 turn_not_found`.
`rating`	`good`, `bad`, `neutral`	Yes
`comment`	string	No	At most 4,000 bytes of UTF-8 text.
`reason`	string	No	A short reason code of your choosing, at most 64 bytes of UTF-8 text.
`target`	JSON object	No	Which part of the turn the rating is about, at most 4,096 bytes.
`submitter_id`	string	No	A stable id for the person rating, from your system.
`feedback_key`	string	No	At most 64 characters. Requires `submitter_id`.

With submitter_id and feedback_key, posting again for the same turn, submitter, and key updates the one annotation instead of adding another. Use it for a thumbs control that the user can change.

curl -X POST https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/conversations/9e4c7a12-5b3d-4f81-a6e0-2d8b1c4f7a93/annotations \
  -H "Authorization: Bearer $NEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "end_user": "u_dana_ortiz",
    "turn_id": "5f2a8c61-3e7b-4d09-b1c4-8a6e2f9d0c75",
    "rating": "bad",
    "reason": "wrong_run_count",
    "comment": "It said three runs failed; there were four.",
    "submitter_id": "u_dana_ortiz",
    "feedback_key": "thumbs"
  }'

import os
import requests

resp = requests.post(
    "https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/conversations/9e4c7a12-5b3d-4f81-a6e0-2d8b1c4f7a93/annotations",
    headers={"Authorization": "Bearer " + os.environ["NEXIO_API_KEY"]},
    json={
        "end_user": "u_dana_ortiz",
        "turn_id": "5f2a8c61-3e7b-4d09-b1c4-8a6e2f9d0c75",
        "rating": "bad",
        "reason": "wrong_run_count",
        "comment": "It said three runs failed; there were four.",
        "submitter_id": "u_dana_ortiz",
        "feedback_key": "thumbs",
    },
)
annotation = resp.json()

const annotation = await fetch(
  "https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/conversations/9e4c7a12-5b3d-4f81-a6e0-2d8b1c4f7a93/annotations",
  {
    method: "POST",
    headers: { Authorization: `Bearer ${process.env.NEXIO_API_KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify({
      end_user: "u_dana_ortiz",
      turn_id: "5f2a8c61-3e7b-4d09-b1c4-8a6e2f9d0c75",
      rating: "bad",
      reason: "wrong_run_count",
      comment: "It said three runs failed; there were four.",
      submitter_id: "u_dana_ortiz",
      feedback_key: "thumbs",
    }),
  },
).then((r) => r.json())

Response 201:

{
  "id": "b4e17c90-2a5d-4f3b-9c81-6e0d2a7f4b35",
  "conversation_id": "9e4c7a12-5b3d-4f81-a6e0-2d8b1c4f7a93",
  "instance_id": "3b9d2f4e-8a1c-4e57-9f02-6c1d8e7a5b34",
  "turn_id": "5f2a8c61-3e7b-4d09-b1c4-8a6e2f9d0c75",
  "end_user": "u_dana_ortiz",
  "rating": "bad",
  "comment": "It said three runs failed; there were four.",
  "reason": "wrong_run_count",
  "submitter_id": "u_dana_ortiz",
  "feedback_key": "thumbs",
  "source": "api",
  "created_at": "2026-09-23T14:10:44.912305Z",
  "updated_at": "2026-09-23T14:10:44.912305Z"
}

Restore after a reload: GET .../annotations?end_user=...&submitter_id=... returns {annotations, truncated}: up to 500 of that submitter’s API annotations on the conversation, newest first. Annotations made in the portal are never returned here.
Retract: DELETE .../annotations/{annotation_id} with a JSON body {end_user, submitter_id}. Answers 204. A mismatch, a portal annotation, or one already retracted is 404 annotation_not_found.

Annotation errors: 400 invalid_turn_id, invalid_rating, invalid_comment, invalid_reason, invalid_target, invalid_feedback_key, invalid_request; 404 conversation_not_found, turn_not_found, annotation_not_found. In the portal, the instance’s Feedback tab shows good and bad ratings from both the API and the portal, and your team can read them there. Nexio staff set a triage status on each one: new, acknowledged, in progress, fixed, won’t fix, or duplicate. Changing it needs review:moderate, which only Nexio staff hold.

Eval scenarios

A scenario is a scripted conversation plus a rubric.

Field	Rules
`name`	At most 200 bytes (UTF-8).
`script`	1 to 8 turns, at most 64 KiB. Each turn has `message` and optional `tool_results` and `confirmations`, both keyed by client tool name.
`rubric`	At most 16 KiB, with at least one deterministic check.
`suite`	`gate` (default), `workflows`, `adversarial`, or `smoke`.

No application is attached during an eval, so the script plays your application’s part. A client tool handed off with no scripted result gets a scripted error. A confirmation with no scripted decision is denied. Platform read tools run for real against your org’s data; platform write tools run as a dry run with no side effect (see Platform tools). Every scripted turn makes real model calls, metered to your org.

Rubric checks

Deterministic checks alone decide pass or fail. judge dimensions are scored by a model and recorded, and never fail a scenario.

Check	Passes when
`must_refuse: [rule ids]`	Each listed refusal rule fired.
`must_escalate: [rule ids]`	Each listed escalation rule fired.
`must_call_tools: [{name, input_contains?}]`	Each named tool was called; with `input_contains`, at least one call’s input contains that text (case-sensitive).
`must_emit_components: [{component}]`	Each named component was emitted.
`must_cite: [refs]`	Each reference appears in a citation or in the final answer text.
`must_not_refuse: true`	No refusal fired anywhere, and the final turn ended normally (not refused, escalated, withheld, or cut off by the token budget).
`final_must_not_contain: [phrases]`	None of the phrases appears in the final answer, compared case-insensitively.
`judge: [{name, criteria}]`	Never decides. Scored and recorded.

{
  "name": "summarizes failed runs with the right count",
  "suite": "workflows",
  "script": [
    { "message": "How many runs failed today?" }
  ],
  "rubric": {
    "must_call_tools": [{ "name": "runs.list", "input_contains": "FAILED" }],
    "must_not_refuse": true,
    "final_must_not_contain": ["I cannot help with that"],
    "judge": [{ "name": "concise", "criteria": "Answers in two sentences or fewer." }]
  }
}

Suites and caps

Suite	Runs	Active scenario cap
`gate`	On every publish, and on demand	40
`workflows`, `adversarial`, `smoke`	On demand only	200 each

A scenario over the cap is refused with 409 conversation_eval_scenario_cap. Editing or retiring a gate scenario while a publish’s gate is running aborts that publish with 409 scenario_set_changed_during_publish.

Scenario routes

GET .../eval-scenarios?suite=... lists active scenarios in authoring order.
POST .../eval-scenarios creates one. 201 with the scenario.
PATCH .../eval-scenarios/{scenario_id} corrects one in place, keeping its id and its past results. Every field is optional; the merged scenario is validated as a whole.
DELETE .../eval-scenarios/{scenario_id} retires one. 204. Past results stay readable. Later selections skip it; an on-demand run that already read its scenario list may still run it. Retiring one that is already retired is 404.

Scenario errors: 400 invalid_eval_scenario (with details), 400 invalid_request, 404 conversation_eval_scenario_not_found, 409 conversation_eval_scenario_cap, 409 instance_follows_canonical.

Promote an annotation

POST /api/v1/conversation-instances/{instance_slug}/annotations/{annotation_id}/promote with {rubric, name?} drafts a gate scenario from the annotated conversation. The script replays the user messages on that turn’s branch up to and including the annotated turn (at most the last 8 turns), with the recorded client tool results scripted in and those calls scripted as approved. The rubric is required and must carry a deterministic check. Answers 201 with the scenario, whose origin is promoted_from_annotation and whose annotation_id points back.

{
  "name": "counts failed runs correctly",
  "rubric": {
    "must_call_tools": [{ "name": "runs.list" }],
    "final_must_not_contain": ["three runs failed"]
  }
}

Errors: 400 invalid_request (body is not JSON), 400 invalid_eval_scenario (no rubric in the request), 404 annotation_not_found, 409 conversation_eval_scenario_cap, 409 instance_follows_canonical, 422 annotation_not_promotable (the annotated turn is not among the conversation’s newest 500 messages, there are no user turns to script, or the drafted scenario is invalid, for example a rubric that is {} or has no checks).

Run a suite on demand

An eval run measures the instance’s latest published version, not the draft. An instance with no published version is 409 instance_not_published.

Start the run

POST .../eval-runs with {"suite": "workflows"}. Needs an organization API key. The response is 201 with the run at status running.

curl -X POST https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/eval-runs \
  -H "Authorization: Bearer $NEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"suite": "workflows"}'

import os
import requests

run = requests.post(
    "https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/eval-runs",
    headers={"Authorization": "Bearer " + os.environ["NEXIO_API_KEY"]},
    json={"suite": "workflows"},
).json()

const run = await fetch("https://api.usenexio.com/api/v1/conversation-instances/workspace-assistant/eval-runs", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.NEXIO_API_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({ suite: "workflows" }),
}).then((r) => r.json())

{
  "id": "e81f4a2c-3b7d-4c95-a0e6-2f9b5d1c8a47",
  "suite": "workflows",
  "status": "running",
  "triggered_by": "manual",
  "version": "4",
  "pass_count": 0,
  "fail_count": 0,
  "created_at": "2026-09-23T15:30:02.448190Z"
}

Poll until it finishes

GET .../eval-runs/{run_id} returns {run, results}. Poll every 30 seconds or more; scenarios run one after another.

{
  "run": {
    "id": "e81f4a2c-3b7d-4c95-a0e6-2f9b5d1c8a47",
    "suite": "workflows",
    "status": "failed",
    "triggered_by": "manual",
    "version": "4",
    "pass_count": 2,
    "fail_count": 1,
    "created_at": "2026-09-23T15:30:02.448190Z"
  },
  "results": [
    {
      "scenario_id": "2d7a9e31-5c4b-4f08-b6e2-8a1d3f7c9b52",
      "scenario_name": "summarizes failed runs with the right count",
      "passed": false,
      "scores": {
        "checks": [
          { "check": "must_not_refuse", "passed": false, "detail": "final turn stopped at escalated" },
          { "check": "must_call_tool:runs.list", "passed": true, "detail": "tool called as expected" },
          { "check": "final_must_not_contain:I cannot help with that", "passed": true, "detail": "phrase absent from the final answer" }
        ],
        "judge": { "concise": 0.8 }
      }
    },
    {
      "scenario_id": "6f1c3a85-0d2e-4b79-8a46-c5e9b2d7f013",
      "scenario_name": "lists engines",
      "passed": true,
      "scores": {
        "checks": [
          { "check": "must_call_tool:engines.list", "passed": true, "detail": "tool called as expected" }
        ]
      }
    },
    {
      "scenario_id": "a3d8e6b2-7c14-4f5a-9e0b-1d2f6c8a4b97",
      "scenario_name": "explains a webhook failure",
      "passed": true,
      "scores": {
        "checks": [
          { "check": "must_call_tool:webhooks.list", "passed": true, "detail": "tool called as expected" }
        ]
      }
    }
  ]
}

Diff against a baseline

GET .../eval-runs/{run_id}/diff compares with the previous completed (passed, failed, or waived) run of the same suite, or with ?baseline={run_id}. With no baseline and no earlier completed run, the answer is 404 conversation_eval_baseline_not_found.

{
  "run_id": "e81f4a2c-3b7d-4c95-a0e6-2f9b5d1c8a47",
  "baseline_run_id": "7c2b9d40-1e6f-4a83-95d2-3b8e0f4a6c19",
  "diff": {
    "newly_failing": ["summarizes failed runs with the right count"],
    "newly_passing": [],
    "still_failing": [],
    "still_passing": ["lists engines", "explains a webhook failure"],
    "added": [],
    "removed": []
  }
}

GET .../eval-runs?suite=...&limit=... lists runs newest first; limit is 1 to 200, default 50.

Eval run states

Status	Meaning
`running`	Queued or executing. The platform picks up runs every 30 seconds and holds each for up to 15 minutes per scenario before another worker may resume it.
`passed`	On-demand run: every scenario passed. Publish gate run: no regression against the prior version’s run. A first gate run with no prior run, or a gate run whose failures were already failing before, is also `passed`. Read `fail_count` and the results for scenario outcomes.
`failed`	On-demand run: at least one scenario failed. Publish gate run: a regression, that is a scenario that newly fails or a new scenario that fails.
`waived`	A publish gate run whose regression was waived.
`error`	The run cannot complete. A permanent failure (the instance, its archived configuration, a stored scenario, or the eval executor is unavailable or unreadable) ends the run at once. A transient execution failure is retried, and the run ends here after 5 attempts. `error` says why; results already recorded stay.

triggered_by is publish, manual, schedule, or event. Starting a run in a deployment without the eval executor is 503 conversation_eval_unavailable.

Read a version’s latest eval run

GET .../versions/{version}/eval-run returns the most recent eval run recorded for that version, whatever started it. The publish gate run is the first, and a later on-demand run for the same version replaces it in this answer: {version, config_version_hash, run, results, diff?}. version is the integer as a string, and diff compares with the run of the version immediately before it (prior_version, newly_failing, new_failing, newly_passing); it is absent when that version has no recorded run. Errors: 400 instance_version_invalid_format, 404 instance_version_not_found, 404 conversation_eval_run_not_found.

Versions and publish

How the gate suite decides a release.

Guardrails

Rules that must_refuse and must_escalate test.

Start here

Engines

Conversations

Connections and data

Records

Events and webhooks

Platform

Reference

Annotations and evaluation

Credentials

Annotations

Eval scenarios

Rubric checks

Suites and caps

Scenario routes

Promote an annotation

Run a suite on demand

Eval run states

Read a version’s latest eval run

Versions and publish

Guardrails

​Credentials

​Annotations

​Eval scenarios

​Rubric checks

​Suites and caps

​Scenario routes

​Promote an annotation

​Run a suite on demand

​Eval run states

​Read a version’s latest eval run

Versions and publish

Guardrails

Credentials

Annotations

Eval scenarios

Rubric checks

Suites and caps

Scenario routes

Promote an annotation

Run a suite on demand

Eval run states

Read a version’s latest eval run