> ## Documentation Index > Fetch the complete documentation index at: https://docs.usenexio.com/llms.txt > Use this file to discover all available pages before exploring further. # Evaluation and quality > Understand how Nexio measures an engine's quality and gates its releases, and what your team contributes and sees. Evaluation answers one question before and after every change: is this engine still right? Nexio operates it for you. Nexio measures an engine against reference cases, compares automated quality judgments with your people's judgments, and uses both to decide whether a new version may be released. It also drafts small configuration changes from the signal your team records and has a person approve them. The evaluation controls in the portal are operated by Nexio (see [Team and roles](/platform/team-and-roles)). There are no public API routes for them. Your team contributes the signal: [outcomes and annotations](/engines/outcomes-and-annotations) through the API, and reviews in the portal. You see the gate's verdict on an engine's first release (`gate_verified`) and the results on your runs. ## Who does what | Nexio operates | Your team contributes | What your team sees | | ---------------------------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | Evaluation sets, labels and thresholds | Outcomes: what happened after a run, for example its recommendation was accepted or overridden | `gate_verified` on an engine's first release from [Versions](/engines/versions) | | Evaluation runs and the release gate | Annotations: ratings and corrections on a run's result | The Review tab on each run, where you annotate it | | The judge and its calibration | Reviews of runs assigned to your reviewers | Runs and Metrics in the portal | | Improvement proposals and their approval | Feedback to Nexio on what the engine should do better | A new version when Nexio approves a proposal | ## How it works 1. Nexio builds an **evaluation set**: a group of reference cases for one engine, for example built from its completed runs. 2. Reviewers **label** cases with the answer the engine should give. 3. Nexio sets **thresholds**: how many cases may disagree with their labels, overall and per scoring dimension. 4. An **evaluation run** executes a candidate configuration against the set and compares its answers with the labels. 5. **Publishing** a version runs the evaluation gate on the candidate. A failing or regressing candidate is not released. 6. The **judge** rates completed runs, and its verdicts are compared with your team's annotations to measure how far it can be trusted. 7. The improvement **Pass** reads recent signal, including your outcomes and annotations, and drafts a **proposal**. A person with approval rights approves it, which releases a new version behind the evaluation gate, or rejects it. ## Evaluation sets | Concept | Meaning | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Evaluation set | A named group of cases for one engine. | | Case | One reference input, with the output the engine gave when the case was created and, once labeled, the expected output. Some cases are held out: they count toward the gate, but the improvement loop never sees their results. | | Set from runs | A set built from the engine's completed runs, 20 of the most recent by default. | | Cold-start set | A set of reference cases written with their expected answers, to measure a new engine before it has production runs. | | Label | A reviewer's statement of the right answer for a case. Only labeled cases can verify a release. | | Mismatch threshold | The share of labeled cases that may disagree with their labels before a candidate counts as failing. | | Dimension thresholds | The same limit, per scoring dimension. | ## The release gate Every publish that creates a new version, through `POST /api/v1/engines/{engine_slug}/versions` or by Nexio in the portal, runs the gate on the candidate configuration: | Engine state | What the gate requires | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | No evaluation set | Nothing. The release goes out with `gate_verified: false`. | | First release, with an evaluation set | A pass for this exact candidate on every set that declares a threshold. If no set declares one, a completed evaluation of the candidate with no case errors. | | Later release | A pass on every set that declares a threshold. Sets without one do not block. | A release is `gate_verified: true` only when it is the engine's first release and the passing evaluation compared the candidate with labeled cases. Every later release reads `gate_verified: false`, even when it passed the gate. A first release that passes on a set with no labeled cases still goes out, with `gate_verified: false`. The errors a blocked publish returns (`cold_start_gate_not_met`, `cold_start_gate_failed`, `cold_start_gate_regressed`) are listed on [Versions and releases](/engines/versions#errors). If a publish is blocked, ask Nexio to review the evaluation result with you. ## The judge and calibration The judge is an automated reviewer that rates run outputs. Its verdicts are useful only if they agree with your people. Nexio reduces each judge verdict and each of your annotations on the same run to positive, neutral or negative and tracks how often they agree. The Pass weighs judge verdicts below your annotations and outcomes when they disagree. ## Improvement proposals The Pass is the engine's improvement loop for engines of the `comparison`, `entity_analysis` and `opportunity` types. 1. The Pass reads the engine's recent annotations (up to 50), outcome counts, judge verdicts, evaluation results and the history of its released versions. 2. It drafts a small, specific change to the configuration and stores it as a proposal, with the reasoning and the kind of defect it addresses. A problem that a configuration change cannot fix is marked for engineering rather than papered over. 3. A person with the approval permission reviews the proposal, can run an evaluation of it, and approves or rejects it. 4. Approval re-validates the proposed configuration and checks it against the evaluation gate. Then, in one step, it checks that the engine's configuration has not changed since the proposal was drafted, promotes the proposal to the draft and releases a new version. On engines with the default pin policy that version is a minor. A stale proposal is refused rather than overwriting newer changes. ## What you do and what you call **Portal:** open a run, then the **Review** tab, to annotate it (needs `review:use`). **Runs** and **Metrics** show each run and the engine's volume, latency and status mix. See [The portal](/platform/portal). **API:** [record outcomes and annotations](/engines/outcomes-and-annotations) to supply the signal, and [publish versions](/engines/versions) to trigger the gate and read `gate_verified`. The signal evaluation reads. Publishing and the release gate.