gate_verified) and the results on your runs.
Who does what
How it works
- Nexio builds an evaluation set: a group of reference cases for one engine, for example built from its completed runs.
- Reviewers label cases with the answer the engine should give.
- Nexio sets thresholds: how many cases may disagree with their labels, overall and per scoring dimension.
- An evaluation run executes a candidate configuration against the set and compares its answers with the labels.
- Publishing a version runs the evaluation gate on the candidate. A failing or regressing candidate is not released.
- The judge rates completed runs, and its verdicts are compared with your team’s annotations to measure how far it can be trusted.
- The improvement Pass reads recent signal, including your outcomes and annotations, and drafts a proposal. A person with approval rights approves it, which releases a new version behind the evaluation gate, or rejects it.
Evaluation sets
The release gate
Every publish that creates a new version, throughPOST /api/v1/engines/{engine_slug}/versions or by Nexio in the portal, runs the gate on the candidate configuration:
A release is
gate_verified: true only when it is the engine’s first release and the passing evaluation compared the candidate with labeled cases. Every later release reads gate_verified: false, even when it passed the gate. A first release that passes on a set with no labeled cases still goes out, with gate_verified: false. The errors a blocked publish returns (cold_start_gate_not_met, cold_start_gate_failed, cold_start_gate_regressed) are listed on Versions and releases. If a publish is blocked, ask Nexio to review the evaluation result with you.
The judge and calibration
The judge is an automated reviewer that rates run outputs. Its verdicts are useful only if they agree with your people. Nexio reduces each judge verdict and each of your annotations on the same run to positive, neutral or negative and tracks how often they agree. The Pass weighs judge verdicts below your annotations and outcomes when they disagree.Improvement proposals
The Pass is the engine’s improvement loop for engines of thecomparison, entity_analysis and opportunity types.
- The Pass reads the engine’s recent annotations (up to 50), outcome counts, judge verdicts, evaluation results and the history of its released versions.
- It drafts a small, specific change to the configuration and stores it as a proposal, with the reasoning and the kind of defect it addresses. A problem that a configuration change cannot fix is marked for engineering rather than papered over.
- A person with the approval permission reviews the proposal, can run an evaluation of it, and approves or rejects it.
- Approval re-validates the proposed configuration and checks it against the evaluation gate. Then, in one step, it checks that the engine’s configuration has not changed since the proposal was drafted, promotes the proposal to the draft and releases a new version. On engines with the default pin policy that version is a minor. A stale proposal is refused rather than overwriting newer changes.
What you do and what you call
Portal: open a run, then the Review tab, to annotate it (needsreview:use). Runs and Metrics show each run and the engine’s volume, latency and status mix. See The portal.
API: record outcomes and annotations to supply the signal, and publish versions to trigger the gate and read gate_verified.
Outcomes and annotations
The signal evaluation reads.
Versions and releases
Publishing and the release gate.