> ## Documentation Index > Fetch the complete documentation index at: https://docs.usenexio.com/llms.txt > Use this file to discover all available pages before exploring further. # Extraction and run review > Know who reviews extracted facts and engine runs, what each review action changes for downstream readers, and how accuracy is measured. The platform has two kinds of human review. **Extraction review** verifies facts read from connected documents before, or after, they are served. **Run review** lets people in your organization inspect engine runs and record annotations on them. They share a name and nothing else: extraction review changes what served records and delivered files say, run review never does. | | Extraction review | Run review | | ---------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | What is reviewed | One fact read from one document | One engine run | | Who reviews | Nexio | People in your organization with `review:use` (the reviewer or member role) | | Where | The connection's Review page in the portal, operated by Nexio | Review, in the portal sidebar | | What it changes | Served values, verification labels, what a delivery holds back | Annotations on the run. Never served data | | Public API | None | None for the review screen. The annotations it writes can also be created through the public API: see [Outcomes and annotations](/engines/outcomes-and-annotations) | ## Extraction review ### How it works 1. The [extraction pipeline](/connections/indexing-and-extraction) flags a fact it cannot trust, for example: the quoted passage was not found in the document, confidence is low, two models disagreed, a page was image-only, or a value regressed. 2. The flag becomes a review item. Its flag class names who owns it, the plain question the reviewer is asked, and the actions the reviewer may take. 3. The reviewer opens the document with the cited passage highlighted, answers the question with one of the allowed actions, and records the disposition under their name. 4. The platform applies the action (table below) and, where the fact can be addressed, records a verification or a correction. A flag class is owned by a `human` (a person can settle it), by the `machine` (the pipeline could not read the value, so only a re-extraction settles it), or is `informational` (an audit record of a disagreement the pipeline already resolved; it goes to no queue). The allowed actions are `accept_value`, `enter_value`, `map_lob` (map a label printed in the document to a code in the document profile's vocabulary), `not_stated`, `not_applicable`, `reject`, and `request_reextraction`. For an item with a registered flag class, the server refuses any action the class does not allow. ### Review item state model | Status | Meaning | What happens next | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | `pending` | Waiting for a reviewer. While pending, a delivery holds back the record it is about when that record's family is one a delivery holds (see [Transformation and delivery](/connections/delivery#how-it-works)). | A reviewer approves, corrects, rejects or requests re-extraction. | | `approved` | A reviewer accepted the extracted value. | Stands until reopened, or until the fact changes. | | `corrected` | A reviewer entered a different value. | Stands until reopened. | | `rejected` | A reviewer set the question aside. | Not asked again on re-index. | | `reextraction_requested` | The document is being read again. | The new extraction raises new items if it still fails a check. | | `superseded` | Closed by the platform, not a person: a newer extraction replaced the served record the item pointed at, or a reviewer answered the same question on the byte-identical original. | Closed. | Transitions: `pending` to `approved`, `corrected`, `rejected`, `reextraction_requested` or `superseded`; `approved`, `corrected` or `rejected` back to `pending` by reopening. ### What each action changes for downstream readers | Action | Served value | Verification label | Undo | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | Accept | Unchanged. | Becomes `human` when the fact can be addressed in the document's current extraction; otherwise the accept is recorded on the item only. | Reopen or dispute returns the item to `pending` and withdraws the verification. | | Correct a field served at once | Changes at once: a new served version is written and the prior version is closed. | `human`, marked corrected. | Reopen. The correction can be withdrawn; it is never deleted. | | Map a label | Future projections use the learned mapping for every document that prints the label. | Unchanged (no verification is written). | Reopen. | | Correct another correctable field | Changes at the next projection. | `human`, marked corrected. | Reopen. | | Reject | Unchanged, and the question is not asked again on re-index. Rejecting a `blind_sample` item records a failed sample verdict instead, and the row is withheld from the next delivery. | Unchanged, except for the failed verdict on a `blind_sample` item. | Reopen. | | Request re-extraction | The document is read again; a new extraction supersedes the old one. | Recomputed from the new extraction. | Not applicable. | Which fields can be corrected, and which of the three paths above each one takes, is fixed in code for each document profile. The original extracted value is kept alongside a correction. Corrections are stored in their own ledger and applied as an overlay each time records are projected, so a rebuild keeps them. A verification is bound to its evidence: the value, the quoted passage, the page, the document's content hash and the extractor's schema version. If a later extraction changes the value or the evidence, or drops the fact, the verification is demoted. A reviewer can also dispute a verified fact, which demotes it and reopens its resolved item in one step. ### Verification tiers Every delivered record row carries the strength of confirmation behind it. The tier says how much stands behind a figure, never that the figure is right. | Tier | Label in delivered files | Meaning | | --------------- | ------------------------ | ------------------------------------------------------------------------------------------------- | | Human | `human` | A person opened the document, looked at the value and its evidence, and accepted or corrected it. | | Model agreement | `agreement` | Two different models read the stored text independently and returned the same figure. | | Single reading | `single` | One model's reading with nothing else behind it. The honest default for most facts. | A person's verification outranks a model agreement at the same fact. ### Re-extraction spend cap Re-extraction calls a model, so it costs money. Requests are capped per connection per day. A request over the cap is refused until spend inside the window falls; nothing is queued silently. An accepted request reports its projected cost. ### Review sessions and sampling A review session is one reviewer's sitting over one connection. It has one of three modes: | Mode | What it walks | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `flagged` | Documents with pending items. | | `sample` | A random draw over the connection's reviewable documents, not only flagged ones. A document is reviewable when it is the latest version from its source, has been extracted and indexed, and is not quarantined, removed at the source, or a byte-identical duplicate of another document. The draw is stratified by document type and takes at least one document from each non-empty type when the requested sample size allows it (a sample smaller than the number of types covers only as many types as it has documents), so an extraction that is confidently wrong can still be found. | | `custom` | A sample narrowed to chosen document types or counterparties, or to documents with pending items. | A `sample` or `custom` session draws 1 to 2,000 documents; a `flagged` session walks every document with pending items. The draw happens once when the session is created, so a half-finished session is never reshuffled. Sessions are workflow only: every choice inside one writes its own disposition and verification. ### Weekly blind sample For every connection where Nexio has armed delivery, the platform draws a random sample of served records each week and places them in the review queue under the `blind_sample` flag class. The draw is blind: a record's confidence, flags and dollar value play no part in which records are chosen, so the sample measures accuracy instead of confirming what the pipeline already doubted. The draw is due Monday at 06:30 UTC. The share of sampled records that pass is the measured accuracy reported with every delivery; until a sample has been fully reviewed, a delivery says accuracy is not yet measured. See [Transformation and delivery](/connections/delivery). ## Run review Run review is a portal surface for people in your organization. It is available today. ### Roles | Role or permission | What it allows | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Reviewer role | Grants `review:use` and `annotations:write`: the Review page and annotating runs. It grants no run, engine, key, or team management permission. Give it to domain experts alongside their base role. | | `review:moderate` | Send runs to review (one at a time, or up to 500 at once), assign them to reviewers, see every reviewer's queue, and delete items. Operated by Nexio. | See [Team and roles](/platform/team-and-roles) for how roles are assigned. ### How it works 1. A moderator sends a run to review and assigns it to a reviewer. 2. The reviewer opens Review in the sidebar. The page has four tabs: Inbox, Snoozed, Done and All. 3. The reviewer reads the run, adds annotations, and finishes the item with a disposition, or snoozes it. 4. Annotations are stored on the run. They do not change served data. ### Review item state model | Status | Meaning | What happens next | | ---------- | ---------------------------- | -------------------------------------------------------------------------------------- | | `pending` | In a reviewer's inbox. | The reviewer finishes it with a disposition, or snoozes it. | | `snoozed` | Hidden until a chosen time. | When the time passes it shows in the inbox again and can be finished or snoozed again. | | `reviewed` | Finished with a disposition. | The original reviewer or a moderator can reopen it to `pending`. | | Disposition | Resulting status | Required | | --------------------------- | ---------------- | --------------------- | | `reviewed_clean` | `reviewed` | Nothing. | | `reviewed_with_annotations` | `reviewed` | Nothing. | | `escalated` | `reviewed` | An escalation reason. | | `skipped` | Stays `pending` | A skip reason. | A reviewed item cannot be snoozed. Finishing or snoozing an item that is reviewed, or snoozed with time still to run, is refused. A reviewer who is not a moderator can act only on items assigned to them. Record annotations on runs through the API. How annotations become evaluation sets.