> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usenexio.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Start an eval run

> Run one suite against the latest published version on demand. Organization API keys only.

## Behavior

The run is queued and executes in the background; the response is `201` with status `running`. Poll [Get an eval run](/api-reference/conversations/versions/get-eval-run) until the status is `passed`, `failed`, or `error`. The state model is on [Annotations and evaluation](/conversations/evaluation#eval-run-states).


## OpenAPI

````yaml POST /api/v1/conversation-instances/{instance_slug}/eval-runs
openapi: 3.1.0
info:
  title: Nexio API
  version: '1.0'
  description: |
    The Nexio public API. Submit runs to configured engines and read their
    results, hold conversations with configured assistants, make authorized
    reads over the planes and families of a connected system of record, and
    manage webhooks and environments.

    Six engine types are registered: comparison, matching, entity_analysis,
    diligence, triage, and opportunity.

    Every route under /api/v1/ requires `Authorization: Bearer <key>`, except
    inbound event ingest (`POST /api/v1/events/ingest/{source_key}`), which
    authenticates each request against its ingest source instead: an
    HMAC-SHA256 signature for `nexio` and `github` sources, or a shared
    authentication code for an `ams360_ons` source. Error
    bodies share the `Error` schema. Call the API from servers only: it sends
    no CORS headers.
  contact:
    email: support@usenexio.com
    url: https://docs.usenexio.com
servers:
  - url: https://api.usenexio.com
    description: >-
      Nexio API. Sandbox or live is chosen by the API key's environment, not by
      the host.
security:
  - BearerAuth: []
tags:
  - name: EngineManagement
    description: >-
      Create, configure, version and manage engines. An engine is a
      configuration of one registered engine type.
  - name: Engines
    description: Engine-scoped endpoints for submitting runs and retrieving results.
  - name: Runs
    description: Retrieve and reconcile submitted runs.
  - name: Records
    description: >-
      Typed, authorized reads over the registered planes and families of a
      connected system of record, and governed writes to the action ledger. The
      generic family read is the core contract.
  - name: Graph
    description: >-
      The organization's data graph: a read-only map of connections and the
      derivations scheduled on them. Node kinds are a closed registry of 16; the
      API serves 14.
  - name: Environments
    description: >-
      The live environment and up to 5 sandbox environments in your org.
      Organization keys only.
  - name: Webhooks
    description: >-
      Manage webhook endpoints that receive terminal run events (run.completed,
      run.failed, run.cancelled) and run corrections (run.superseded).
  - name: Events
    description: >-
      Send inbound events to your organization's event log through an ingest
      source. Source kinds are nexio, github and ams360_ons; a nexio source
      sends event types from your own vocabulary.
  - name: Converse
    description: The raw stateless conversational model turn (caller-owned state).
  - name: Conversations
    description: >-
      Conversation instances (orchestrators over engines) and their managed
      conversations, turns, annotations, evals, and exports.
  - name: Platform
    description: Service health. No key needed.
paths:
  /api/v1/conversation-instances/{instance_slug}/eval-runs:
    parameters:
      - $ref: '#/components/parameters/InstanceSlug'
    post:
      tags:
        - Conversations
      summary: Start an on-demand eval run
      description: |
        Queues a run of one eval suite against the instance's latest
        published version (never the draft). The run executes in the
        background with real model calls metered to the organization, one
        scenario after another; poll the run until its status leaves
        `running`. Platform write tools run as a dry run during evals.

        Scoped keys are refused (`403 insufficient_capability`); use an
        organization API key or the portal's Evals tab. A follower instance
        refuses with `instance_follows_canonical`.
      operationId: createConversationEvalRun
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required:
                - suite
              properties:
                suite:
                  type: string
                  enum:
                    - gate
                    - workflows
                    - adversarial
                    - smoke
            example:
              suite: workflows
      responses:
        '201':
          description: The queued run.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ConversationEvalOnDemandRun'
        '400':
          description: >-
            The instance slug is empty (`missing_instance_slug`), or the body is
            not JSON or `suite` is not one of the four suites
            (`invalid_request`).
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '401':
          $ref: '#/components/responses/Unauthorized'
        '403':
          description: >-
            Every scoped key is refused (`insufficient_capability`), whatever
            capabilities it holds, or the instance is archived
            (`instance_archived`).
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '404':
          description: >-
            The instance was not found in this organization
            (`instance_not_found`).
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '409':
          description: >-
            The instance has no published version (`instance_not_published`) or
            follows a platform-managed instance (`instance_follows_canonical`).
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '429':
          $ref: '#/components/responses/RateLimited'
        '500':
          description: The run could not be started (`internal_error`).
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '503':
          description: >
            No eval executor is configured in this deployment
            (`conversation_eval_unavailable`).


            Also `auth_unavailable`: API key authentication was briefly
            unavailable before the route ran. That cause is transient; retry it
            with backoff.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '504':
          $ref: '#/components/responses/RequestTimeout'
      security:
        - BearerAuth: []
components:
  parameters:
    InstanceSlug:
      name: instance_slug
      in: path
      required: true
      description: Conversation instance identifier slug (e.g. `platform-assistant`).
      schema:
        type: string
  schemas:
    ConversationEvalOnDemandRun:
      type: object
      description: One eval run, from a publish gate or started on demand.
      required:
        - id
        - suite
        - status
        - triggered_by
        - pass_count
        - fail_count
        - created_at
      properties:
        id:
          type: string
          format: uuid
        suite:
          type: string
          enum:
            - gate
            - workflows
            - adversarial
            - smoke
        status:
          type: string
          enum:
            - running
            - passed
            - failed
            - waived
            - error
          description: >
            `running` while queued or executing. On an on-demand run, `passed`

            when every scenario passed and `failed` when at least one failed.

            On a publish gate run, `passed` means no regression against the

            prior version's run (a first gate run, or failures that were

            already failing, still pass; read `fail_count`), and `failed`

            means a scenario newly fails or a new scenario fails. `waived` for a
            publish

            gate run whose regression was waived; `error` when the run cannot

            complete: at once for a permanent failure (the instance, its

            archived configuration, a stored scenario, or the eval executor is

            unavailable or unreadable), or after 5 attempts when execution

            failed transiently each time.
        triggered_by:
          type: string
          enum:
            - publish
            - manual
            - schedule
            - event
        version:
          type: string
          description: >-
            The released instance version the run measured, as an integer string
            (for example `"4"`). Absent when not recorded.
        pass_count:
          type: integer
        fail_count:
          type: integer
        error:
          type: string
          description: Why the run ended in `error`. Absent otherwise.
        created_at:
          type: string
          format: date-time
    Error:
      type: object
      required:
        - code
        - message
      properties:
        code:
          type: string
          description: >
            Stable snake_case error identifier. Safe to match programmatically.

            New codes are added over time; treat an unknown code by its HTTP
            status.


            Known codes include:

            `invalid_request`, `invalid_input`, `unauthorized`,

            `auth_unavailable`, `rate_limited`, `missing_run_id`,

            `invalid_run_id`, `run_not_found`, `invalid_offerings`,

            `missing_input`, `queue_unreachable`, `engine_not_found`,

            `engine_slug_conflict`, `engine_archived`, `instance_not_found`,

            `instance_slug_conflict`, `instance_archived`,

            `instance_follows_canonical`, `instance_not_published`,

            `invalid_engine_type`, `validation_error`, `webhook_not_found`,

            `webhook_limit_exceeded`, `invalid_url`, `invalid_events`,

            `invalid_description`, `invalid_auth_token`,

            `missing_endpoint_id`, `invalid_endpoint_id`,

            `missing_delivery_id`, `invalid_delivery_id`,

            `delivery_not_found`, `delivery_not_resendable`,

            `insufficient_capability`, `engine_version_required`,

            `engine_version_exact_required`, `engine_version_not_found`,

            `engine_version_invalid_format`,

            `engine_version_draft_requires_sandbox_key`,

            `engine_version_none_released`, `test_scenario_sandbox_only`,

            `test_scenario_forbidden`, `test_scenario_exact_version_required`,

            `test_scenario_version_not_supported`, `invalid_test_scenario`,

            `request_bound_exceeded`, `run_cap_exceeded`,

            `scoped_key_required`, `engine_binding_forbidden`,

            `request_timeout`, `internal_error`, `idempotency_key_reused`,

            `invalid_idempotency_key`, `acting_principal_mismatch`,

            `run_requires_acting_principal`, `environment_slug_invalid`,

            `environment_slug_reserved`, `environment_slug_taken`,

            `environment_name_invalid`, `environment_limit_reached`,

            `environment_live_immutable`, `environment_not_found`,

            `environment_in_use`, `environment_operation_failed`,

            `list_environments_failed`, `engine_release_unservable`,

            `engine_config_hash_inconsistent`,

            `engine_config_hash_unavailable`,

            `engine_config_version_unavailable`,

            `engine_version_resolve_failed`,

            `engine_config_version_load_failed`,

            `engine_version_publish_mismatch`,

            `engine_version_schema_change_requires_major`,

            `engine_config_hash_missing`, `engine_config_changed`,

            `invalid_engine_config`, `cold_start_gate_not_met`,

            `cold_start_gate_failed`, `cold_start_gate_regressed`,

            `cancel_run_failed`, `event_id_reused`, `invalid_event_id`,

            `missing_event_id`, `invalid_event_type`, `invalid_payload`,

            `reason_text_too_long`, `reason_taxonomy_version_unknown`,

            `invalid_rating`, `missing_comment`, `invalid_target`,

            `invalid_time_on_task`, `scoped_annotation_required`,

            `missing_instance_slug`, `invalid_instance_config`,

            `invalid_eval_waiver`, `instance_config_hash_missing`,

            `config_changed_during_publish`,

            `scenario_set_changed_during_publish`,

            `conversation_eval_regressed`,

            `conversation_eval_execution_failed`,

            `conversation_eval_gate_unavailable`, `publish_failed`,

            `conversation_not_found`, `conversation_archived`,

            `invalid_cursor`, `message_not_found`, `invalid_turn_request`,

            `message_too_long`, `invalid_tool_result`, `invalid_confirmation`,

            `invalid_edit_target`, `turn_in_progress`, `pending_turn`,

            `turn_state_conflict`, `confirmation_environment_unpinned`,

            `instance_config_missing`, `instance_config_invalid`,

            `instance_model_unavailable`, `guardrail_evaluation_failed`,

            `guardrail_config_invalid`, `invalid_conversation_history`,

            `streaming_unsupported`, `converse_unavailable`, `provider_error`,

            `provider_unavailable`, `unknown_model`, `too_many_messages`,

            `too_many_tools`, `invalid_max_tokens`, `invalid_message_role`,

            `invalid_message`, `invalid_content_block`, `invalid_tool`,

            `attachments_not_enabled`, `attachments_unavailable`,

            `attachment_rejected`, `attachment_too_large`,

            `attachments_too_large`, `too_many_attachments`,

            `attachment_not_accepted`, `attachment_not_found`,

            `attachment_changed`, `attachment_read_failed`,

            `attachment_member_delete`, `attachment_not_reservable`,

            `invalid_offset`, `invalid_turn_id`, `invalid_comment`,

            `invalid_reason`, `invalid_feedback_key`, `turn_not_found`,

            `annotation_not_found`, `annotation_not_promotable`,

            `invalid_eval_scenario`, `conversation_eval_scenario_not_found`,

            `conversation_eval_scenario_cap`,

            `conversation_eval_run_not_found`,

            `conversation_eval_baseline_not_found`,

            `conversation_eval_unavailable`,

            `instance_version_invalid_format`, `instance_version_not_found`,

            `document_class_not_open`, `presign_failed`,

            `markets_directory_unavailable`, `book_unavailable`,

            `book_connection_ambiguous`, `identity_unmapped`,

            `identity_needs_review`, `identity_suspended`, `identity_stale`,

            `scope_unavailable`, `assertion_invalid`, `assertion_stale`,

            `cursor_filter_mismatch`, `cursor_expired`,

            `action_schema_unknown`, `action_payload_invalid`,

            `action_out_of_scope`, `action_list_too_large`,

            `overlay_read_only`, `unsupported_node_type`,

            `acting_principal_required`, `action_denied`,

            `appetite_read_denied`, `approval_required`,

            `catalog_access_denied`, `catalog_connection_ambiguous`,

            `create_webhook_failed`, `dataset_denied`, `delivery_id_reused`,

            `document_type_not_servable`, `document_unclassified`,

            `egress_manifest_version_mismatch`,

            `egress_manifest_version_required`, `event_type_not_allowed`,

            `event_type_reserved`, `execution_confirm_required`,

            `execution_prohibited`, `ingest_source_disabled`,

            `ingest_source_misconfigured`, `ingest_source_not_found`,

            `invalid_acting_assertion`, `invalid_active`,

            `invalid_credentials`, `invalid_payload_mode`,

            `invalid_signature`, `load_run_failed`, `load_solutions_failed`,

            `load_work_items_failed`, `missing_engine_slug`, `not_found`,

            `object_store_unconfigured`, `provider_not_approved`,

            `resolve_retry_exhausted`, `run_lookup_failed`,

            `service_identity_unknown`, `stale_timestamp`, `surface_denied`,

            `unresolved_market_question`, `unsupported_market_filter`.


            The full list with causes and fixes is at

            https://docs.usenexio.com/reference/errors.
        message:
          type: string
          description: Human-readable error message. May change between versions.
        details:
          description: |
            Optional request-specific details. Request-bound failures use the
            `RequestBoundDetails` object. Validation failures may use an array
            of field issues or another documented object.

            `409 environment_in_use` is the one exception to this envelope: it
            carries a top-level `blockers` object (see `EnvironmentInUseError`)
            instead of `details`.
  responses:
    Unauthorized:
      description: Missing or invalid API key.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
          example:
            code: unauthorized
            message: Missing or invalid API key
    RateLimited:
      description: Rate limit exceeded. Retry after the window resets.
      headers:
        Retry-After:
          $ref: '#/components/headers/RetryAfter'
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
          example:
            code: rate_limited
            message: Rate limit exceeded
    RequestTimeout:
      description: >-
        `request_timeout`: the 30 second route timeout expired before the
        handler answered.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
          example:
            code: request_timeout
            message: Request timed out
  headers:
    RetryAfter:
      description: Whole seconds to wait before retrying, at least 1.
      schema:
        type: integer
        minimum: 1
  securitySchemes:
    BearerAuth:
      type: http
      scheme: bearer
      description: >
        Send the key as `Authorization: Bearer <key>`. Two kinds of key exist.


        Organization keys are issued in the portal (Settings, then API keys),

        each bound to one environment, shaped `nx_<environment slug>_<64 hex>`.
        They carry

        no capabilities and pass every capability check, with one exception:

        routes under /api/v1/records, /api/v1/engines/{id}/opportunities,

        /api/v1/graph and /api/v1/catalog/documents accept an

        organization key only when its environment is `live`, and refuse any

        other with 403 `scoped_key_required`. Revocation takes effect within 60

        seconds.


        Scoped keys are issued by Nexio on request, shaped

        `nxsk_v1_<24 hex key id>_<43 character secret>`. Each is bound to one

        org, one environment, a set of engines and a set of capabilities. A

        malformed, unknown or revoked `nxsk_` key fails with 401 and is never

        retried as an organization key. Revocation takes effect on the next

        request. A scoped key without a route's capability gets 403

        `insufficient_capability`; a scoped key not bound to the engine gets 403

        `engine_binding_forbidden`.


        Key-grantable capabilities: `engines:read`, `runs:write`, `runs:read`,

        `runs:defensibility:read`, `runs:test`, `catalog:read`,

        `catalog:documents:read`, `webhooks:manage`, `conversations:use`,

        `conversations:export`, `records:read`, `records:opportunities:run`,

        `actions:write`, `actions:read`, `graph:read`, `records:analyze`.


        Routes that accept organization keys only (every scoped key gets 403

        `insufficient_capability`): environment management, engine create,

        update, configuration and publish, and conversation instance

        authoring. Each operation description names the capability a scoped

        key needs.

````