> ## Documentation Index
> Fetch the complete documentation index at: https://docs.retab.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Workflow Evals API

> API contract for workflow eval definitions, async eval runs, and per-eval result rows

Workflow evals let you freeze inputs for one workflow block and assert something
about that block's output on a later run. This page documents the API shapes for
eval definitions, assertions, parent eval runs, and result rows. For the
conceptual workflow and dashboard flow, see [Evals](/workflows/evals).

## Eval Definition

A `WorkflowEval` is a saved assertion plus the inputs needed to replay one
block.

```json theme={null}
{
  "id": "wfnodeeval_...",
  "workflow_id": "wf_...",
  "target": { "type": "block", "block_id": "block_extract_invoice" },
  "source": {
    "type": "manual",
    "handle_inputs": {
      "input-json-0": { "type": "json", "data": { "invoice_id": "INV-001" } }
    }
  },
  "name": "invoice total",
  "assertion": {
    "id": "assert_xyz",
    "target": { "output_handle_id": "output-json-0", "path": "total" },
    "condition": { "kind": "equals", "expected": 1234.56 },
    "label": null
  },
  "schema_drift": "none",
  "assertion_drift_status": "valid",
  "freshness": { "status": "fresh", "reasons": [] },
  "latest_run_summary": null,
  "latest_passing_run_summary": null,
  "latest_failing_run_summary": null
}
```

### `target`

`target` is a discriminated union by `type`.

| `type`  | Fields     | Meaning                                   |
| ------- | ---------- | ----------------------------------------- |
| `block` | `block_id` | Replay one block in the current workflow. |

The API shape is a union so workflow-level targets can be added later without
renaming the field.

### `source`

`source` is also a discriminated union by `type`.

| `type`     | Fields                       | Meaning                                                                                          |
| ---------- | ---------------------------- | ------------------------------------------------------------------------------------------------ |
| `manual`   | `handle_inputs`              | Explicit handle inputs supplied by the caller.                                                   |
| `run_step` | `run_id`, optional `step_id` | Capture the inputs a block received during a previous workflow run. Use `step_id` for loop rows. |

Manual `handle_inputs` values are typed:

```json theme={null}
{
  "input-json-0": { "type": "json", "data": { "amount": 1234.56 } },
  "input-document-0": {
    "type": "file",
    "document": {
      "id": "file_invoice_q1",
      "filename": "invoice.pdf",
      "mime_type": "application/pdf"
    }
  }
}
```

When an eval is created from a `run_step`, Retab snapshots the inputs at create
time. File handles are materialized as durable Retab file references so later
eval runs do not depend on the original upload session.

### `assertion`

Workflow evals use one assertion per eval. An assertion targets one output
handle and an optional path inside that handle's payload.

```json theme={null}
{
  "target": { "output_handle_id": "output-json-0", "path": "vendor.name" },
  "condition": { "kind": "contains", "expected": "Acme" },
  "label": "vendor includes Acme"
}
```

The current `condition.kind` values are:

| Kind                                  | Required fields                                       |
| ------------------------------------- | ----------------------------------------------------- |
| `exists`, `not_exists`                | none                                                  |
| `equals`, `not_equals`                | `expected`                                            |
| `contains`, `not_contains`            | `expected`                                            |
| `number_compare`                      | `op`, `expected`                                      |
| `between`                             | `lower`, `upper`, optional `inclusive`                |
| `starts_with`, `ends_with`            | `expected`                                            |
| `matches_regex`                       | `pattern`                                             |
| `object_contains`                     | `expected` object                                     |
| `array_contains`                      | `expected` object                                     |
| `length_compare`                      | `op`, `expected`                                      |
| `json_schema_valid`                   | `schema`                                              |
| `all_items_match`, `any_item_matches` | nested `condition`                                    |
| `similarity_gte`                      | `reference`, `threshold`, optional `method`           |
| `llm_judged_as`, `llm_not_judged_as`  | `rubric`, optional `expected_label` (default `"yes"`) |
| `split_iou_gte`                       | `expected`, optional `threshold`                      |

`number_compare` and `length_compare` support `op` values `gt`, `gte`, `lt`,
`lte`, `eq`, and `neq`.

For `llm_judged_as` and `llm_not_judged_as`, `expected_label` defaults to
`"yes"` and the judge is instructed to answer `"yes"`/`"no"` for a yes/no
rubric. Set it only when the rubric returns some other vocabulary, and only to a
label that rubric can actually produce — an unreachable `expected_label` makes
`llm_judged_as` always fail and `llm_not_judged_as` always pass. See
[Evals](/workflows/evals#choosing-the-llm-judge-label).

## Runs

Create a parent eval run with:

```http theme={null}
POST /v1/workflows/evals/runs
```

The request body requires `workflow_id`. `scope` is optional:

```json theme={null}
{
  "workflow_id": "wf_abc123xyz",
  "scope": { "type": "single", "eval_id": "wfnodeeval_hsLEQiM61ez9Piv147MWk" }
}
```

Scope variants:

| `scope.type` | Fields     | Meaning                               |
| ------------ | ---------- | ------------------------------------- |
| omitted/null | none       | Run every saved eval in the workflow. |
| `workflow`   | none       | Run every saved eval in the workflow. |
| `block`      | `block_id` | Run every eval for one block.         |
| `single`     | `eval_id`  | Run one saved eval.                   |

A `WorkflowEvalRun` is the parent resource for one batch:

```json theme={null}
{
  "id": "wfevalrun_q1z2",
  "workflow_id": "wf_abc123xyz",
  "workflow_version_id": "draft_2026_05_18",
  "trigger": { "type": "api" },
  "lifecycle": { "status": "pending" },
  "timing": {
    "created_at": "2026-05-18T10:00:00Z",
    "started_at": null,
    "completed_at": null,
    "duration_ms": null
  },
  "target": { "type": "block", "block_id": "block_extract_invoice" },
  "eval_id": "wfnodeeval_hsLEQiM61ez9Piv147MWk",
  "total_evals": 1,
  "counts": {
    "lifecycle_counts": {
      "pending": 1,
      "queued": 0,
      "running": 0,
      "completed": 0,
      "error": 0,
      "cancelled": 0
    },
    "outcome": {
      "passed": 0,
      "failed": 0,
      "blocked": 0
    }
  }
}
```

Run lifecycle values are `pending`, `queued`, `running`, `completed`, `error`,
and `cancelled`.

## Results

After a parent run finishes, list child rows with:

```http theme={null}
GET /v1/workflows/evals/results?run_id={run_id}
```

Each `WorkflowEvalResult` is the immutable record for one eval execution:

```json theme={null}
{
  "id": "wfresult_a",
  "workflow_eval_run_id": "wfevalrun_q1z2",
  "eval_id": "wfnodeeval_a",
  "workflow_id": "wf_abc123xyz",
  "block_id": "block_extract_invoice",
  "block_type": "extract",
  "lifecycle": { "status": "completed" },
  "timing": {
    "created_at": "2026-05-18T10:00:00Z",
    "started_at": "2026-05-18T10:00:02Z",
    "completed_at": "2026-05-18T10:00:20Z",
    "duration_ms": 18221
  },
  "handle_inputs": {
    "input-document-0": {
      "type": "file",
      "document": {
        "id": "file_invoice_q1",
        "filename": "invoice.pdf",
        "mime_type": "application/pdf"
      }
    }
  },
  "handle_outputs": {
    "output-json-0": {
      "type": "json",
      "data": { "total": 1234.56, "vendor": { "name": "Acme Inc" } }
    }
  },
  "assertion_result": {
    "assertion_id": "assert_xyz",
    "condition_kind": "equals",
    "outcome": "passed",
    "actual_value": 1234.56,
    "expected_value": 1234.56,
    "failure": null
  },
  "verdict": "passed",
  "verdict_summary": {
    "passed": true,
    "assertions_passed": 1,
    "assertions_failed": 0,
    "blocked_assertions": 0,
    "failed_assertion_ids": []
  }
}
```

Result lifecycle values match parent run lifecycle values. `verdict` and
`assertion_result.outcome` are assertion outcomes: `passed`, `failed`, or
`blocked`. Execution errors are represented in `lifecycle.status = "error"` and
the lifecycle error details, not as a fourth verdict value.

## Freshness and Drift

Read responses recompute drift against the current workflow draft.

| Field                    | Values                                  | Meaning                                      |
| ------------------------ | --------------------------------------- | -------------------------------------------- |
| `schema_drift`           | `none`, `partial`, `drifted`, `unknown` | Whether the assertion target still resolves. |
| `assertion_drift_status` | `valid`, `drifted`, `broken`            | Whether the saved assertion is still usable. |
| `freshness.status`       | `fresh`, `stale`, `unknown`             | Whether the latest run matches the baseline. |
| `drift.status`           | `none`, `drifted`, `broken`, `unknown`  | Artifact-level drift summary.                |

Latest-run summaries expose lifecycle and outcome separately:

```json theme={null}
{
  "run_record_id": "wfresult_a",
  "status": "completed",
  "outcome": "passed",
  "started_at": "2026-05-18T10:00:02Z",
  "completed_at": "2026-05-18T10:00:20Z",
  "assertions_passed": 1,
  "assertions_failed": 0,
  "blocked_assertions": 0
}
```

## Endpoint Map

| Method   | Path                                            | Purpose                                                     |
| -------- | ----------------------------------------------- | ----------------------------------------------------------- |
| `POST`   | `/v1/workflows/evals`                           | [Create](/api-reference/workflows/evals/create)             |
| `GET`    | `/v1/workflows/evals?workflow_id={workflow_id}` | [List](/api-reference/workflows/evals/list)                 |
| `GET`    | `/v1/workflows/evals/{eval_id}`                 | [Get](/api-reference/workflows/evals/get)                   |
| `PATCH`  | `/v1/workflows/evals/{eval_id}`                 | [Update](/api-reference/workflows/evals/update)             |
| `DELETE` | `/v1/workflows/evals/{eval_id}`                 | [Delete](/api-reference/workflows/evals/delete)             |
| `POST`   | `/v1/workflows/evals/runs`                      | [Create Run](/api-reference/workflows/evals/runs/create)    |
| `GET`    | `/v1/workflows/evals/runs`                      | [List Runs](/api-reference/workflows/evals/runs/list)       |
| `GET`    | `/v1/workflows/evals/runs/{run_id}`             | [Get Run](/api-reference/workflows/evals/runs/get)          |
| `POST`   | `/v1/workflows/evals/runs/{run_id}/cancel`      | [Cancel Run](/api-reference/workflows/evals/runs/cancel)    |
| `GET`    | `/v1/workflows/evals/results?run_id={run_id}`   | [List Results](/api-reference/workflows/evals/results/list) |
| `GET`    | `/v1/workflows/evals/results/{result_id}`       | [Get Result](/api-reference/workflows/evals/results/get)    |

## MCP Tools

The eval API is also exposed through MCP tools:

* `workflows_evals_create`
* `workflows_evals_list`
* `workflows_evals_get`
* `workflows_evals_update`
* `workflows_evals_delete`
* `workflows_evals_runs_create`
* `workflows_evals_runs_get`
* `workflows_evals_results_list`
* `workflows_evals_results_get`

The tool input schemas match the request bodies above. See the
[MCP](/workflows/MCP) page for setup.