> ## Documentation Index
> Fetch the complete documentation index at: https://docs.retab.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> Validate workflow block outputs with reusable input snapshots and assertions

## What are Workflow Evals?

Workflow evals are saved, block-level regression checks. Each eval freezes the
inputs for one workflow block, replays that block against the current workflow
draft, and evaluates one assertion against the block output.

Use evals when you want to answer questions like:

* Does this Extract block still return the expected invoice total?
* Does this Function block still compute the validation flag correctly?
* Does this Split block still assign pages to the right subdocuments?
* Does this Classifier block still route a file to the expected category?

Evals are intentionally block-scoped. Running an eval executes the selected
block with saved inputs instead of replaying the entire workflow, which keeps
the feedback loop short while you adjust schemas, prompts, function code,
categories, or split definitions.

## Supported Blocks

Workflow evals are currently supported for:

| Block          | What you usually assert                                     |
| -------------- | ----------------------------------------------------------- |
| **Extract**    | Extracted JSON fields                                       |
| **Function**   | Returned JSON fields from the function output schema        |
| **Split**      | Split manifest quality or produced subdocument file handles |
| **Classifier** | Produced category file handles                              |

Other block types, such as input blocks, notes, API calls, conditionals, review
gates, and loops, do not currently produce workflow eval runs directly.

## Eval Shape

A workflow eval stores:

1. **Target** - the block to run, currently `{ "type": "block", "block_id": "..." }`.
2. **Source** - the handle inputs to replay.
3. **Assertion** - one expected condition for one declared output handle.

```json theme={null}
{
  "target": { "type": "block", "block_id": "block_extract_invoice" },
  "source": {
    "type": "manual",
    "handle_inputs": {
      "input-document-0": {
        "type": "file",
        "document": {
          "id": "file_invoice_q1",
          "filename": "invoice.pdf",
          "mime_type": "application/pdf"
        }
      }
    }
  },
  "assertion": {
    "target": { "output_handle_id": "output-json-0", "path": "total" },
    "condition": { "kind": "equals", "expected": 1234.56 }
  }
}
```

`source.type` can be:

| Source     | Fields                       | Use it when                                                    |
| ---------- | ---------------------------- | -------------------------------------------------------------- |
| `manual`   | `handle_inputs`              | You want to provide explicit JSON or file inputs.              |
| `run_step` | `run_id`, optional `step_id` | You want to capture the inputs a block received in a real run. |

For blocks executed inside a loop, provide `step_id` so Retab knows which
iteration's inputs to capture. File inputs are materialized as durable Retab
file references so the eval does not depend on the original browser upload
session.

## Assertions

An assertion targets a declared output handle and, optionally, a dotted path
inside that handle's JSON payload.

| Block          | Output target examples                                  |
| -------------- | ------------------------------------------------------- |
| **Extract**    | `output-json-0.total`, `output-json-0.vendor.name`      |
| **Function**   | `output-json-0.is_valid`, `output-json-0.error_message` |
| **Split**      | `output-json-splits` or a subdocument file handle       |
| **Classifier** | A category file handle                                  |

The current condition kinds are:

| Kind                                        | Use for                                             |
| ------------------------------------------- | --------------------------------------------------- |
| `exists`, `not_exists`                      | Presence checks.                                    |
| `equals`, `not_equals`                      | Strict value equality or inequality.                |
| `contains`, `not_contains`                  | Substring or list membership checks.                |
| `number_compare`, `between`                 | Numeric comparisons.                                |
| `starts_with`, `ends_with`, `matches_regex` | String pattern checks.                              |
| `object_contains`, `array_contains`         | Subset checks for objects and arrays of objects.    |
| `length_compare`                            | Length checks for strings, arrays, or objects.      |
| `json_schema_valid`                         | JSON Schema validation for a target subtree.        |
| `all_items_match`, `any_item_matches`       | Nested assertions over array items.                 |
| `similarity_gte`                            | Similarity thresholds.                              |
| `llm_judged_as`, `llm_not_judged_as`        | Rubric-based LLM judging.                           |
| `split_iou_gte`                             | Intersection-over-Union for split page assignments. |

### Condition payloads

Each condition is a `{ "kind": ..., ... }` object. The fields alongside `kind`
depend on the kind — most value checks use `expected`, but the comparison,
pattern, metric, and nested kinds use their own fields:

| Kind                                  | Fields alongside `kind`                                                        |
| ------------------------------------- | ------------------------------------------------------------------------------ |
| `exists`, `not_exists`                | *(none)*                                                                       |
| `equals`, `not_equals`                | `expected` (any JSON value)                                                    |
| `contains`, `not_contains`            | `expected` (substring, or an element of a list)                                |
| `object_contains`                     | `expected` (non-empty object subset)                                           |
| `array_contains`                      | `expected` (non-empty object subset matched against array items)               |
| `number_compare`                      | `op` (`gt`/`gte`/`lt`/`lte`/`eq`/`neq`) + `expected` (number)                  |
| `length_compare`                      | `op` (same values) + `expected` (integer)                                      |
| `between`                             | `lower`, `upper` (numbers) + `inclusive` (bool)                                |
| `starts_with`, `ends_with`            | `expected` (string)                                                            |
| `matches_regex`                       | `pattern` (string)                                                             |
| `json_schema_valid`                   | `schema` (a JSON Schema object)                                                |
| `all_items_match`, `any_item_matches` | `condition` (a nested `{ "kind": ..., ... }` condition)                        |
| `similarity_gte`                      | `reference` (value) + `threshold` (0.0–1.0) + `method` (default `levenshtein`) |
| `split_iou_gte`                       | `expected` (`{ "splits": [...] }`) + `threshold` (0.0–1.0)                     |
| `llm_judged_as`, `llm_not_judged_as`  | `rubric` (string) + optional `expected_label` (string, default `"yes"`)        |

Examples:

```json theme={null}
{ "kind": "number_compare", "op": "gt", "expected": 3 }
{ "kind": "contains", "expected": "STAGING" }
{ "kind": "starts_with", "expected": "INV-" }
{ "kind": "between", "lower": 100, "upper": 500, "inclusive": true }
{ "kind": "matches_regex", "pattern": "^INV-[0-9]{6}$" }
{ "kind": "similarity_gte", "reference": "ACME Corp", "threshold": 0.9 }
{ "kind": "llm_judged_as", "rubric": "The summary mentions the invoice total." }
```

### Choosing the LLM judge label

The judge applies your `rubric` to the value and returns a single label, which
the assertion compares against `expected_label` (case-insensitive).

Leave `expected_label` out for a yes/no rubric — it defaults to `"yes"`, and the
judge is instructed to answer `"yes"` or `"no"`. Only set it when your rubric
asks for a label from some other vocabulary, and set it to a label that rubric
can actually return:

```json theme={null}
{ "kind": "llm_judged_as", "rubric": "Is this text an all-uppercase greeting?" }
{
  "kind": "llm_judged_as",
  "rubric": "Classify this document as invoice, receipt, or statement.",
  "expected_label": "invoice"
}
```

<Warning>
  An `expected_label` the rubric never produces makes the assertion useless, and
  the two kinds fail in opposite directions. A yes/no rubric with
  `"expected_label": "pass"` can only ever return `"yes"` or `"no"`, so
  `llm_judged_as` always fails — noisy, but you notice it. `llm_not_judged_as`
  always **passes**, because the label never matches and the kind inverts the
  match. That is a green eval that can never catch a regression. If an
  `llm_not_judged_as` assertion has never failed, check that its
  `expected_label` is a label the rubric can return.
</Warning>

<Note>
  In the CLI, the inline `--path`/`--equals` flags build an `equals` assertion
  only. To use any other condition kind, pass the full assertion JSON with
  `--assertion-file` (or `--assertion-file -` to read from stdin).
</Note>

## Running Evals

You can run one eval, every eval for one block, or every eval in a workflow. In
the API, create a parent eval run with:

```http theme={null}
POST /v1/workflows/evals/runs
```

The request body always includes `workflow_id`. `scope` is optional:

| Scope                    | Meaning                               |
| ------------------------ | ------------------------------------- |
| omitted or `null`        | Run every saved eval in the workflow. |
| `{ "type": "workflow" }` | Run every saved eval in the workflow. |
| `{ "type": "block" }`    | Run every saved eval for one block.   |
| `{ "type": "single" }`   | Run one saved eval by `eval_id`.      |

When an eval run starts, Retab:

1. Loads the current workflow draft and current block configuration.
2. Rebuilds the saved handle inputs into normal runtime inputs.
3. Executes only the selected block.
4. Stores the block artifact, handle outputs, routing decisions, warnings, and
   timing.
5. Resolves the assertion target from the handle outputs.
6. Records the assertion outcome and the result verdict.

Eval runs are asynchronous. Poll
`GET /v1/workflows/evals/runs/{run_id}` until the parent run reaches a terminal
lifecycle, then read child rows from
`GET /v1/workflows/evals/results?run_id={run_id}`.

## Results

A parent `WorkflowEvalRun` has a lifecycle:

| Lifecycle   | Meaning                                            |
| ----------- | -------------------------------------------------- |
| `pending`   | The run was created but execution has not started. |
| `queued`    | The run is waiting for a worker.                   |
| `running`   | One or more eval results are executing.            |
| `completed` | The run finished; inspect outcome counts or rows.  |
| `error`     | The run failed before normal completion.           |
| `cancelled` | The run was cancelled.                             |

Parent `counts` separates lifecycle from assertion outcomes:

```json theme={null}
{
  "lifecycle_counts": {
    "pending": 0,
    "queued": 0,
    "running": 0,
    "completed": 1,
    "error": 0,
    "cancelled": 0
  },
  "outcome": {
    "passed": 1,
    "failed": 0,
    "blocked": 0
  }
}
```

Each `WorkflowEvalResult` row has its own lifecycle and, once completed, a
`verdict` of `passed`, `failed`, or `blocked`. The nested
`assertion_result.outcome` uses the same three outcome values and includes
`actual_value`, `expected_value`, optional score/threshold fields, and failure
details when the assertion cannot pass.

Result rows also include the saved `handle_inputs`, produced `handle_outputs`,
workflow/block fingerprints, the execution artifact, routing decisions,
warnings, and timing.

## Freshness and Drift

Workflow evals are tied to the block inputs and output schema that existed when
the eval was created or last updated. When the workflow draft changes, Retab
reports several freshness signals:

| Field                    | Values                                  | Meaning                                      |
| ------------------------ | --------------------------------------- | -------------------------------------------- |
| `schema_drift`           | `none`, `partial`, `drifted`, `unknown` | Whether the assertion target still resolves. |
| `assertion_drift_status` | `valid`, `drifted`, `broken`            | Whether the saved assertion is still usable. |
| `freshness.status`       | `fresh`, `stale`, `unknown`             | Whether the latest run matches the baseline. |
| `drift.status`           | `none`, `drifted`, `broken`, `unknown`  | Artifact-level drift summary.                |

The eval also stores `latest_run_summary`, `latest_passing_run_summary`, and
`latest_failing_run_summary`. Each summary separates run lifecycle (`status`)
from assertion outcome (`outcome`).

Staleness does not automatically mean the workflow is broken. It means the eval
should be rerun or recaptured before you rely on its latest result.

## Recommended Workflow

1. Run the workflow on representative documents.
2. Open the Evals page and create an eval from a completed run, or create one
   with explicit manual inputs.
3. Pick the block output field or handle you want to protect.
4. Define one assertion for the expected behavior.
5. Run the eval after changing schemas, prompts, code, categories, or split
   definitions.
6. Use stale evals as a review queue before publishing workflow changes.

Evals work best when each one protects one behavior. Prefer multiple focused
evals over one broad assertion so failures point directly to the changed output.

## API Reference

| Action            | Endpoint                                                                     |
| ----------------- | ---------------------------------------------------------------------------- |
| Create/list       | [`/v1/workflows/evals`](/api-reference/workflows/evals/create)               |
| Get/update/delete | [`/v1/workflows/evals/{eval_id}`](/api-reference/workflows/evals/get)        |
| Run evals         | [`/v1/workflows/evals/runs`](/api-reference/workflows/evals/runs/create)     |
| Read results      | [`/v1/workflows/evals/results`](/api-reference/workflows/evals/results/list) |