Scenarios

A Scenario is a plain-English description of one expected behaviour, scoped to a Project. It carries the user_prompt the agent receives and the expected_behavior the judge grades against. Scenarios live under a Project and can carry tags.

Add and edit Scenarios in the dashboard

A Developer or Admin adds a Scenario from the dashboard: Add a Scenario opens a form for its Project, name, user prompt, expected behaviour and tags. Create saves it and opens the Project, where the Scenarios list shows it. A Viewer cannot add or edit. This is the no-CLI way to give a Project Scenarios for Run on ProbeRun. Deleting a Scenario asks first. The Project page links each Scenario to its page, and the Scenario page’s Back link returns to its Project.

How a Run passes, on the same form when creating or editing, is one choice:

• A model judge decides: no assertion set. A model reads the expected behaviour and grades each Run. It uses credits (or your own model key), and the Scenario cannot be monitored nightly.
• Checks decide: rows, each one exact rule: Calls these tools (any order, other tools allowed), Calls these tools in this order, Calls no tools but these, The conversation contains, The conversation never contains. Tool names are comma-separated. Under the rows the form says in words what a Run will check. Checks are free, give the same answer every time, and are what Monitoring needs.

Edit as JSON opens the same assertion set as text. A set that uses keys the rows cannot show (for example must_modify or a tool call with arguments) opens as JSON and is saved exactly as it was unless you change it.

Opening a Scenario edits it. Save replaces every field, including the assertion set in the editor beside the form; clear the editor (or leave {}) to save without checks. A Scenario stays in its Project. When a save is refused the form shows the API’s reason and keeps what you typed: for example a name another live Scenario in the Project already has, or any edit in the Example Project, which is read-only.

Renaming keeps every Run with the Scenario. Pushes match Scenarios by name, though, so a push that still uses the old name no longer reaches it: a results-only push reports the name as unknown, and a bundle that declares the Scenario creates a new one under the old name. The form says so before you save a rename. An assertion-set edit changes how future Runs are graded; Verdicts already recorded keep their Judge and do not change. Every saved edit is recorded in the audit log as scenario.updated, with the fields that changed.

Assertions, and previewing them before you commit

A Scenario may also carry an assertion set: a short structural contract the deterministic judge grades without any model call — which files must or must not change, what the Trajectory must or must not contain, an expected tool-call path, pinned Policies. The Scenario page has an assertion editor whose output panel is locked to a preview: it shows exactly which bytes were graded (the Run or recording you chose, never an implicit “latest”), one line per assertion, and a Score as passed / counted beside a count of assertions that could not be evaluated. A preview writes nothing. An assertion set must contain at least one check — a vacuous “pass” is refused at authoring time.

Bringing a trace in instead of writing a Scenario

If your agent is already traced, probe import --source <format> --input <export.json> turns one exported trace into a reviewable recording plus a drafted Scenario, with no server and no database. Three formats are read, and you name yours: openinference (OpenInference over collector OTLP/JSON), genai (the OpenTelemetry GenAI semantic conventions over the same envelope, as OpenTelemetry’s OpenAI instrumentation and OpenLLMetry write them), and langfuse (one trace saved from Langfuse’s public API, no collector). Anything the trace could not evidence is written into the recording as a named gap rather than filled in: a GenAI trace recorded with message content off keeps its model calls as empty Steps and drafts no prompt. A trace that does not show the run finished is refused unless you say so explicitly, and the override is recorded where a reviewer will see it; Langfuse records no success status, so a Langfuse import always needs that override. How to produce each export is in Import a trace from OpenTelemetry GenAI, Phoenix or Langfuse.

In the dashboard, Add a Scenario offers Start from a trace: paste one OpenInference export (OTLP JSON, or JSON lines from a collector, up to 5 MB) or choose the file, and press Draft from trace. ProbeRun fills the name and the user prompt the trace records and saves nothing. The expected behaviour stays empty for you to write: a trace shows what happened, not what should. The tools the agent called are shown as a suggested path with its mode, and become a check only when you press Use as a check; one trace is one path the agent took, not the path it must always take. An export holding several traces asks which one; a run the trace does not show finishing is flagged. The same drafting as probe import runs behind it (POST /scenarios/draft-from-trace, Developer or Admin).

The drafted tool path

probe import and probe capture both end the drafted scenario.yaml with a commented-out expected_path: the tools the recording called, as a contract you can accept. It is YAML comments, so no run, gate or --write-config sees it until you review each call and delete the leading # (the block and the path_tolerance: line under it). probe capture previews it as “expected_path draft (not active)” and prints the line numbers to uncomment; after an import, probe preview --against fixture:<dir> checks it once accepted. Tool names only, never arguments: those carry ids and timestamps that change every run.

--path-tolerance picks the draft’s strictness:

--path-tolerance	Drafted mode	What it lists
actions (default)	in_order	calls other than read-only lookups (Read, Glob, Grep, LS, NotebookRead, WebFetch, WebSearch), in order; extra calls allowed
exact	exact	every call, in order, nothing else
any_order	any_order	every call, any order, nothing else
tool_set	contains_all	each distinct tool once, any order, extras allowed

A recording of only read-only lookups, or one over 25 entries, falls back to contains_all over the distinct tool names, and the draft’s header says why. No tool calls means no draft, and the output says so. An agent that only ran tools and changed no files still gets a scenario.yaml from probe capture, with the drafted path as its only (commented) assertion.

One limit to know: the accepted path fails only when the agent actually runs. A replayed candidate, the default probe gate candidate, replays the recording the draft came from and passes it every time; the path catches a change on live Runs (probe run --mode live, a drift sentinel’s live leg, managed monitoring).

Triage exported traces before you import them

When you have a directory of exports rather than one, probe triage --source openinference <files or directories> (a directory contributes the .json files directly inside it) ranks them by how much the agent struggled, so you import the worst first. It writes nothing, needs no server, and ends with the probe import command for each of the top flagged traces (--top, default 5).

By default it counts three struggle signals straight from the trace, with no model and no network:

• tool_loop — the same tool called with identical arguments three or more times (different arguments are not a loop).
• action_failure — a tool call whose span carries an error status.
• model_repeat — the same model output (20 characters or more) twice or more.

With --jev it also asks TypeSafe’s Jev four questions per trace: unresolved (was the request met by the end?), and — about the first prompt only — manipulation_attempt, abuse and spam. This sends the trace to TypeSafe under your TYPESAFE_API_KEY — the first user prompt, every model output, and every tool call’s arguments and results, which is where secrets and personal data usually sit in a trace. It says so before the first request; TypeSafe is then a subprocessor for that data, as it is for the jev compliance judge (see “Who judges plain-language Policies” below). Each answer lands in a band: flag at 0.7 or above, clear at 0.3 or below, uncertain in between — uncertain is shown, never rounded away. A correctly refused prompt injection or spam message is not counted as unresolved: that signal is reported as not applicable.

One file ProbeRun cannot read — not JSON, nested too deeply, or shaped in a way the reader does not expect — is listed as refused and the rest of the batch is still ranked. Symlinks inside a directory are not followed. There is no size cap on a single file, so a very large export is read into memory whole.

A signal that could not be checked is never reported as clear. Without --jev the four Jev signals are not checked; with it, a trace Jev refuses as too long, a service still unavailable after three attempts, or an answer that does not parse or comes from a model other than the pinned jev-1.13.0 becomes a named gap on that trace, and the rest of the batch is still scored. A key Jev rejects stops the run.

Exit codes match probe import: 0 every file was ranked (flags are not failures — triage ranks, it does not gate), 1 at least one file was refused the way probe import would refuse it, 2 a usage or configuration error. A batch larger than --limit (default 1000) is refused whole rather than silently cut.

Limits: triage scores what the agent did. An imported trace keeps only the first user message, so user-side friction across later turns (repeating themselves, getting frustrated) is not scored. Jev’s values can differ slightly between identical runs (0.46 and 0.42 for the same trace on 2026-09-28), so treat a value near a band edge as a reason to read the trace, not as a measurement. Jev reads the trace itself, so text written into a trace can steer its answers; the structural signals cannot be steered that way. The suggested probe import commands are shell-quoted, and names from a trace are printed with control and invisible formatting characters removed.