Runs, Trajectories and Verdicts

A Run is one execution of a Scenario. It owns:

• a Trajectory: the ordered steps the agent took (user input, agent messages, tool calls, tool results), which is the unit ProbeRun scores;
• a Verdict: pass, fail, error or pending. Always categorical, never a number;
• a Score: the fraction of named sub-checks that passed, derived by counting, never asserted by the judge;
• the judge’s reasoning.

On the Run page the Trajectory is shown as the conversation the end user saw: their messages right-aligned in a box, the agent’s replies on the left. What the user did not see (tool calls and their results) sits between the replies as quiet “backstage” lines; open one for the arguments and the result, or switch Show backstage off to read the pure customer transcript. Raw JSON shows the underlying steps as a collapsible tree.

Above the transcript a compact strip shows the Trajectory’s shape, one square per step: an outlined square for a user message, a filled one for an agent step, a hairline for a backstage tool result. The first divergent step (on the diff page) carries a ring and a step with pinned findings carries a small dot; select any square to jump to that step. A one-line legend under the strip names each mark it is showing (user, agent, tool result, diverged, finding). Each backstage line leads with a preview of the call built from its arguments, such as lookup_order(order_id="5102"), and the step range it covers (steps 1-2), so the path reads without opening anything. A long call wraps onto a second line rather than being cut short; a very long result is cut at 400 characters and says so, with the full value under Raw JSON. The Run page leads with the Scenario name and the Verdict, then the Trajectory and its checks; calibration comes after the evidence.

When the structural judge graded the Run, each check that it decided by reading a particular step is pinned to that step on the transcript: a pass or fail token, the check’s name and its detail, on the row where the step appears, with the number of pinned findings in the trajectory heading. Only checks that located a step are pinned. A check with nothing to point at (a numeric bound, a needle found nowhere, or a workdir rule that held because no diff was recorded) is never pinned and stays in the Checks panel, which in turn notes “pinned to step N” for the checks that are. Runs graded before this existed simply show no pins.

Run on ProbeRun (the Project page’s batch action, formerly “Run all”) executes every Scenario as a batch on the server. It appears only when there is something to run against: a Project agent endpoint (below) or a server runner other than stub. Clicking it first shows what it will do: the endpoint it calls, how many Scenarios run, how many have no assertions (those are recorded as errors), and, when your plan bounds live work, the server Runs used this month with the reset date and the live jobs running now. Nothing runs until you confirm. A refusal, such as a plan limit, is shown in the API’s own words. A Run whose agent or judge failed to complete is an error, and the reason is recorded rather than left blank; the same is true when a spend cap stops a batch early. A Scenario with no assertion set says so in its error (“This Scenario has no assertion set”), not as a judge failure to retry.

A Project’s agent endpoint (ProbeRun Cloud)

On ProbeRun Cloud, an Admin points a Project at the agent to test in the Project page’s Agent tab: an endpoint URL and, optionally, the value of an Authorization header. Saving makes one test call with one of the Project’s Scenarios and stores nothing unless the endpoint answers with a trajectory, so a Project needs at least one Scenario first. Only public addresses are accepted. The header is stored encrypted and never shown again: the tab says only whether one is set and its last four characters, and leaving the field empty keeps it (pointing the endpoint at a different host needs the header again).

Run on ProbeRun then calls that endpoint once per Scenario, using the same request as the HTTP runner below, and grades each reply with the Scenario’s checks (the structural Judge), which uses no credits. A Scenario without checks is graded by a model judge instead: the Tenant’s own key, or ProbeRun’s hosted judge on credits, and the confirmation says which before anything runs. Each Run counts toward your plan’s server live Runs and the batch toward its concurrent live jobs. Developers see the endpoint’s host in the confirmation; only Admins can change it. Self-hosted servers point the whole server at an agent with RUNNER=http instead.

Each press of Run on ProbeRun is its own Execution Batch, named dashboard-<UTC time>-<suffix>, so its Runs appear in the Project’s Batches, its headline comparison and /compare like a pushed Batch. Runs to a target on a gated Project. Without a gate policy, Run on ProbeRun runs each Scenario once. Once a Batch of the Project has run under a gate: block, the dialog reads the run plan for main (see Target drop below) and states it in plain words before anything runs: “catch a 50-point drop 8 times in 10”, the total Runs this run spends, and the Scenarios the target does not cover or that have too little history (those run the gate’s default count). A Developer can pick a 60, 50 or 40-point target for this run; the plan is read again. Live Scenarios run their planned count and replay Scenarios once, and the Batch records the plan and the gate policy. A plan larger than the live Runs your plan has left this month is refused before it starts. When the plan cannot be read, the dialog says so and every Scenario runs the default count.

Run after every deploy. Once an endpoint is saved, the Agent tab shows a curl and a GitHub Actions step that call POST /runs/trigger/<project id> with an API key (X-API-Key), the deployed commit as commit_sha and a Batch id per deploy. Create the key under Team and API keys with the Developer role and keep it in your pipeline’s secrets as PROBE_API_KEY; the page never shows a key.

Running your own agent

A Run calls whatever the runner points at. To test an agent you host, in any language, set RUNNER=http and HTTP_RUNNER_URL: ProbeRun POSTs each Scenario’s prompt to that endpoint and reads the agent’s steps from the reply. The request body, the accepted reply shapes (a native Trajectory, or chat messages in the OpenAI, Anthropic or LangChain style) and a minimal server are in Connect your agent over HTTP. An agent on localhost also needs HTTP_RUNNER_ALLOW_PRIVATE=true, because by default ProbeRun calls only public addresses.

Budgets and cap stops

A budget: block in your probe.yaml sets a spend ceiling for an invocation (and, optionally, for any one Scenario), checked before each model call; calls already in flight can go over it, and the overshoot is reported. probe run --estimate projects the spend of a suite without making a call, creating a Run or starting a batch. When a cap is hit, the Scenarios it stopped are recorded as Runs with a stop reason — they are never silently skipped, never counted as drift, and never confused with a broken recording. Budgets are included in every plan and edition at no additional charge and are not gated by any paid plan or add-on.

Replays reuse the Judge’s Verdict

Replaying a recording makes no call to your agent. An LLM Judge (JUDGE=litellm or claude) is still paid the first time it judges a replayed Trajectory, but probe run keeps that pass or fail in .probe/judge-cache/ and reuses it on the next replay as long as nothing it was shown changed: the Trajectory, the Scenario’s name, prompt and expected behaviour, and the Judge’s kind, model and prompt template. A reused Verdict makes no Judge call and costs nothing, and the run says how many it reused (judge cache: 3 of 3 Verdict(s) reused ...). An error is never reused, live Runs are always judged, and probe run --no-judge-cache judges everything again. In CI, keep .probe/judge-cache/ between jobs to get the same saving there.

A replay of an edited agent fails the gate

A replay never runs your agent; it replays what the agent did when it was recorded. So probe capture also records a digest of the agent file (its frontmatter and system prompt), and every replay checks the agent file it loaded against it. If you edit the agent and replay an old recording, the Run’s determinism line says SUBJECT CHANGED, and probe diff / probe gate exit 1 with the gating condition subject_changed, even though every Verdict still passes: the gate refuses to call a change clean that it never ran. Since 2026-10-08 you rarely see that: probe run (without --mode) checks every replay Scenario’s agent file before it runs, and a Scenario whose agent changed runs live instead, so the gate grades your change. The output says so with a line per Scenario, ran live: <name> (agent changed since capture), $<spend> (or “spend not metered” without a budget:), and the PR summary repeats it. The gate fails with subject_changed only when live could not happen: you passed --no-live-on-change, set PROBE_NO_LIVE_ON_CHANGE=1 or an explicit --mode, the live runner failed for a reason outside the agent (missing CLI or credentials, a spend cap), in which case the Scenario is replayed and the line says why, or a budget: has no price for the live model, in which case it is not promoted and stderr says so. On the Hobby plan a live Run past the monthly limit is refused at probe push, which exits 1; do not ignore that exit, or pass --fail-on-dropped to probe gate, because the refused Scenario is otherwise only a dropped row. Fix a subject_changed gate by re-capturing (probe capture --force). Recordings captured before this check existed say subject not recorded and do not fail the gate until you re-capture them.

probe sentinel applies the same check to its replay leg. If a recording there was captured from an agent file that has since changed, the baseline is the old agent and a drift verdict against it means nothing, so the sentinel exits 2 (a malfunction, not drift) before the live leg spends anything, and its stderr names the stale Scenarios and says to re-capture them with probe capture --force. Recordings that say subject not recorded do not stop the sentinel.

Why an error Run errored

Every Run whose Verdict is error records an error cause, so a crash in the harness is never mistaken for a change in your agent (bundles pushed by a CLI older than this feature are stored without one):

• harness: the failure was outside the agent under test. The Judge crashed, the model provider failed when ProbeRun itself called it (with RUNNER=litellm: a connection failure, a rate limit, a provider 5xx, a rejected key or an unknown model route), or the runner was misconfigured (an unset or malformed HTTP_RUNNER_URL, an address ProbeRun refuses to call, a missing Fixture or recording).
• agent: the agent under test failed. Its endpoint timed out, could not be resolved or reached, or returned non-2xx, non-JSON or no steps; with RUNNER=litellm the model timed out, overflowed its context, refused on content policy, or returned an empty or malformed completion; its process crashed. Any other error is also agent, never harness.
• budget: a spend cap stopped the Run (it also carries its stop reason).

probe diff, probe gate and the comparison page label a harness error could not run. It still fails the gate. A Scenario that went from pass to error is still listed under “Newly failing” and still exits 1, whatever the cause, because a Scenario nobody checked must never read as passing. The label tells you what to do next: re-run just those Runs rather than debug the agent. One limit: every Judge error is harness, even when the agent’s output provoked it (a Trajectory long enough to overflow an LLM Judge’s context), so if a re-run fails the same way, look at the Trajectory. For example:

Newly failing (2):
  - checkout  (pass → error, 1.00 → -, could not run: harness)
  - refund  (pass → fail, 1.00 → 0.00)
  1 of these could not run (harness errors, not agent behaviour). Re-run just those Runs.

The cause is carried in the results file, the push bundle, the stored Run and each comparison row (after_error_cause in probe-compare.json). A comparison that is otherwise clean still names the compared Scenarios that could not run in the candidate. Runs without a cause (recorded before this existed, or pushed by an older CLI) read as failures.

What the probe CLI tells you about a run

Before probe run shows any result, its first line names the runner and the judge that produced the verdicts, for example runner: subagent (replay) · judge: structural. A runner that can replay recordings (the subagent runner, and the HTTP runner for Scenarios recorded with probe capture --runner http) is named with the mode its Scenarios ran in, after any --mode override: (replay), (live), or (replay and live) when they differ. If either one is the stub, which is what you get with no RUNNER or JUDGE configured, the line also says so for each side: a stub runner means no agent was called, a stub judge means the verdicts are synthetic. With either stub, each result row is then marked PASS* or FAIL*, and the JSON results carry "synthetic": true. A stub verdict shows that the wiring works. It says nothing about your agent. The mark survives probe push: the dashboard shows such a Run as pass* / fail* without its pass or fail colour, run detail says it is synthetic, and a comparison that includes synthetic Runs says so above its verdict. Runs the server executes with a stub runner or judge are marked the same way. probe doctor shows which runner and judge are selected and what each still needs, and it calls no model.

Evidence provenance. probe run --bundle records the runner kind (stub, claude, http, subagent, litellm) and the judge’s identity, and probe push sends both, so every pushed Run carries its runner kind beside its judge kind (the Run page and the Quick Diagnosis drawer show both). From the two, each Run has one of three provenance states:

• synthetic: a stub runner or a stub judge produced it. Its Verdict carries the * mark wherever it renders (runs table, Run page, drawer, comparison rows, Run diff, review queue; never on your own calibration label), and the Run’s provenance list leads with “No agent was called or no model judged this Run”, in the attention colour, ahead of a diverged replay.
• agent: a known runner and judge, neither a stub. Nothing extra is shown.
• unreported: the Run was pushed without runner or judge identity, which is every Run pushed before the server recorded runner identity (and a cap-stopped Run, which no judge graded). Its provenance list says whether an agent produced it is not known; it is never shown as an agent’s.

The comparison page names each side’s synthetic and unreported Run counts above its verdict. probe gate refuses a comparison with synthetic Runs on either side: it exits 2 with synthetic_evidence, naming the side and the count, and reports no verdict, since a stub’s pass or regression says nothing about your agent. --allow-synthetic gates anyway and warns. Unreported Runs only warn, so batches pushed before this keep gating. A drift-history night (from probe sentinel or managed monitoring) over a synthetic Run is recorded as a check-failed night with the reason synthetic_evidence, never as a quiet one. Provenance is reported by the key that pushes, like the Verdicts it travels with: it marks stub output honestly, it does not attest that a real agent ran.

Upgrade order. Deploy the server before the CLI that sends the runner kind. probe push (and each probe sentinel leg) exits 2 when the server does not echo the runner kind back, naming the server as too old to record runner identity, rather than report a push the server stored as unreported.

An ERROR row is followed by its reason on one line (the full text stays in the results JSON), so a refused address or an unparseable reply is visible without opening the file. probe doctor warns when HTTP_RUNNER_URL points at localhost or a private address without HTTP_RUNNER_ALLOW_PRIVATE=true, since every Run would refuse it. The starter probe.yaml that probe init writes gives each Scenario one deterministic check, so JUDGE=structural returns a real, model-free Verdict on it. probe diff warns when the two files were scored by a different runner or judge, or when only one side holds synthetic verdicts, so a stub baseline against a real run is not read as a regression.

probe.yaml rejects keys it does not know, at the top level and in each Scenario, instead of silently ignoring them: probe run exits 2 naming the file, the Scenario index, the bad key and, when one is close, the key you probably meant (unknown key 'asertion_set' (did you mean 'assertion_set'?)).

probe.yaml can name the runner and judge with top-level runner: and judge: keys (for example runner: http and judge: structural). The RUNNER / JUDGE environment variables, including a local .env, override the file, so CI can switch the runner without editing the checked-in file; with neither set, both are stub. A value probe does not know fails the load with a suggestion (unknown runner 'htp' (did you mean 'http'?)). Settings a runner needs, such as HTTP_RUNNER_URL, stay in the environment. probe doctor reports the runner and judge actually in effect. The server ignores these keys: hosted and self-hosted runs use the server’s environment.

probe diff lists a Scenario whose tool sequence changed under Path drift, with its before and after tool paths on separate lines. If that Scenario also flipped to failing, its row says (also newly failing).

When both files are bundles (probe run --bundle), probe diff also prints First divergence, right under the verdict line and before the per-class lists: for each Scenario whose Verdict or path changed, the Step where its two Runs first differ, by role and tool (step 3 agent escalate_to_human → agent issue_refund), read from the Run that stands for the Scenario on each side. A Scenario whose steps match says “only the result changed”. Run sets (--runs N) are left out, since their outcome is a pass rate, not one pair of Runs. Plain results files carry no Trajectory, so the section is skipped; the exit code and the --compare-json document never change. examples/flip-demo/ holds a ready pair, and probe init --example writes the same pair (shipped inside the package) to flip-demo/ with the probe diff command to run on it; it refuses to overwrite an existing demo unless --force is given.

For scripts, probe run, diff, gate, preview, import, triage and doctor accept --json. Stdout then carries exactly one JSON document, and every human-readable line goes to stderr. For diff and gate that document is the same probe-compare.json that --compare-json writes. The exit code does not change.