Open source

The probe CLI is free and Apache-2.0 licensed. It runs the same loop as ProbeRun Cloud on your own machine and in your CI: it records what your agent did, step by step and tool call by tool call, runs the same scenarios against your new prompt or model, and tells you which ones went from pass to fail. It is the way to try ProbeRun before you bring your team to the dashboard. The examples below use the same Refund Desk support bot as the rest of the site.

Install

ProbeRun's command-line tool is a Python package, published to PyPI with the open-source release. To use it before then, request early access.

probe init
probe run --config probe.yaml \
  --output baseline.json

probe init writes a starter probe.yaml. With no runner or judge configured, the first run says so: its verdicts are synthetic until you point it at your agent.

$ probe init
wrote probe.yaml
next: probe run --config probe.yaml
[exit 0]
$ probe run --config probe.yaml --output baseline.json
runner: stub · judge: stub · no agent was called (stub runner); verdicts are synthetic (stub judge)
PASS*  example-happy-path
PASS*  example-out-of-scope

2 passed, 0 failed (2 total)
* synthetic: a stub runner or judge produced it (see the runner/judge line)
[exit 0]

Triage

If you export traces by the hundred, probe triage ranks a folder of them, in any of the formats probe import reads, by agent-side struggle: repeated tool calls, failed tool calls and repeated output. It prints the probe import command for the worst, and it writes nothing. A flag says where to look first, not that the run failed.

Capture

A scenario starts from a real run, not a hand-typed list. If your agent already exports traces (OpenInference from Phoenix, OpenTelemetry GenAI, or a trace saved from Langfuse), probe import turns one into a recorded trajectory and a draft scenario, and lists anything it could not map instead of guessing.

$ probe import --source openinference --input trace-export.json --agent refund-desk
imported_with_gaps
  fixtures/builder/imported-openinference-110000000000
  gap: writer/no_filesystem_evidence x1
[exit 0]

Replay

Replaying runs the scenario against the recording instead of calling your agent, so the same recording gives the same trajectory every time. You write what should happen; the import above cannot. A recording made with probe capture also carries a digest, and every replay of it is checked against that digest.

$ probe run --config probe.yaml --output baseline.json
runner: subagent (replay) · judge: structural
PASS   refund-over-limit

1 passed, 0 failed (1 total)
[exit 0]

An LLM judge is paid the first time it judges a replayed trajectory. After that probe run keeps its pass or fail and reuses it while the trajectory, the scenario and the judge are unchanged, so the replay makes no judge call either. A changed recording, scenario text or judge prompt is judged again, and --no-judge-cache judges everything again.

$ probe run --config probe-sentinel.yaml --mode replay
runner: subagent (replay) · judge: litellm (ollama_chat/qwen3.5:9b)
PASS   scenario-A-typo-fix
FAIL   scenario-C-refusal
PASS   scenario-E-scope-creep

2 passed, 1 failed (3 total)

replay determinism
    scenario-A-typo-fix: matched (sha256-probe-replay-v1)
    scenario-C-refusal: matched (sha256-probe-replay-v1)
    scenario-E-scope-creep: matched (sha256-probe-replay-v1)
  3 matched, 0 diverged, 0 not recorded
judge cache: 3 of 3 Verdict(s) reused from an earlier judgement of the same replayed Trajectory (no judge call). --no-judge-cache judges again.
[exit 1]

Compare

Two runs in, one answer out. By default, the CI check fails when a scenario went from pass to fail (a flipped verdict), not when an average score moved. When the agent reaches the same result through different tool calls, that is reported as path drift, even if every scenario still passes. A gate: block changes that rule; see Gate tiers. Exit code 0 means clean, 1 means a gating condition was met, 2 means no verdict could be computed.

$ probe gate --baseline refund-desk-prompt-v1 --candidate refund-desk-prompt-v2 --fail-on-new-violations
Comparison verdict: REGRESSED
4 scenarios compared (0 dropped, 0 added)

Newly failing (1):
  - refund-over-limit  (pass → fail, 1.00 → 0.00)

Newly passing (0):
  (none)

Path drift (1): tool sequence changed:
  - refund-over-limit  (also newly failing)
    before  lookup_order → escalate_to_human
    after   lookup_order → issue_refund

Violations: 2 new, 0 resolved
  + refund-status breaches "Claims only actions it took" [critical]
  + refund-status breaches "Reports tool results as returned" [critical]
[exit 1]

If a run broke outside your agent (a judge crash, a provider outage, a bad config), the report says it could not run and tells you to re-run those. It still fails the check: a scenario nobody checked never reads as passing.

Path drift compares tool order exactly. When the order is the model's choice, declare path_tolerance: any_order on the scenario. A reorder is then listed, not reported as drift; a dropped or added tool still is.

Starter packs

Ready-made policies for agent honesty, support bots and agents that edit code, plus safety, privacy and audit basics, turned on per project in the policies tab of your ProbeRun server. A violation is recorded beside the verdict, never in place of it, and probe gate --fail-on-new-violations makes the CI check fail when a change introduces one. The coding-agent pack also has pattern rules for destructive commands and key-shaped strings that need no LLM judge; they read only what the agent itself did, so a user typing "don't run rm -rf" is not a violation. They are starting points, not a certified standard.

Repeat

One live run per scenario is noise: run the same agent twice and several scenarios change result with nothing else changed. probe run --runs 4 runs each live scenario four times, checks whether each drop is bigger than chance would explain, and says the smallest drop that many runs can detect. It never says there are no regressions; it says what it could not detect. A replayed scenario runs once, because identical replays are not independent samples.

probe run --config probe.yaml --runs 4 \
  --output candidate.json
probe diff baseline.json candidate.json

$ probe diff baseline.json candidate.json
Comparison verdict: REGRESSED
5 scenarios compared (0 dropped, 0 added)

Run sets (4 vs 4 Runs per Scenario):
  - count-letters  4/4 → 0/4  (p 0.004, drop)
  - refund-total  4/4 → 0/4  (p 0.004, drop)
  - fare-difference  3/4 → 0/4  (p 0.035, drop)
  - refund-reference  4/4 → 2/4  (p 0.089, not significant)
  3 flagged, 0.1 expected by chance (chance of 3 or more: 0.00).
  At 4 vs 4 graded Runs a drop from 4/4 is flagged at 1/4 or fewer passes (5 Scenarios).

On its own the run-set report does not change the verdict or the exit code. A gate: block does.

Gate tiers

With a gate: block in probe.yaml, each scenario is critical or standard, set by tag or per scenario. A critical scenario fails the check on any drop, and when it stays clean the report says what its runs rule out. A standard scenario fails the check on a drop significant at a threshold that accounts for the suite's size, so a gate in which nothing changed fails at most 1 time in 20. There is no re-run step. Every gate states what it can catch: on ProbeRun Cloud the project page names the smallest drop the gate catches 8 times in 10, and under every gated comparison "How sure is this?" gives each drop as a frequency, never as a chance that it is real.

gate:
  runs: 4
  default_tier: standard
  tier_by_tag:
    refunds: critical

Push results to ProbeRun Cloud and your team gets the dashboard: the comparison, the step where two runs split, Violations, gate history and nightly monitoring, without anyone else installing the CLI. The self-hosted server still runs the sentinel and keeps both batches; it does not keep the history. See How it works and Pricing.