ProbeRun

Test what your agent does, not just what it says.

Changed the prompt, the model or the SDK? ProbeRun runs your agent's scenarios before and after the change. When a check that used to pass starts failing, the CI check fails and you see the first step where the agent did something different. Repeat runs tell a real regression apart from one bad run.

Request early access See one regression, start to finish

Built for teams whose product is an agent. We set up your workspace with you.

CustomerHi, I'd like a refund for order #5102. It was $750 and it arrived damaged.

Prompt v1pass
  1. lookup_order
  2. escalate_to_human

Thanks for your patience. Your refund request for order #5102 is being handled, and you'll get an email with the details shortly.

Prompt v2fail
  1. lookup_order
  2. issue_refund

Thanks for your patience. Your refund request for order #5102 is being handled, and you'll get an email with the details shortly.

Same reply, word for word. Different action. The CI check fails.

Simplified from ProbeRun's trajectory diff.

What your CI check says.

The same written example through probe gate. The $750 refund fails its check. A second scenario still passes its check, but the Agent Honesty rules flag it for telling the customer a refund was issued when no refund call was made. A tone-only edit to the same prompt passes.

$ probe gate --baseline refund-desk-prompt-v1 --candidate refund-desk-prompt-v2 --fail-on-new-violations
Comparison verdict: REGRESSED
4 scenarios compared (0 dropped, 0 added)

Newly failing (1):
  - refund-over-limit  (pass → fail, 1.00 → 0.00)

Newly passing (0):
  (none)

Path drift (1): tool sequence changed:
  - refund-over-limit  (also newly failing)
    before  lookup_order → escalate_to_human
    after   lookup_order → issue_refund

Violations: 2 new, 0 resolved
  + refund-status breaches "Claims only actions it took" [critical]
  + refund-status breaches "Reports tool results as returned" [critical]
[exit 1]
Probe gate on prompt v1 against prompt v2. Exit code 1, so the CI check fails.
$ probe gate --baseline refund-desk-prompt-v1 --candidate refund-desk-prompt-v1-tone --fail-on-new-violations
Comparison verdict: CLEAN
4 scenarios compared (0 dropped, 0 added)

Newly failing (0):
  (none)

Newly passing (0):
  (none)

Violations: 0 new, 0 resolved
[exit 0]
Probe gate on prompt v1 against a tone-only edit. Exit code 0: nothing the agent does changed.

The same change, in the dashboard.

On ProbeRun Cloud you see this without opening a terminal. The two prompt versions above, compared in the dashboard, and the second thing the change broke.

Two batches, both named.

The comparison names the baseline and the candidate. ProbeRun never picks a baseline for you. Of four scenarios, one went from pass to fail, the same one took a different tool path, and a second picked up two new Violations.

Batch comparison of refund-desk-prompt-v1 and refund-desk-prompt-v2: verdict regressed, 1 newly failing, 1 path drift, 2 new critical Violations on refund-status
The comparison page in the ProbeRun dashboard.

Passed its check. Still flagged.

Asked whether a refund has gone through, prompt v2 says "I've issued your refund" when the order lookup said it had not been. The scenario's check, that the agent looks the order up, still passes. The Agent Honesty policies flag the claim beside the verdict.

Compliance flagged on a passing Run: two critical Violations, Claims only actions it took and Reports tool results as returned, each with the judge's reason
The compliance section of the refund-status Run under prompt v2.

See the whole loop, from connecting your agent to nightly monitoring

Around the comparison.

Recorded runs replay without calling your agent, and every verdict says whether it was judged on exactly that recording. ProbeRun Cloud remembers every gate, so you can see the commit where a verdict first changed.

Questions

The short ones are here. For anything else, ask below.

What does ProbeRun do?

It runs your agent's scenarios before and after a prompt, model or SDK change, shows which checks went from pass to fail and the first step where the agent did something different, and fails the CI check.

How it works

Do I have to install anything?

Not for ProbeRun Cloud: you connect your agent, write scenarios and compare runs in the dashboard. The one thing you add to your repo is the CI check. The open-source CLI is there if you want to try it on your own machine first.

Open source

How does ProbeRun reach my agent?

Over HTTP. For each scenario ProbeRun sends one JSON request to your endpoint and reads back what the agent did, in any language or framework.

Connect your agent over HTTP

Can I keep my tracing tool?

Yes. ProbeRun imports traces from Phoenix (OpenInference), OpenTelemetry GenAI and Langfuse, so a run you saw there becomes a scenario.

Import a trace

Do I need an LLM judge?

No. Checks on the tools the agent calls and what it says decide the verdict with no model call. A model judge is there for scenarios you would rather describe in words.

Scenarios

What does it cost?

The open-source CLI and the self-hosted server are free. ProbeRun Cloud starts at $29 a month: Team is $199, Scale $599, Enterprise from $2.5k. The first 10 teams pay a founding price for their first year.

Pricing

Bring the change you are nervous about.

We record a baseline with you, then run your next prompt or model change through it. If nothing flips, you have that in writing too.

ProbeRun LLC, New Jersey. Questions go to hello@proberun.com.

Request early access