ProbeRun

Docs / Using ProbeRun

Your first Project

A Project is one AI product under test. On a fresh Tenant the dashboard’s Overview offers a New project form (name, optional description) to anyone with the Developer or Admin role; submitting it opens the Project, ready for probe push. Viewers see who can create one instead. From CI or a script, the same thing is one call with an API key that carries the Developer role — no workspace id is needed, the Project lands in the Tenant’s default Workspace:

curl -s -X POST http://localhost:8000/projects -H "X-API-Key: $PROBE_API_KEY" \
  -H "content-type: application/json" -d '{"name": "Support bot"}'

(Use your API’s origin; the content-type header matters — without it the API answers 422.)

The response carries the new id to pass to probe push --project-id. A workspace_id in the body is still honoured for anyone who has one.

A fresh Project needs no Scenarios set up first. probe run --bundle records the Scenario definitions it graded, and probe push creates any the Project does not have yet before uploading the Runs (created 2 scenario(s): …). If a Scenario of that name already exists with a different definition, push keeps the server’s version, attaches the Runs to it, and prints a warning naming the fields that differ. A bundle written by an older CLI has no definitions; push then reports unknown scenario name and asks you to re-run with the current CLI. A Project holds one live Scenario per name: creating a second one of the same name (or restoring a deleted one over a live one) answers 409 scenario_exists, so pushes running in parallel never split a batch. Subagent Scenarios are not created by push; create them first.

An empty Project shows only its first-run block, which names three ways to the first Run: from your CI or laptop with the CLI; from your agent’s endpoint, saved on the Project’s Agent tab and run with Run on ProbeRun once the Project has Scenarios; and writing those Scenarios here with Add a Scenario (see “Add and edit Scenarios in the dashboard”). The other tabs stay one click away.

The CLI path gives the commands in order: install the CLI, export PROBE_API_URL (this deployment’s API) and PROBE_API_KEY (a key from Team → API keys), run with --bundle, then push. The key’s Download .env carries both lines; when the deployment declares no public API URL, the PROBE_API_URL line is commented out for you to fill in.

See a flip before wiring up an agent: the Example Project

An empty Project’s first-run block also offers Load an example comparison to a Developer or Admin (a Viewer is told who can). It creates a separate Project, Example: support bot, holding the hand-built support-bot data probe init --example ships: three Scenarios and two Batches. In the second Batch the bot refunds a $750 order itself instead of escalating it, so refund-over-threshold flips from pass to fail at step 3. The Project page opens on that flip in its headline.

  • It is marked example data everywhere it is named: its page header, the Projects list, the sidebar, the breadcrumbs and the dashboard’s Recent runs. Its Runs are left out of the dashboard’s pass rate.
  • Nothing runs and nothing is spent: the Runs are recorded data, not a live agent or a judge.
  • It takes no new data: pushing to it, adding a Scenario, or running it is refused with 409 example_read_only.
  • It does not use your plan’s Project allowance or monthly Run line, so it loads even on a plan that is at its limits.
  • A team has one at a time. Once it exists, an empty Project links to it instead of offering another. Delete it like any Project; after that you can load it again.

The same is one call: POST /projects/example (Developer); a second one answers 409 example_exists naming the live one.