Connect your agent over HTTP

RUNNER=http lets ProbeRun test any agent you can put behind an HTTP endpoint, in any language or framework: a LangGraph graph, a Microsoft Foundry agent, or your own code. For each Scenario, ProbeRun POSTs one JSON request to your endpoint and reads back what the agent did (its Trajectory). This page is the whole contract.

Settings

Variable	Default	What it does
`RUNNER`	`stub`	Set to `http` to use your endpoint.
`HTTP_RUNNER_URL`	(empty)	The endpoint ProbeRun POSTs to. Required.
`HTTP_RUNNER_AUTH_HEADER`	(empty)	Sent as the `Authorization` header, e.g. `Bearer sk-...`.
`HTTP_RUNNER_TIMEOUT_S`	`120`	Limit for the whole call, not per read.
`HTTP_RUNNER_ALLOW_PRIVATE`	`false`	Set `true` to call `localhost` or a private address (see below).
`HTTP_RUNNER_MAX_RESPONSE_BYTES`	`10485760`	Largest response ProbeRun reads (10 MiB).

The probe CLI reads these from the environment or a .env file in the current directory. A self-hosted server reads them from its .env.

Testing an agent on your own machine

By default ProbeRun calls only public (globally routable) addresses, so a stray URL cannot reach your internal network. An agent on localhost therefore needs HTTP_RUNNER_ALLOW_PRIVATE=true; without it every Scenario is recorded as an error (“resolves to a disallowed address”), and the message names the flag. Leave it false anywhere the URL is not under your control. Link-local addresses, including cloud metadata services such as 169.254.169.254, stay blocked even with the flag on.

The request

POST to HTTP_RUNNER_URL, Content-Type: application/json:

{
  "scenario_id": null,
  "scenario_name": "refund-under-threshold",
  "user_prompt": "I want a refund for order #4821.",
  "expected_behavior": "Approve refunds under $500 without escalation.",
  "tags": ["billing"]
}

user_prompt is the message to give your agent.
scenario_id is null when the Scenario comes from a local probe.yaml; it is the Scenario’s id when a ProbeRun server starts the Run.
expected_behavior and tags are there for logging; your agent should not need them.

Redirects are not followed.

The response

Return HTTP 2xx with a JSON object in one of the shapes below. Anything else is recorded as an error Run, never as a pass: a non-2xx status, a body that is not JSON, a body larger than the cap, no answer within the timeout, or a body with no recognisable steps (“agent endpoint returned no trajectory steps”).

Native Trajectory (preferred)

{"trajectory": [
  {"step": 0, "role": "user",  "content": "I want a refund for order #4821."},
  {"step": 1, "role": "agent", "content": "Checking.",
   "tool_calls": [{"name": "lookup_order", "args": {"order_id": "4821"}}]},
  {"step": 2, "role": "tool",  "name": "lookup_order", "result": {"total": 120}},
  {"step": 3, "role": "agent", "content": "Refund of $120 approved."}
]}

Steps are kept as you send them; a missing step number is filled in from the position.

Chat messages

If you already have a chat transcript, return it as messages and ProbeRun maps it to Steps:

{"messages": [
  {"role": "user", "content": "I want a refund for order #4821."},
  {"role": "assistant", "content": "",
   "tool_calls": [{"function": {"name": "lookup_order", "arguments": "{\"order_id\": \"4821\"}"}}]},
  {"role": "tool", "name": "lookup_order", "content": "{\"total\": 120}"},
  {"role": "assistant", "content": "Refund of $120 approved."}
]}

assistant (or agent) becomes an agent Step, user a user Step, tool a tool Step.
Tool calls may use the OpenAI Chat Completions form ({"function": {"name", "arguments"}}) or a flat {"name", "args"}.
If the list does not start with a user message, ProbeRun puts the Scenario’s prompt first.

The same messages list may hold what common SDKs produce, so you can usually return your framework’s message history as it is:

Source	Items ProbeRun reads
OpenAI Chat Completions	`user` / `assistant` / `tool` messages; `tool_calls`; a tool reply without a `name` is named after the call its `tool_call_id` answers
OpenAI Responses / Agents SDK	`function_call` and `function_call_output` items (they carry no `role`), message content as `output_text` blocks
Anthropic Messages	content blocks `text`, `tool_use` (in the assistant turn) and `tool_result` (in the user turn)
LangChain	message dicts with `type` `human` / `ai` / `tool`, fields at the top level or under `data`

Items ProbeRun does not recognise, such as system prompts and reasoning items, are skipped.

A bare reply

An agent with no tool calls can return just its answer, recorded as one agent Step after the prompt: {"reply": "..."} (or output / content holding a string).

If none of these match, the Run’s error names the top-level keys it got and the shapes it accepts.

Make sure tool calls reach the Trajectory. An assertion such as trajectory_must_not_contain can only fail on a tool call ProbeRun can see.

A minimal endpoint

A dozen lines of Python standard library, no framework. Replace answer with a call to your agent:

from http.server import BaseHTTPRequestHandler, HTTPServer
import json

class Agent(BaseHTTPRequestHandler):
    def do_POST(self):
        req = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        answer = f"You asked: {req['user_prompt']}"  # call your agent here
        body = json.dumps({"messages": [{"role": "assistant", "content": answer}]}).encode()
        self.send_response(200); self.send_header("Content-Type", "application/json"); self.end_headers()
        self.wfile.write(body)

HTTPServer(("127.0.0.1", 8952), Agent).serve_forever()

Run it, then in another terminal:

probe init
RUNNER=http HTTP_RUNNER_URL=http://localhost:8952/ HTTP_RUNNER_ALLOW_PRIVATE=true \
  probe run --config probe.yaml

The judge is still the stub, so the rows read PASS*. For a real Verdict set a judge, for example JUDGE=litellm LITELLM_JUDGE_MODEL=ollama_chat/qwen3.5:9b with the proberun[litellm] extra installed and Ollama running. Against the echo server above that judge returns FAIL for both starter Scenarios, as it should: echoing the question is not the expected behaviour.

Fuller examples served over this contract: a LangGraph agent in examples/langgraph-agent/, and a Microsoft Foundry prompt agent (Responses API items, function tools run by the shim) in examples/foundry-agent/.

Record once, replay after

probe capture --runner http calls the endpoint once and writes a replayable Fixture under FIXTURES_DIR/builder/<out>/: the normalized Steps (recorded_trajectory.jsonl), a sidecar with a pinned digest, and a draft scenario.yaml whose subagent_config names the Fixture.

export HTTP_RUNNER_URL=https://agent.example.com/run
export HTTP_RUNNER_AGENT_VERSION="$(git rev-parse HEAD:agent)"   # tree hash of the agent's source
probe capture --runner http --prompt "I want a refund for order #4821." --out refund

Write expected_behavior (and accept the drafted expected_path if it is right), add the Scenario to probe.yaml, and probe run replays it without calling the endpoint. The run header reads runner: http (replay) and the determinism line matched ... subject matched.

Replay re-grades, it does not re-execute. The judge sees the Steps the agent returned at capture time.
The version signal decides replay or live. With HTTP_RUNNER_AGENT_VERSION unset, or different from the one pinned at capture, nothing proves the deployed agent is the one recorded, so probe run calls the endpoint live and says why (ran live: ... (agent changed since capture)). A commit SHA would change on every push and never replay; use a tree hash of the agent’s own directory, an image digest or a release tag.
The prompt is part of the recording. Edit a Scenario’s user_prompt after capture and the recording no longer answers it, so probe run calls the endpoint live; re-capture to replay the new prompt. Only Fixtures written by probe capture --runner http replay through the HTTP runner; a subagent Fixture named by the same Scenario is ignored and the endpoint is called.
No silent fallback. When a live call cannot run (for example a private address without HTTP_RUNNER_ALLOW_PRIVATE) and nothing proves the recording is this agent, the Run stays an error rather than replaying the recording.
Secrets. Capture refuses (exit 2, naming the Step) when a Step holds a credential-shaped string; --allow-sensitive writes it anyway, after you have checked. Header values are never recorded.
Drift sentinel. probe sentinel replays the recording and calls the endpoint live, then compares. An endpoint that does not resolve or refuses the connection is a dead sensor (live error ... check the endpoint is reachable), not drift.