Standing. Once the Project has Runs, the Overview opens with whether the agent is passing,
counted from each Scenario’s newest Run: 1 of 2 Scenarios failing, the failing Scenario’s name,
the Judge’s reason and Open the Run; or All 2 Scenarios passing. Scenarios that have not
run are counted aloud (1 not run yet); a stub’s Verdict or an error never counts as a pass.
Under it, every Scenario with its last Verdict, how that Run was graded (its checks, or a model
judge named by model), and its number of Runs. A Scenario with no checks says so, and a
Developer or Admin gets add checks.
Folded detail. Batches, failures grouped by tool calls, failures grouped by the Judge’s
reasons, and judge agreement sit closed under the runs table, each with its count or state in
its summary (1 group, No labels yet, Could not be read). The cluster count in the summary
line opens its section.
Summary line. The runs table opens with one line: how many Runs, the pass rate
and how many failure clusters, for example 18 runs · 61% pass · 5 failure clusters.
The cluster count jumps to the clusters panel below. The pass rate counts only Verdicts a real
runner and Judge produced: a Verdict a stub produced is shown as pass* or fail*, left out of
the rate, and counted aloud (6 synthetic, not counted), so a Project graded only by stubs shows
— pass, never 100%. The Dashboard and the Projects list use the same rule.
Run table columns.Your label is the Verdict you or a reviewer gave the Run;
Violations is the Policy check on it (clean, scan failed or not scanned). A legend line
under each run table says the same.
Overview and triage table. Filter and sort Runs; click a row to open the
quick-diagnosis drawer without losing your place. The URL tracks your
filters and the open Run so it can be shared.
Failure clusters. Failed Runs grouped by their Trajectory signature,
the ordered sequence of tool names the agent invoked. One cluster is one
kind of failure: fix the pattern once, fix many.
Semantic clusters. A second, additional lens grouping failed Runs by
the meaning of the judge’s reasoning. It never replaces the signature
view, and every response states its coverage (how many failing Runs could
be considered and why the rest were excluded), so it is never silently
partial. Runs are embedded only when an embedding key is configured.
Judge agreement. One row per judge configuration: how often the judge
agreed with your reviewers’ labels, always shown as a count over a count,
with the 2×2 matrix behind it and a chance-corrected agreement (Cohen’s κ)
that stays blank — with the reason stated — until at least 30 labelled
Runs exist. A judge that always says “pass” looks excellent on a mostly-
passing corpus; κ is what catches that.
The Overview tab of a Project. Three batches, 12 Runs.
