Calibration review

A Calibration label is your reviewer’s ground-truth Verdict for one Run (pass or fail), recorded after inspecting its Trajectory. ProbeRun compares it with the judge’s Verdict at read time: agreed, diverged, or unreviewed. A diverged Run is the thing to look at when you suspect the judge, not a number to trend.

In the dashboard you read agreement from two columns rather than a word: the Overview’s Recent runs list and each project’s runs table show Verdict (the judge’s) and Human (your reviewer’s label) side by side. A human Verdict that differs from the judge’s is printed in its own colour — that row is diverged; one that agrees is printed in muted text; a run nobody has labelled shows a dash (the same “—” the Duration and Score columns use for “no value”) in the Human column, read as “unreviewed” by a screen reader, so an unlabelled run never looks reviewed. The Diverged filter above the list shows the Runs where your reviewer and the judge disagree; it is free in every edition.

Every label write is recorded in the audit log, in every edition: calibration.label_set (the Run, the old and new verdict, and whether it came from the label control or from promoting feedback) and calibration.label_deleted. The reasoning text is never copied into the event.

Review queue (commercial edition)

The Review queue says which Runs someone should label next, per Project.

• What may enter. Only Runs a Judge judged: a Verdict of pass or fail, not stopped by a spend cap, in a live Scenario of that Project. A Run that errored or is still pending is refused with the reason not_judged; one from elsewhere is not_found; one already waiting is already_queued. The rest of the request still goes through. Nothing enters on its own, not even a Run with pending user feedback.
• Picking or sampling. A Developer either names Runs (up to 200 at a time, reason picked) or asks for a random sample of up to 100 unlabelled Runs, optionally narrowed to one Scenario and one judge configuration (reason sampled). The answer always states requested, enqueued and available, so a short sample reads as short. Sample randomly when you are building toward the 30-label floor: a queue fed only with hand-picked suspicious Runs pulls the agreement rate down.
• States. An item is open, claimed, skipped or withdrawn, and reads labelled as soon as its Run’s label is written after it entered. Nothing else completes it: you label through the ordinary label control (or by promoting feedback), and deleting the label reopens the item. A Run labelled before it was queued for a re-check stays open until someone labels it again, even with the same verdict. Skipped and withdrawn are final; queue the Run again to get a new item. A Run whose item already reads labelled can be picked again for a re-check: the old item keeps reading labelled while that label stands, and a new open item starts.
• Claim and assignment (a separate entitlement, calibration.review_assignment). A member can claim an open item, and a Developer can assign or release one; the assignee must be an active Developer or Admin, and a machine key cannot claim. Of two people claiming the same item at once, exactly one wins; the other is told already_claimed. Assignment coordinates reviewers and never limits who may label: any Developer can still label any Run.
• Roles. A Viewer can read the queue; every change needs a Developer.
• Audit. review_queue.enqueued (one per request), review_queue.state_changed, review_queue.claimed and review_queue.assigned are recorded; reads and refusals are not. A skip note’s length is recorded, never its words.
• In the dashboard. Every project page has a Review tab, in every edition. It lists the open and claimed items oldest first: the Scenario, a link to the Run, the Judge’s Verdict, your reviewer’s label, the item’s state and why it entered (picked or sampled). With calibration.review_assignment it adds an Assignee column and Claim, Release and Assign controls; without it no such control is drawn, and an item left claimed shows that state and can still be withdrawn. Assign offers the Project’s active Developers and Admins: choose one, then press Assign. A Viewer sees who holds an item as “a member”, without an email address. Skip passes on a Run; Withdraw takes out a Run that should not have been queued. On a phone the table leads with Scenario, state and Verdict and leaves out the entry reason. An empty queue says so and offers a random sample. A judge configuration still under the 30-label floor gets its own sample offer, filled in with how many labels it is short. If the queue cannot be read, the tab says so and never shows an empty queue. The open-source edition shows the tab with the one-line edition note in place of the queue. On a queued Run’s page, a line under the label control shows the item’s state and who holds it, a Skip control while it is open, and a link to the next open item. To add Runs by hand, tick them in a project’s runs table or in the Overview’s Recent runs list (including under the Diverged filter) and choose Add selected to Review: each Run joins the Review tab of its own Project; each refused Run is listed with its reason.
• In the open-source edition the queue routes answer 402 entitlement_required naming calibration.review_queue or calibration.review_assignment. Labels, derived agreement, the Diverged filter and the per-configuration agreement rate are unaffected.