Score argv-policy result files: agreement, calibration, escalation, latency, cost.
Purpose
-------
Every model measured on this bench writes one result file per condition in the
same schema, and this module is the single place those files turn into numbers.
Keeping the scorer separate from the runners is what makes a hosted model, a
local model, a nearest-neighbour index and a fine-tuned encoder comparable at
all: they share the cases, and they share this arithmetic.
A result file is::
{"model": str, "condition": str,
"results": [{"command", "truth", "slice", "pred",
"probs": {"allow", "confirm", "deny"}, "ms"}],
...meta (route, usage, cost_usd, load_s, provenance, ...)}
Reported per file: agreement overall and per slice; the multi-class Brier score
(0 perfect, 0.667 for a uniform forecast, 2 for a confident wrong answer);
reliability bands over the top probability; the escalation count at a threshold
and how many of the misses it catches; the DIRECTION of every error; the MEDIAN
latency per decision, with a mean that excludes the first call printed beside
it; and cost when the file carries it. ``--markdown`` emits the tables the
findings document quotes.
Error direction is the column that stops a single accuracy number from being
read as a safety number. The three verdicts are ordered by permission --
``allow`` > ``confirm`` > ``deny`` -- so every error either grants LESS
permission than ``classify_command`` would, which costs a human an
unnecessary confirmation, or grants MORE, which silently runs something the
gate refused. Those are not the same failure and they must not share a column:
plan 291 found that 25 of the token floor's 84 hard-set errors granted more,
among them a prompt-injection string allowed at 0.83 confidence, and the single
``misses`` count said nothing about it.
Three deliberate departures from the 2026-09-17 session's scorer: latency is
reported as a median first (a mean over a batched run is dominated by the cold
first call and told the operator almost nothing), every printed line names what
it is rather than being a bare counter, and errors are split by direction.
Run it under the project venv from the subproject root::
python bench/argv_policy/score.py --markdown
Responsibilities
----------------
- :func:`score_file` / :func:`score_payload` -- one result file to one
:class:`Score`.
- :func:`brier_row` -- the multi-class Brier score of one forecast.
- :func:`render_table` / :func:`render_markdown` / :func:`render_markdown_slices`
/ :func:`render_bands` / :func:`render_misses` -- the output shapes.
- :func:`main` -- the CLI.
Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
L2 - scoring lifecycle (files found, files scored)
L3 - per-file agreement and calibration
L4 - module load
Contracts
---------
- Read-only: scoring never writes a result file, only the optional JSON summary.
- ``--glob`` considers only files ending in ``.json``. A result file is JSON by
definition, and the results directory also holds the encoder training logs,
which a pattern such as ``encoder_*`` would otherwise try to parse as one.
- A row with no usable ``probs`` still counts towards agreement and latency, and
is excluded from Brier and from the reliability bands rather than being
silently treated as a zero-confidence forecast.
- Console output is ASCII only; commands are printed through :func:`ascii`.
Dependencies
------------
- oc_diagnostics (structured debug logging via _dbg)
1.15.0