|
| dict | load_budget (str|Path|None path=None) |
| list[str] | validate_budget (dict budget) |
| None | _validate_entry (int idx, Any entry, dict[tuple[str, str, str], set[float]] consistency) |
| float | _require_positive (Any value, str where) |
| None | _require_positive_or_null (Any value, str where) |
| Verdict | check_report (dict report, dict budget) |
| str | _classify (bool invalid, list[Diff] regressions, bool no_budget) |
| str | _missing_required (Any provider, Any model, Any effective_mode, Any scenario_name, Any messages0) |
| dict[tuple, dict] | _index_entries (dict budget) |
| dict | _provenance (dict report, dict budget) |
Benchmark regression-budget checker — fail-closed, data-driven gate.
Purpose
-------
Lock Plan 135's prompt-slimming win (composed ``messages[0]`` 21,344 ->
5,847 chars) by comparing a benchmark :class:`RunReport` against a
tracked, provenance-stamped ``benchmark-budget.json`` and returning a
typed :class:`Verdict`. The correctness of the gate is the **metric
split** (plan 139 §3.1): the only v1 HARD gate is ``messages0_size``
(composition-only, repo- and price-invariant); ``messages1_size`` and
``cold_full_miss_usd`` are advisory (deterministic-for-a-commit but
drifting across commits/prices); noisy metrics (cache-hit ratio, TTFT,
tool-name distribution) are not surfaced in v1. The gate is
**fail-closed**: a clean ``pass`` means a hard gate actually ran against
valid, comparable data — ``no_budget`` / ``invalid_report`` / an errored
provider all classify as non-``pass``.
Responsibilities
----------------
- :func:`load_budget` — read a budget JSON with a clean error on a
missing/unreadable/malformed file (no traceback).
- :func:`validate_budget` — reject an unknown ``budget_version``,
unknown/duplicate keys, non-numeric/negative thresholds, and missing
provenance; warn (never raise) on inconsistent ``messages0_size_max``
across a ``(provider, model, mode)`` group.
- :func:`check_report` — a pure ``(report, budget) -> Verdict`` function
with the explicit fail-closed iteration loop.
- :class:`Verdict` / :class:`Diff` / :class:`Status` — the typed result.
Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
L2 — lifecycle events
L3 — semantic details
L4 — deep tracing
Contracts
---------
- :func:`check_report` is hermetic: dicts in, verdict out; no key, no
network, no filesystem.
- Only deliberately-optional data is skipped (sub-loop prompt when
``sub_loop_prompt_source == "unavailable"``; advisory price metrics
when pricing is absent; ``messages1_size`` when ``null``). A missing
*required* field is ``invalid_report``, never a skipped pass.
- Thresholds live only in the budget config; this module holds only the
which-metric-is-which-kind logic.
Dependencies
------------
- json (reads the budget file)
- dataclasses (dataclass, field - the Verdict and Diff result types)
- pathlib (Path - the budget file location)
- typing (Any - type hints for the untyped budget and report values)
- oc_diagnostics (structured debug logging via _dbg)