oc_chat
Loading...
Searching...
No Matches
oc_chat.benchmarks.budget Namespace Reference

Classes

class  Status
class  BudgetError
class  Diff
class  Verdict

Functions

dict load_budget (str|Path|None path=None)
list[str] validate_budget (dict budget)
None _validate_entry (int idx, Any entry, dict[tuple[str, str, str], set[float]] consistency)
float _require_positive (Any value, str where)
None _require_positive_or_null (Any value, str where)
Verdict check_report (dict report, dict budget)
str _classify (bool invalid, list[Diff] regressions, bool no_budget)
str _missing_required (Any provider, Any model, Any effective_mode, Any scenario_name, Any messages0)
dict[tuple, dict] _index_entries (dict budget)
dict _provenance (dict report, dict budget)

Variables

 DEFAULT_BUDGET_PATH = Path(__file__).with_name("benchmark-budget.json")
int SUPPORTED_BUDGET_VERSION = 1
str _SUBLOOP_UNAVAILABLE = "unavailable"
dict _ALLOWED_ENTRY_KEYS
dict _ALLOWED_HARD_KEYS = {"messages0_size_max", "sub_loop_prompt_size_max"}
dict _ALLOWED_ADVISORY_KEYS = {"messages1_size_warn", "cold_full_miss_usd_warn"}

Detailed Description

Benchmark regression-budget checker — fail-closed, data-driven gate.

Purpose
-------
Lock Plan 135's prompt-slimming win (composed ``messages[0]`` 21,344 ->
5,847 chars) by comparing a benchmark :class:`RunReport` against a
tracked, provenance-stamped ``benchmark-budget.json`` and returning a
typed :class:`Verdict`. The correctness of the gate is the **metric
split** (plan 139 §3.1): the only v1 HARD gate is ``messages0_size``
(composition-only, repo- and price-invariant); ``messages1_size`` and
``cold_full_miss_usd`` are advisory (deterministic-for-a-commit but
drifting across commits/prices); noisy metrics (cache-hit ratio, TTFT,
tool-name distribution) are not surfaced in v1. The gate is
**fail-closed**: a clean ``pass`` means a hard gate actually ran against
valid, comparable data — ``no_budget`` / ``invalid_report`` / an errored
provider all classify as non-``pass``.

Responsibilities
----------------
- :func:`load_budget` — read a budget JSON with a clean error on a
  missing/unreadable/malformed file (no traceback).
- :func:`validate_budget` — reject an unknown ``budget_version``,
  unknown/duplicate keys, non-numeric/negative thresholds, and missing
  provenance; warn (never raise) on inconsistent ``messages0_size_max``
  across a ``(provider, model, mode)`` group.
- :func:`check_report` — a pure ``(report, budget) -> Verdict`` function
  with the explicit fail-closed iteration loop.
- :class:`Verdict` / :class:`Diff` / :class:`Status` — the typed result.

Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
    L2 — lifecycle events
    L3 — semantic details
    L4 — deep tracing

Contracts
---------
- :func:`check_report` is hermetic: dicts in, verdict out; no key, no
  network, no filesystem.
- Only deliberately-optional data is skipped (sub-loop prompt when
  ``sub_loop_prompt_source == "unavailable"``; advisory price metrics
  when pricing is absent; ``messages1_size`` when ``null``). A missing
  *required* field is ``invalid_report``, never a skipped pass.
- Thresholds live only in the budget config; this module holds only the
  which-metric-is-which-kind logic.

Dependencies
------------
- json (reads the budget file)
- dataclasses (dataclass, field - the Verdict and Diff result types)
- pathlib (Path - the budget file location)
- typing (Any - type hints for the untyped budget and report values)
- oc_diagnostics (structured debug logging via _dbg)

Function Documentation

◆ _classify()

str oc_chat.benchmarks.budget._classify ( bool invalid,
list[Diff] regressions,
bool no_budget )
protected
Verdict precedence: invalid_report > regressed > no_budget > pass.

Definition at line 379 of file budget.py.

Here is the caller graph for this function:

◆ _index_entries()

dict[tuple, dict] oc_chat.benchmarks.budget._index_entries ( dict budget)
protected

Definition at line 410 of file budget.py.

Here is the caller graph for this function:

◆ _missing_required()

str oc_chat.benchmarks.budget._missing_required ( Any provider,
Any model,
Any effective_mode,
Any scenario_name,
Any messages0 )
protected

Definition at line 390 of file budget.py.

Here is the caller graph for this function:

◆ _provenance()

dict oc_chat.benchmarks.budget._provenance ( dict report,
dict budget )
protected

Definition at line 420 of file budget.py.

Here is the caller graph for this function:

◆ _require_positive()

float oc_chat.benchmarks.budget._require_positive ( Any value,
str where )
protected

Definition at line 230 of file budget.py.

Here is the caller graph for this function:

◆ _require_positive_or_null()

None oc_chat.benchmarks.budget._require_positive_or_null ( Any value,
str where )
protected

Definition at line 236 of file budget.py.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ _validate_entry()

None oc_chat.benchmarks.budget._validate_entry ( int idx,
Any entry,
dict[tuple[str, str, str], set[float]] consistency )
protected

Definition at line 187 of file budget.py.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ check_report()

Verdict oc_chat.benchmarks.budget.check_report ( dict report,
dict budget )
Compare *report* against *budget* and return a fail-closed :class:`Verdict`.

Definition at line 242 of file budget.py.

Here is the call graph for this function:

◆ load_budget()

dict oc_chat.benchmarks.budget.load_budget ( str | Path | None path = None)
Load a budget JSON, raising :class:`BudgetError` on any read/parse failure.

Definition at line 131 of file budget.py.

◆ validate_budget()

list[str] oc_chat.benchmarks.budget.validate_budget ( dict budget)
Validate a budget dict; return consistency warnings, raise on fatal errors.

Definition at line 149 of file budget.py.

Here is the call graph for this function:

Variable Documentation

◆ _ALLOWED_ADVISORY_KEYS

dict oc_chat.benchmarks.budget._ALLOWED_ADVISORY_KEYS = {"messages1_size_warn", "cold_full_miss_usd_warn"}
protected

Definition at line 86 of file budget.py.

◆ _ALLOWED_ENTRY_KEYS

dict oc_chat.benchmarks.budget._ALLOWED_ENTRY_KEYS
protected
Initial value:
1= {
2 "provider", "model", "mode", "scenario", "source_report", "hard", "advisory",
3}

Definition at line 82 of file budget.py.

◆ _ALLOWED_HARD_KEYS

dict oc_chat.benchmarks.budget._ALLOWED_HARD_KEYS = {"messages0_size_max", "sub_loop_prompt_size_max"}
protected

Definition at line 85 of file budget.py.

◆ _SUBLOOP_UNAVAILABLE

str oc_chat.benchmarks.budget._SUBLOOP_UNAVAILABLE = "unavailable"
protected

Definition at line 80 of file budget.py.

◆ DEFAULT_BUDGET_PATH

oc_chat.benchmarks.budget.DEFAULT_BUDGET_PATH = Path(__file__).with_name("benchmark-budget.json")

Definition at line 74 of file budget.py.

◆ SUPPORTED_BUDGET_VERSION

int oc_chat.benchmarks.budget.SUPPORTED_BUDGET_VERSION = 1

Definition at line 77 of file budget.py.