oc_chat.benchmarks.__main__ Namespace Reference

Functions
argparse.ArgumentParser	_build_parser ()
list[str]	_resolve_providers (args)
str|None	_resolve_mode (args)
int	_run_one (scenario, args, providers, mode, price_table)
None	_print_verdict (verdict, *, bool downgraded=False)
tuple[dict|None, int]	_load_report_for_check (args, providers, mode, price_table)
int	_handle_check (args)
int	main (list[str]|None argv=None)

Variables
tuple	DEFAULT_RESULTS_DIR

Detailed Description

Benchmark CLI — ``python -m oc_chat.benchmarks [run|run-all|list|check]``.

Purpose
-------
The manual entry point for the suite (live runs need a real provider
key and are deliberately outside the auto pytest suite). Drives one or
all scenarios, writes a timestamped ``.md`` + ``.json`` report per
scenario, isolates per-provider failures, and gates a report against the
tracked regression budget (plan 139).

Responsibilities
----------------
- Parse args; resolve the provider list, mode, and price table.
- ``list`` — print the scenario set.
- ``run`` / ``run-all`` — drive scenario(s) and emit reports.
- ``check`` — compare a report (``--report`` existing, or ``--run``
  fresh) against ``benchmark-budget.json`` and exit non-zero on a hard
  regression / ``no_budget`` / ``invalid_report`` (fail-closed §3.4).
- Clean exit codes: unknown scenario / missing label / mode conflict /
  single-provider failure → exit 1 (no traceback); multi-provider
  failures are per-provider skip entries and the run continues.

Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
    L2 — lifecycle events
    L3 — semantic details
    L4 — deep tracing

Contracts
---------
- Default reports go to the tracked results dir; implementer smoke runs
  pass ``--results-dir <temp>`` so canonical results are never polluted.
- Keys come from the provider's env var; a missing key on a
  single-provider run is a clean error, not a traceback.

Dependencies
------------
- oc_chat.benchmarks.runner (run_benchmark - drives a scenario across the providers)
- oc_chat.benchmarks.scenarios (get, all_scenarios - the scenario set for list, run and run-all)
- oc_chat.benchmarks.prices (load_prices - the price table for a run)
- oc_chat.benchmarks.report (RunReport, unique_report_paths - the report and its collision-proof file paths)
- oc_chat.benchmarks.budget (load_budget, validate_budget, check_report - the regression-budget gate behind check)
- oc_diagnostics (structured debug logging via _dbg)

Function Documentation

_build_parser()

argparse.ArgumentParser oc_chat.benchmarks.__main__._build_parser ( )

protected

Definition at line 75 of file __main__.py.

Here is the caller graph for this function:

_handle_check()

int oc_chat.benchmarks.__main__._handle_check ( args )

protected

Load + validate the budget, resolve the report, gate it, map the exit code.

Definition at line 223 of file __main__.py.

Here is the call graph for this function:

Here is the caller graph for this function:

_load_report_for_check()

tuple[dict | None, int] oc_chat.benchmarks.__main__._load_report_for_check ( args, providers, mode, price_table )

protected

Resolve the report dict for `check` from --report or --run; (report, rc).

Definition at line 195 of file __main__.py.

Here is the caller graph for this function:

_print_verdict()

None oc_chat.benchmarks.__main__._print_verdict ( verdict, * , bool downgraded = False )

protected

Print a compact verdict: status, per-metric table, provenance, reasons.

Definition at line 170 of file __main__.py.

Here is the caller graph for this function:

_resolve_mode()

str | None oc_chat.benchmarks.__main__._resolve_mode ( args )

protected

Definition at line 131 of file __main__.py.

Here is the caller graph for this function:

_resolve_providers()

list[str] oc_chat.benchmarks.__main__._resolve_providers ( args )

protected

Definition at line 124 of file __main__.py.

Here is the caller graph for this function:

_run_one()

int oc_chat.benchmarks.__main__._run_one ( scenario, args, providers, mode, price_table )

protected

Run a single scenario, write its report, return a partial exit code.

Definition at line 137 of file __main__.py.

Here is the caller graph for this function:

main()

int oc_chat.benchmarks.__main__.main

(

list[str] | None

argv = None

)

Definition at line 251 of file __main__.py.

Here is the call graph for this function:

Variable Documentation

DEFAULT_RESULTS_DIR

tuple oc_chat.benchmarks.__main__.DEFAULT_RESULTS_DIR

Initial value:

=  (
    Path(__file__).resolve().parents[1] / "docs" / "architecture" / "benchmarks" / "results"
)

Definition at line 70 of file __main__.py.