oc_chat.benchmarks.report Namespace Reference

Classes
class	ProviderResult
class	RunReport

Functions
dict	capture_env ()
str	_git_commit ()
dict	_call_to_dict (CallMetrics c)
dict	_turn_to_dict (TurnMetrics t)
tuple[Path, Path]	unique_report_paths (str|Path results_dir, *, str label, str provider, str model, str scenario)

Detailed Description

Benchmark report model + deterministic Markdown / JSON writers.

Purpose
-------
Hold one scenario run's results (across one or more providers) and
emit two sibling files: a human Markdown report and a diff-stable JSON
sidecar. The schema is **additive across providers** (a ``providers``
array), so pointing the suite at a new provider never changes existing
fields; a failed/skipped provider records ``status`` + ``reason`` and
never aborts the file. ``messages[1]`` size is **nullable** (absent in
``standard`` mode — ``null`` in JSON, "absent" in Markdown, distinct
from a present-but-empty ``0``). No secret material is ever written —
only env-var *names* appear in a report.

Responsibilities
----------------
- :class:`ProviderResult` — one provider's composed-prompt sizes,
  per-turn metrics, effective mode/profile, and optional ``$`` pricing.
- :class:`RunReport` — run metadata + the ``providers`` array, with
  :meth:`to_dict` / :meth:`write_json` / :meth:`write_markdown`.
- :func:`capture_env` — UTC timestamp, git commit, Python, platform.
- :func:`unique_report_paths` — collision-proof ``.md`` + ``.json``
  paths (ms-precision UTC + ``+NN`` suffix).

Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
    L2 — lifecycle events
    L3 — semantic details
    L4 — deep tracing

Contracts
---------
- :meth:`write_json` uses ``sort_keys=True`` + rounded floats for clean
  diffs; reloading the JSON round-trips :meth:`to_dict`.
- ``messages1_size`` of ``None`` means *absent* and serialises to JSON
  ``null`` / Markdown "absent"; ``0`` means present-but-empty.
- :func:`unique_report_paths` guarantees two runs in the same second
  get distinct filenames.

Dependencies
------------
- oc_diagnostics (structured debug logging via _dbg)
- oc_chat.benchmarks.metrics (aggregate_turn / aggregate_run, TurnMetrics)
- oc_chat.benchmarks (HARNESS_VERSION, SCHEMA_VERSION - the version stamps written into every report)

Function Documentation

_call_to_dict()

dict oc_chat.benchmarks.report._call_to_dict ( CallMetrics c )

protected

Definition at line 100 of file report.py.

Here is the caller graph for this function:

_git_commit()

str oc_chat.benchmarks.report._git_commit ( )

protected

Short HEAD hash, or "unknown" — never raises into a report.

Definition at line 88 of file report.py.

Here is the caller graph for this function:

_turn_to_dict()

dict oc_chat.benchmarks.report._turn_to_dict ( TurnMetrics t )

protected

Definition at line 111 of file report.py.

Here is the call graph for this function:

capture_env()

dict oc_chat.benchmarks.report.capture_env

(

)

Capture the reproducibility-relevant run environment.

Definition at line 76 of file report.py.

Here is the call graph for this function:

unique_report_paths()

tuple[Path, Path] oc_chat.benchmarks.report.unique_report_paths	(	str | Path	results_dir,
		*	,
		str	label,
		str	provider,
		str	model,
		str	scenario )

Return collision-proof ``(md_path, json_path)`` under *results_dir*.

Filename: ``<UTC-ms-timestamp>__<label>__<provider>__<model>__<scenario>``
with a ``+NN`` suffix appended if a sibling already exists, so two
runs in the same second never collide.

Definition at line 285 of file report.py.