OC Chat benchmark suite (plan 136) — re-runnable prompt/provider economics.
Purpose
-------
A manual, CLI-launchable harness that drives a fixed, scripted
multi-turn session through OC Chat's existing public entry points
(no production-code changes) and records the per-turn economics Plan
135 needs for its before/after comparison: prompt / cache / reasoning
token splits, TTFT, ReAct iterations, tool-call names, the composed
``messages[0]``/``messages[1]`` sizes, and the effective sub-loop
system-prompt size.
The pure logic (metric extraction, report writing, scenarios) is
hermetically unit-tested with no key and no network; the live runs are
a manual CLI (``python -m oc_chat.benchmarks``) that needs a real
provider key and is deliberately excluded from the auto pytest suite.
Responsibilities
----------------
- :mod:`oc_chat.benchmarks.metrics` — pure raw-usage → typed metrics +
aggregation (hermetically tested).
- :mod:`oc_chat.benchmarks.report` — :class:`RunReport` + Markdown/JSON
writers with deterministic, diff-stable output.
- :mod:`oc_chat.benchmarks.prices` — dated, provenance-stamped price
table; tokens are the source of truth, ``$`` is best-effort.
- :mod:`oc_chat.benchmarks.presenter` — :class:`BenchmarkPresenter`,
the metric tap over the presenter contract.
- :mod:`oc_chat.benchmarks.scenarios` — the fixed, version-stamped
scenario registry.
- :mod:`oc_chat.benchmarks.runner` — the live driver: register →
compose via the real startup path → isolate persistence → run →
report.
- :mod:`oc_chat.benchmarks.__main__` — the ``run`` / ``run-all`` /
``list`` CLI.
Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
L2 — lifecycle events
L3 — semantic details
L4 — deep tracing
Contracts
---------
- This package touches no shipped OC Chat module; it drives the engine
through ``Agent`` / ``run_turn`` / the real startup composition path.
- Default runs are ephemeral: a benchmark-local autosave-suppression
policy keeps chat logs, metadata, and tool artifacts out of the
user's session store unless ``--persist`` is given.
- Provider keys come from the environment only and are never written
into reports (only env-var *names* appear).
See Also
--------
oc_chat/plans/136-benchmark-suite.md — the authoring plan.
oc_chat/docs/architecture/benchmarks/benchmark-methodology.md — the spec.
1.15.0