__init__.py

Go to the documentation of this file.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# oc_chat/benchmarks/__init__.py
 
"""
OC Chat benchmark suite (plan 136) — re-runnable prompt/provider economics.
 
Purpose
-------
A manual, CLI-launchable harness that drives a fixed, scripted
multi-turn session through OC Chat's existing public entry points
(no production-code changes) and records the per-turn economics Plan
135 needs for its before/after comparison: prompt / cache / reasoning
token splits, TTFT, ReAct iterations, tool-call names, the composed
``messages[0]``/``messages[1]`` sizes, and the effective sub-loop
system-prompt size.
 
The pure logic (metric extraction, report writing, scenarios) is
hermetically unit-tested with no key and no network; the live runs are
a manual CLI (``python -m oc_chat.benchmarks``) that needs a real
provider key and is deliberately excluded from the auto pytest suite.
 
Responsibilities
----------------
- :mod:`oc_chat.benchmarks.metrics` — pure raw-usage → typed metrics +
  aggregation (hermetically tested).
- :mod:`oc_chat.benchmarks.report` — :class:`RunReport` + Markdown/JSON
  writers with deterministic, diff-stable output.
- :mod:`oc_chat.benchmarks.prices` — dated, provenance-stamped price
  table; tokens are the source of truth, ``$`` is best-effort.
- :mod:`oc_chat.benchmarks.presenter` — :class:`BenchmarkPresenter`,
  the metric tap over the presenter contract.
- :mod:`oc_chat.benchmarks.scenarios` — the fixed, version-stamped
  scenario registry.
- :mod:`oc_chat.benchmarks.runner` — the live driver: register →
  compose via the real startup path → isolate persistence → run →
  report.
- :mod:`oc_chat.benchmarks.__main__` — the ``run`` / ``run-all`` /
  ``list`` CLI.
 
Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
    L2 — lifecycle events
    L3 — semantic details
    L4 — deep tracing
 
Contracts
---------
- This package touches no shipped OC Chat module; it drives the engine
  through ``Agent`` / ``run_turn`` / the real startup composition path.
- Default runs are ephemeral: a benchmark-local autosave-suppression
  policy keeps chat logs, metadata, and tool artifacts out of the
  user's session store unless ``--persist`` is given.
- Provider keys come from the environment only and are never written
  into reports (only env-var *names* appear).
 
See Also
--------
oc_chat/plans/136-benchmark-suite.md — the authoring plan.
oc_chat/docs/architecture/benchmarks/benchmark-methodology.md — the spec.
"""
from __future__ import annotations
 
SCHEMA_VERSION = 1
HARNESS_VERSION = "1.0.0"
 
__all__ = ["SCHEMA_VERSION", "HARNESS_VERSION"]