|
| bool | ensure_provider_registered (str provider_name) |
| Iterator[Path|None] | autosave_suppression (bool persist) |
| tuple[int, str] | effective_sub_loop_prompt (list|None tool_schemas) |
| "ChatSession" | _compose_session ("Agent" agent, str model) |
| tuple[int, int|None] | _messages_sizes ("ChatSession" session) |
| None | _run_warmup ("Agent" agent, "ChatSession" session, str first_prompt) |
| ProviderResult | run_scenario_for_provider (*, str provider_name, str|None model, Scenario scenario, bool warmup=False, bool persist=False, PriceTable|None price_table=None) |
| RunReport | run_benchmark (*, Scenario scenario, list[str] providers, str|None model=None, str|None mode=None, bool warmup=False, bool persist=False, PriceTable|None price_table=None) |
The live benchmark driver — register, compose, isolate, run, report.
Purpose
-------
Reproduce the **real REPL/startup composition path** (not the bare
``Agent`` path) so the measured surface is the one Plan 135 changes,
then drive a scenario's scripted turns and assemble a
:class:`~oc_chat.benchmarks.report.RunReport`. Per run (plan §3.2):
1. register the provider (own helper) before any ``Agent(...)``;
2. apply the mode via ``init_runtime_settings`` (scenario-mode
authoritative; a conflicting ``--mode`` is a clean error);
3. build the agent, then compose the session through
``_create_new_session`` — the model passed outright (plan 220 P1;
the model choice is an interactive drill-down now) and only the
session NAME answered by a canned ``PromptDriver`` (inherits
whatever composition Plan 135 changes — no copy-drift);
4-5. wrap everything in a benchmark-local autosave-suppression policy
that patches **all four** write bindings so a default run never
touches the user's session store;
6. (optional ``--warmup``) warm the server prefix on a *throwaway*
``Agent`` + cloned session with its own presenter, fully discarded;
7. drive each turn, labelling turn 1 cold (warmup-aware);
8. capture the effective sub-loop prompt + composed ``messages[0]`` /
``messages[1]`` (nullable) sizes + effective mode/profile.
Responsibilities
----------------
- :func:`ensure_provider_registered` — import a provider bundle.
- :func:`autosave_suppression` — the four-binding persistence guard.
- :func:`effective_sub_loop_prompt` — version-independent sub-loop
prompt size.
- :func:`run_scenario_for_provider` / :func:`run_benchmark` — drive +
assemble reports.
Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
L2 — lifecycle events
L3 — semantic details
L4 — deep tracing
Contracts
---------
- Default (``persist=False``) runs leave the real sessions dir
untouched (chat log / metadata no-op'd; both ``save_tool_artifact``
bindings redirected to a temp dir that still returns a valid Path).
- Warmup never pollutes the measured session or presenter (separate
throwaway agent + presenter + cloned session, all discarded).
- A conflicting ``mode`` vs the scenario's mode raises ``ValueError``
(the CLI turns it into a clean exit-1).
Dependencies
------------
- oc_diagnostics (structured debug logging via _dbg)
- oc_chat.core.agent (lazy; Agent, AgentConfig)
- oc_chat.core.startup (lazy; _create_new_session)
- oc_chat.core.settings (lazy; init_runtime_settings, runtime_mode, active_profile)
- oc_chat.core.provider_plugin (lazy; registry)
- oc_chat.benchmarks.presenter (BenchmarkPresenter - the measurement tap for measured and warmup turns)
- oc_chat.benchmarks.metrics (aggregate_run, derived_full_miss_cost - run totals and the cold full-miss cost)
- oc_chat.benchmarks.report (ProviderResult, RunReport - the per-provider result and the run report)
- oc_chat.benchmarks.scenarios (Scenario - the scripted turns and declared mode)
- oc_chat.benchmarks.prices (PriceTable, load_prices - prices the run from its token totals)
- oc_chat.core.api_context (lazy; its save_tool_artifact binding, redirected to a temp dir)
- oc_chat.core.deployment_shape (lazy; check_deployment_shape - the deployment-shape gate before a provider import and a run)
- oc_chat.core.persistence (lazy; its save_tool_artifact binding, redirected to a temp dir)
- oc_chat.core.react_loop (lazy; its chat-log and metadata save bindings, suppressed)
- oc_chat.core.risk_profile (lazy; resolve_risk_profile - the profile the deployment-shape gate checks against)
- oc_chat.core.session (lazy; ChatSession - type hints for the composed and warmup sessions)
- oc_chat.core.sub_loop (lazy; the sub-loop prompt builder or run_sub_loop default, for the prompt size)
- oc_chat.tools.filesystem (lazy; _sandbox_root - the write sandbox the deployment-shape gate checks)
- providers (not imported; the provider bundle loaded by name through importlib so it self-registers)
1.15.0