oc_chat.benchmarks.metrics Namespace Reference

Classes
class	CallMetrics
class	TurnMetrics

Functions
int	_as_int (object value)
int	_reasoning_tokens (dict raw_usage)
CallMetrics	extract_call_metrics (dict|None raw_usage)
CallMetrics	aggregate_calls (list[CallMetrics] calls)
CallMetrics	aggregate_turn (TurnMetrics turn)
CallMetrics	aggregate_run (list[TurnMetrics] turns)
float	derived_full_miss_cost (int prompt_tokens, float miss_rate_per_mtok)

Detailed Description

Pure metric extraction + aggregation for the benchmark suite.

Purpose
-------
Turn a raw provider ``usage`` dict (the one ``presenter.show_usage``
receives — the only place ``reasoning_tokens`` and DeepSeek's prefix-
cache split survive; ``provider_plugin.UsageStats`` drops the former
and the harness wants both) into a typed :class:`CallMetrics`, then
aggregate calls → turns → run totals. **Hit and miss tokens are kept
distinct and never summed** — cache-hit is ~120x cheaper on DeepSeek
V4-pro, so collapsing them would erase the very economics the suite
measures.

All functions are pure: no I/O, no network, no global state. This is
the hermetically-tested core (plan 136 §6 tests 1-2).

Responsibilities
----------------
- :class:`CallMetrics` — one API call's token figures.
- :class:`TurnMetrics` — one scripted turn (its calls + TTFT + tool
  names + wall-clock + iteration/tool counts + cold/warm label).
- :func:`extract_call_metrics` — raw usage dict → :class:`CallMetrics`.
- :func:`aggregate_turn` / :func:`aggregate_run` — sum the token
  fields, hit/miss kept separate.
- :func:`derived_full_miss_cost` — the deterministic cold-turn cost
  (``prompt_tokens x miss-rate``), Plan 135's win #2.

Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
    L2 — lifecycle events
    L3 — semantic details
    L4 — deep tracing

Contracts
---------
- ``extract_call_metrics`` never raises on a malformed/partial usage
  dict — missing fields yield zeros, not ``KeyError``.
- Cache hit and miss are independent fields; no function sums them.
- Cache extraction delegates to the production
  :func:`oc_chat.core.cache_usage.usage_prompt_cache_hit_miss` so the
  numbers match the runtime exactly (and inherit its
  DeepSeek-then-OpenAI fallback).

Dependencies
------------
- oc_diagnostics (structured debug logging via _dbg)
- oc_chat.core.cache_usage (usage_prompt_cache_hit_miss — read-only)

Function Documentation

_as_int()

int oc_chat.benchmarks.metrics._as_int ( object value )

protected

Coerce a usage value to int, treating ``None``/garbage as 0.

Definition at line 100 of file metrics.py.

Here is the caller graph for this function:

_reasoning_tokens()

int oc_chat.benchmarks.metrics._reasoning_tokens ( dict raw_usage )

protected

Extract reasoning tokens (DeepSeek nests them under details).

Primary location: ``completion_tokens_details.reasoning_tokens``.
Fallback: a top-level ``reasoning_tokens`` if a provider emits one.

Definition at line 108 of file metrics.py.

Here is the call graph for this function:

Here is the caller graph for this function:

aggregate_calls()

CallMetrics oc_chat.benchmarks.metrics.aggregate_calls

(

list[CallMetrics]

calls

)

Sum a list of :class:`CallMetrics`. Hit and miss stay separate.

Definition at line 141 of file metrics.py.

Here is the caller graph for this function:

aggregate_run()

CallMetrics oc_chat.benchmarks.metrics.aggregate_run

(

list[TurnMetrics]

turns

)

Sum every call across every turn into the run total.

Definition at line 158 of file metrics.py.

Here is the call graph for this function:

aggregate_turn()

CallMetrics oc_chat.benchmarks.metrics.aggregate_turn

(

TurnMetrics

turn

)

Sum one turn's calls into a single :class:`CallMetrics`.

Definition at line 153 of file metrics.py.

Here is the call graph for this function:

derived_full_miss_cost()

float oc_chat.benchmarks.metrics.derived_full_miss_cost	(	int	prompt_tokens,
		float	miss_rate_per_mtok )

The deterministic cold-turn cost: a full cache-miss of the prompt.

``prompt_tokens x miss-rate`` (rate is $/Mtok). This follows purely
from prompt size, so it proves Plan 135's win #2 (a slim prompt is
cheaper on the guaranteed-cold first turn) regardless of whatever
server cache state the run actually hit.

Definition at line 166 of file metrics.py.

extract_call_metrics()

CallMetrics oc_chat.benchmarks.metrics.extract_call_metrics

(

dict | None

raw_usage

)

Normalise a single raw provider ``usage`` dict into :class:`CallMetrics`.

Never raises: a ``None`` or partial dict yields zero-filled metrics.
Cache hit/miss come from the production helper so they match the
runtime; reasoning tokens come from the raw nested field that
``UsageStats`` does not carry.

Definition at line 120 of file metrics.py.

Here is the call graph for this function: