Classes
class	CombineError
class	Decision
class	SweepRow
class	_PayloadCache

Functions
float\|None	top_probability (Row row)
bool	is_unsure (Row row, float threshold)
dict[str, float]\|None	_usable_probs (Row row)
int	_rank (str prediction)
Decision	_keep_first (Row first)
Decision	rule_escalate (Row first, Row\|None second, float threshold)
Decision	rule_replace (Row first, Row\|None second, float threshold)
Decision	rule_stricter (Row first, Row\|None second, float threshold)
Decision	rule_disagree (Row first, Row\|None second, float threshold)
dict[str, Row]	_index_by_command (list[Row] rows, str label)
list[Row\|None]	join_rows (list[Row] first_rows, list[Row]\|None second_rows, str first_label, str second_label)
float\|None	_effective_ms (Row first, Row\|None second, bool consulted)
str	_threshold_tag (float threshold)
str	_digest (Path path)
str	_relative_path (Path path, Path bench_dir)
list[Row]	combine_rows (list[Row] first_rows, list[Row\|None] seconds, str rule, float threshold)
Payload	build_payload (Payload first_payload, Payload\|None second_payload, *, str rule, float threshold, Path first_path, Path\|None second_path, Path bench_dir, str date)
Payload	load_payload (Path path)
SweepRow	_sweep_row (_PayloadCache cache, str first, str second, str rule, float threshold, Path bench_dir)
str	_ms_cell (float\|None value)
str	_cost_cell (float\|None value)
str	_threshold_table (list[SweepRow] rows)
int	_confirm_issued (Score score)
list[tuple[tuple[object,...], list[SweepRow]]]	_grouped (list[SweepRow] rows, Callable[[SweepRow], tuple[object,...]] key)
str	_repeat_rows_table (list[SweepRow] rows)
str	_repeat_range_table (list[SweepRow] rows)
str	_threshold_range_table (list[SweepRow] rows)
str	_range_cell (list[int] values)
str	render_sweep (Path results_dir, Path bench_dir)
argparse.ArgumentParser	_build_parser ()
int	_run_sweep (argparse.Namespace args, Path bench_dir)
int	_run_combination (argparse.Namespace args, Path bench_dir)
int	main (list[str]\|None argv=None)

Variables
	Row = dict[str, Any]
	Payload = dict[str, Any]
tuple	RULES = ("escalate", "replace", "stricter", "disagree")
tuple	CONSULTING_RULES = ("replace", "stricter", "disagree")
str	ESCALATION_VERDICT = "confirm"
int	UNKNOWN_RANK = max(PERMISSIVENESS.values()) + 1
float	DEFAULT_THRESHOLD = 0.9
tuple	SWEEP_FIRSTS
tuple	SWEEP_SECONDS_HARD
tuple	SWEEP_SECONDS_HAND = ("jev_direct_hand", "jev_direct_hand_ref")
tuple	SWEEP_THRESHOLDS = (0.5, 0.7, 0.8, 0.9, 0.95, 0.99)
dict	RULE_FUNCTIONS

Detailed Description

Compose two committed result files into one: the first decides, the second is
consulted only where the first is unsure.

Purpose
-------
Every leg of this bench so far has measured one model alone. The escalation
column says what a model would hand to a human below a confidence threshold,
and plan 293 showed that Jev, asked to flag below 0.9, catches every one of its
own permission-granting errors. The fine-tuned encoders beat Jev on agreement
at half the latency and no per-call cost, and at 0.9 they hand 42 to 51 of 300
hard-set decisions upwards. Nobody has measured what happens when those
escalated decisions go to Jev instead of to a human.

Every piece of that measurement is already on disk, so this is a join rather
than a run: the encoder result files and the Jev result files carry the same
commands in the same order, and a combination is a pure function of two rows
and a threshold. Nothing here opens a socket, loads a checkpoint or touches a
GPU; the two inputs are read-only and the output is a result file in the bench
schema that :mod:`score` reads unchanged.

A row is UNSURE when the first model's top probability is strictly below the
threshold or the row has no usable ``probs``. At or above the threshold every
rule keeps the first model's verdict and ``probs`` untouched, so the four rules
differ only in what they do with the unsure rows:

``escalate``
    The unsure row becomes ``confirm`` with no ``probs`` and nothing is
    consulted. This is the CONTROL: the encoder alone with a human behind it,
    which is the operating point plans 293 and 300 already recommend.
``replace``
    The unsure row takes the second model's verdict and ``probs``.
``stricter``
    The unsure row takes whichever of the two verdicts grants LESS permission,
    with the ``probs`` of the model whose verdict was taken.
``disagree``
    The unsure row keeps the first's verdict and ``probs`` where the two agree,
    and becomes ``confirm`` with no ``probs`` where they differ.

``--sweep`` writes ``results/combined_sweep.md``: the same compositions at six
thresholds, and the movement across the repeated Jev runs, scored in memory so
that a threshold curve costs no committed files. It carries no generation date
precisely so that a second run reproduces it byte for byte.

Run it under the project venv from the subproject root::

    python bench/argv_policy/combine.py --first results/encoder_small-7-c299_hard.json \\
        --second results/jev_direct_hard.json --rule replace --threshold 0.9 \\
        --out results/combined_encoder_small-7-c299_replace_t90_hard.json

    python bench/argv_policy/combine.py --sweep

Responsibilities
----------------
- :func:`rule_escalate` / :func:`rule_replace` / :func:`rule_stricter` /
  :func:`rule_disagree` -- the four rules, each a pure function of one first
  row, one second row and the threshold.
- :func:`join_rows` -- the command-keyed join, and the refusal when the two
  files do not carry the same commands.
- :func:`build_payload` -- two payloads to one result payload with provenance.
- :func:`render_sweep` -- the generated markdown of the threshold sweep and the
  repeated-run movement.
- :func:`main` -- the CLI.

Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
    L2 - combination lifecycle (files read, file written, sweep rendered)
    L3 - per-combination counts (consulted, disagreeing)
    L4 - module load

Contracts
---------
- Read-only in both inputs: a combination never rewrites a result file, and the
  only paths written are the ``--out`` target and the sweep document.
- No network, no torch, no key: the Jev files ARE the Jev runs.
- The permission ordering is :data:`score.PERMISSIVENESS` itself, imported
  rather than restated, so ``stricter`` can never disagree with the direction
  columns the scorer reports.
- A threshold comparison is STRICT: a top probability exactly equal to the
  threshold is sure, which is the same reading ``score.py`` uses for its
  escalation count.
- Provenance stores each input path RELATIVE to the bench directory when it
  lies inside it, so a committed combination is reproducible from any checkout
  rather than naming the worktree it was made in.
- Console output is ASCII only; commands are printed through :func:`ascii`.

Dependencies
------------
- oc_diagnostics (structured debug logging via _dbg)
- score (the bench scorer, for PERMISSIVENESS and for the sweep's arithmetic)

Function Documentation

◆ _build_parser()

argparse.ArgumentParser combine._build_parser ( )

protected

Construct the CLI argument parser.

Definition at line 767 of file combine.py.

Here is the caller graph for this function:

◆ _confirm_issued()

int combine._confirm_issued ( Score score )

protected

How many ``confirm`` verdicts one scored combination issued.

Definition at line 572 of file combine.py.

◆ _cost_cell()

str combine._cost_cell ( float | None value )

protected

A cost in US dollars, to four decimals, or a dash.

Definition at line 549 of file combine.py.

◆ _digest()

str combine._digest ( Path path )

protected

The SHA-256 of one input file, which is what pins a combination to its inputs.

Definition at line 363 of file combine.py.

Here is the caller graph for this function:

◆ _effective_ms()

float \| None combine._effective_ms	(	Row	first,
		Row \| None	second,
		bool	consulted )

protected

The latency of the combined decision: the first's, plus the second's when asked.

Definition at line 347 of file combine.py.

Here is the caller graph for this function:

◆ _grouped()

list[tuple[tuple[object, ...], list[SweepRow]]] combine._grouped	(	list[SweepRow]	rows,
		Callable[[SweepRow], tuple[object, ...]]	key )

protected

Group scored rows by a key, in the order the keys first appear.

Grouping on the row's OWN fields rather than on the module constants is
what keeps a range table honest: a key spelled by hand can miss every row
and print an empty table, which is exactly what a reader would read as "no
movement between runs".

Definition at line 578 of file combine.py.

Here is the caller graph for this function:

◆ _index_by_command()

dict[str, Row] combine._index_by_command	(	list[Row]	rows,
		str	label )

protected

Index one file's rows by command, refusing a duplicate.

A repeated command would make the join ambiguous -- two truths, two
forecasts, and no rule for which one wins -- so it is refused here rather
than resolved silently. No case file on this bench carries one.

Definition at line 295 of file combine.py.

Here is the caller graph for this function:

◆ _keep_first()

Decision combine._keep_first ( Row first )

protected

The sure case, which every rule shares: the first model, untouched.

Definition at line 218 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _ms_cell()

str combine._ms_cell ( float | None value )

protected

A latency in milliseconds, to one decimal, or a dash.

Definition at line 543 of file combine.py.

◆ _range_cell()

str combine._range_cell ( list[int] values )

protected

``n`` when every run agrees, ``lo-hi`` when they do not.

Definition at line 652 of file combine.py.

◆ _rank()

int combine._rank ( str prediction )

protected

How much permission one verdict grants, under the scorer's own ordering.

Definition at line 212 of file combine.py.

Here is the caller graph for this function:

◆ _relative_path()

str combine._relative_path	(	Path	path,
		Path	bench_dir )

protected

One input path, relative to the bench directory when it lies inside it.

A committed combination names ``results/jev_direct_hard.json`` rather than
the worktree it happened to be made in, so anyone can rerun it from their
own checkout and compare digests instead of paths.

Definition at line 369 of file combine.py.

Here is the caller graph for this function:

◆ _repeat_range_table()

str combine._repeat_range_table ( list[SweepRow] rows )

protected

The range each (first, rule) covers across the repeated runs at one threshold.

Definition at line 610 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _repeat_rows_table()

str combine._repeat_rows_table ( list[SweepRow] rows )

protected

Every (first, rule, run) reading at one threshold, one line each.

Definition at line 595 of file combine.py.

Here is the caller graph for this function:

◆ _run_combination()

int combine._run_combination	(	argparse.Namespace	args,
		Path	bench_dir )

protected

The single-combination path: compose two files and write the result.

Definition at line 816 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _run_sweep()

int combine._run_sweep	(	argparse.Namespace	args,
		Path	bench_dir )

protected

The ``--sweep`` path: render the document and write it.

Definition at line 801 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _sweep_row()

SweepRow combine._sweep_row	(	_PayloadCache	cache,
		str	first,
		str	second,
		str	rule,
		float	threshold,
		Path	bench_dir )

protected

Compose and score one combination in memory, writing nothing.

Definition at line 527 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _threshold_range_table()

str combine._threshold_range_table ( list[SweepRow] rows )

protected

The same range, at every swept threshold, for reading an operating point.

A recommendation made at one threshold needs its noise band at THAT
threshold. The pinned table above is at 0.90 because that is where the
plan's main tables sit; this one is what says whether a point chosen
elsewhere on the curve survives a second run of the second model.

Definition at line 627 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _threshold_table()

str combine._threshold_table ( list[SweepRow] rows )

protected

The threshold sweep: one line per first, rule and threshold.

Definition at line 555 of file combine.py.

Here is the caller graph for this function:

◆ _threshold_tag()

str combine._threshold_tag ( float threshold )

protected

The threshold as the two digits the file names and model names carry.

Definition at line 357 of file combine.py.

◆ _usable_probs()

dict[str, float] | None combine._usable_probs ( Row row )

protected

A copy of the row's ``probs``, or ``None`` when it has none worth carrying.

Definition at line 203 of file combine.py.

Here is the caller graph for this function:

◆ build_payload()

Payload combine.build_payload	(	Payload	first_payload,
		Payload \| None	second_payload,
		*	,
		str	rule,
		float	threshold,
		Path	first_path,
		Path \| None	second_path,
		Path	bench_dir,
		str	date )

Two loaded result payloads to one combined result payload.

The output is a bench result file and nothing more exotic: ``score.py``
reads it with no change, and the three fields it does not read --
``consulted``, ``first_pred``, ``second_pred`` -- are there so a reader can
take any row apart without rerunning the combination.

Definition at line 409 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ combine_rows()

list[Row] combine.combine_rows	(	list[Row]	first_rows,
		list[Row \| None]	seconds,
		str	rule,
		float	threshold )

Apply one rule down the two joined row lists, in the first file's order.

Definition at line 384 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ is_unsure()

bool combine.is_unsure	(	Row	row,
		float	threshold )

Whether the first model is unsure enough on this row to be combined.

Strictly below the threshold, so a top probability exactly equal to it is
SURE and the row passes through untouched. A row with no usable forecast is
unsure: a model that produced no distribution has expressed no confidence,
and reading that as certainty is the one direction this must not round in.

Definition at line 190 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ join_rows()

list[Row \| None] combine.join_rows	(	list[Row]	first_rows,
		list[Row] \| None	second_rows,
		str	first_label,
		str	second_label )

Line the second file's rows up with the first file's, in the first's order.

Returns one entry per first row: the matching second row, or ``None`` for
every row when there is no second file. Refuses when the two files do not
carry exactly the same commands, or when they disagree about the truth of
one -- both of those mean the two runs are not the same measurement, and
every figure downstream would be a join artefact rather than a reading.

Definition at line 312 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ load_payload()

Payload combine.load_payload ( Path path )

Read one result file, refusing a path that is not there.

Definition at line 467 of file combine.py.

Here is the caller graph for this function:

◆ main()

int combine.main ( list[str] | None argv = None )

CLI entry point: one combination, or the whole sweep document.

Definition at line 848 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ render_sweep()

str combine.render_sweep	(	Path	results_dir,
		Path	bench_dir )

The whole generated sweep document.

Deliberately DATELESS. The document is a pure function of the committed
result files it names, and it carries their digests instead of a timestamp,
so a second run reproduces it byte for byte and a diff means an input
moved.

Definition at line 659 of file combine.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ rule_disagree()

Decision combine.rule_disagree	(	Row	first,
		Row \| None	second,
		float	threshold )

An unsure row survives agreement and goes to a human on disagreement.

Definition at line 271 of file combine.py.

Here is the call graph for this function:

◆ rule_escalate()

Decision combine.rule_escalate	(	Row	first,
		Row \| None	second,
		float	threshold )

The control: an unsure row goes to a human, and nothing is consulted.

``second`` is accepted and deliberately unread. Every rule shares one
signature so that the sweep can run all four over the same joined rows, and
the control is the rule that has a second opinion available and declines to
ask for it -- which is exactly the operating point it controls for.

Definition at line 224 of file combine.py.

Here is the call graph for this function:

◆ rule_replace()

Decision combine.rule_replace	(	Row	first,
		Row \| None	second,
		float	threshold )

An unsure row takes the second model's verdict and its ``probs``.

Definition at line 239 of file combine.py.

Here is the call graph for this function:

◆ rule_stricter()

Decision combine.rule_stricter	(	Row	first,
		Row \| None	second,
		float	threshold )

An unsure row takes whichever verdict grants less permission.

The ``probs`` travel with the verdict that was taken, so the forecast a row
carries is always the forecast of the model that decided it. Where the two
agree -- including where they agree on a verdict neither label set knows --
the first model's forecast is kept, because it is the model that would have
answered anyway.

Definition at line 250 of file combine.py.

Here is the call graph for this function:

◆ top_probability()

float | None combine.top_probability ( Row row )

The row's top forecast probability, or ``None`` when it has none.

The same reading ``score.py`` takes of the same field, and the tests assert
that equality on a battery of rows rather than trusting this comment: a row
whose ``probs`` is missing, empty or not a mapping has no forecast at all,
and is not a zero-confidence one.

Definition at line 174 of file combine.py.

Here is the caller graph for this function:

Variable Documentation

◆ CONSULTING_RULES

tuple combine.CONSULTING_RULES = ("replace", "stricter", "disagree")

Definition at line 126 of file combine.py.

◆ DEFAULT_THRESHOLD

float combine.DEFAULT_THRESHOLD = 0.9

Definition at line 138 of file combine.py.

◆ ESCALATION_VERDICT

str combine.ESCALATION_VERDICT = "confirm"

Definition at line 129 of file combine.py.

◆ Payload

combine.Payload = dict[str, Any]

Definition at line 119 of file combine.py.

◆ Row

combine.Row = dict[str, Any]

Definition at line 118 of file combine.py.

◆ RULE_FUNCTIONS

dict combine.RULE_FUNCTIONS

Initial value:

=  {
    "escalate": rule_escalate,
    "replace": rule_replace,
    "stricter": rule_stricter,
    "disagree": rule_disagree,
}

Definition at line 287 of file combine.py.

◆ RULES

tuple combine.RULES = ("escalate", "replace", "stricter", "disagree")

Definition at line 122 of file combine.py.

◆ SWEEP_FIRSTS

tuple combine.SWEEP_FIRSTS

Initial value:

=  (
    "encoder_small-7-c297",
    "encoder_small-7-c299",
    "encoder_xsmall-7-c297-sc4-dw1",
)

Definition at line 141 of file combine.py.

◆ SWEEP_SECONDS_HAND

tuple combine.SWEEP_SECONDS_HAND = ("jev_direct_hand", "jev_direct_hand_ref")

Definition at line 155 of file combine.py.

◆ SWEEP_SECONDS_HARD

tuple combine.SWEEP_SECONDS_HARD

Initial value:

=  (
    "jev_direct_hard",
    "jev_direct_hard_rep1",
    "jev_direct_hard_rep2",
)

Definition at line 148 of file combine.py.

◆ SWEEP_THRESHOLDS

tuple combine.SWEEP_THRESHOLDS = (0.5, 0.7, 0.8, 0.9, 0.95, 0.99)

Definition at line 158 of file combine.py.

◆ UNKNOWN_RANK

int combine.UNKNOWN_RANK = max(PERMISSIVENESS.values()) + 1

Definition at line 134 of file combine.py.

Classes

Functions

Variables

Detailed Description

Function Documentation

◆ _build_parser()

◆ _confirm_issued()

◆ _cost_cell()

◆ _digest()

◆ _effective_ms()

◆ _grouped()

◆ _index_by_command()

◆ _keep_first()

◆ _ms_cell()

◆ _range_cell()

◆ _rank()

◆ _relative_path()

◆ _repeat_range_table()

◆ _repeat_rows_table()

◆ _run_combination()

◆ _run_sweep()

◆ _sweep_row()

◆ _threshold_range_table()

◆ _threshold_table()

◆ _threshold_tag()

◆ _usable_probs()

◆ build_payload()

◆ combine_rows()

◆ is_unsure()

◆ join_rows()

◆ load_payload()

◆ main()

◆ render_sweep()

◆ rule_disagree()

◆ rule_escalate()

◆ rule_replace()

◆ rule_stricter()

◆ top_probability()

Variable Documentation

◆ CONSULTING_RULES

◆ DEFAULT_THRESHOLD

◆ ESCALATION_VERDICT

◆ Payload

◆ Row

◆ RULE_FUNCTIONS

◆ RULES

◆ SWEEP_FIRSTS

◆ SWEEP_SECONDS_HAND

◆ SWEEP_SECONDS_HARD

◆ SWEEP_THRESHOLDS

◆ UNKNOWN_RANK