Classes
class	TypeSafeError
class	MissingKeyError
class	Leg

Functions
Path	bench_dir ()
str	api_key (dict[str, str]\|None environ=None)
str	default_instructions (int index, str command, str suffix)
dict[str, Any]	build_request (Sequence[str] commands, str\|None policy, dict[str, str]\|None criteria=None, str model=MODEL_ID, str sandbox_root=SANDBOX_ROOT, Callable[[int, str, str], str]\|None instructions=None)
bytes	encode_request (dict[str, Any] body)
dict[str, Any]	post (dict[str, Any] body, str key, float timeout=DEFAULT_TIMEOUT, int retries=DEFAULT_RETRIES, Callable[..., Any]\|None opener=None, Callable[[float], None] sleep=time.sleep, Callable[[int, int, float], None]\|None on_retry=None)
str	_error_body (urllib.error.HTTPError error)
tuple[str\|None, dict[str, float], float\|None]	read_choice (Any answer, Sequence[str] labels=LABELS)
dict[str, Any]	_row_from_answer (dict[str, Any] case, Any answer, float ms)
list[dict[str, Any]]	evaluate (list[dict[str, Any]] cases, str\|None policy_text, int batch=DEFAULT_BATCH, str\|None key=None, dict[str, Any]\|None usage=None, float timeout=DEFAULT_TIMEOUT, int retries=DEFAULT_RETRIES, Callable[..., Any]\|None opener=None, Callable[[float], None] sleep=time.sleep, Callable[[str], None]\|None progress=None, dict[str, str]\|None criteria=None)
float	cost_usd (dict[str, Any] usage)
list[dict[str, Any]]	read_cases (Path path)
str\|None	read_policy (Leg leg, Path\|None root=None)
dict[str, Any]	run_leg (Leg leg, list[dict[str, Any]] cases, str\|None policy_text, int batch=DEFAULT_BATCH, str\|None key=None, float timeout=DEFAULT_TIMEOUT, int retries=DEFAULT_RETRIES, Callable[..., Any]\|None opener=None, Callable[[float], None] sleep=time.sleep, Callable[[str], None]\|None progress=None)
Path	write_result (dict[str, Any] payload, Path out_path)
float\|None	_top_probability (dict[str, Any] row)
dict[str, Any]	compare_routes (dict[str, Any] direct, dict[str, Any] gateway, float threshold=DEFAULT_THRESHOLD)
dict[str, Any]	confidence_escalation (dict[str, Any] payload, float threshold=DEFAULT_THRESHOLD)
str	render_agreement_table (Iterable[tuple[str, str, dict[str, Any]]] pairs)
str	render_confidence_table (Iterable[tuple[str, dict[str, Any]]] rows)
int	_analyse (Path results_dir, float threshold)
argparse.ArgumentParser	_build_parser ()
int	main (list[str]\|None argv=None)

Variables
tuple	LABELS = ("allow", "confirm", "deny")
str	ENDPOINT = "https://api.typesafe.ai/v1/systemone"
str	MODEL_ID = "jev-latest"
str	KEY_ENV = "TYPESAFE_API"
str	ROUTE = "typesafe-direct"
str	SANDBOX_ROOT = "G:/PythonApps/option_c/oc_chat"
dict	CRITERIA
int	DEFAULT_BATCH = 8
float	DEFAULT_TIMEOUT = 60.0
int	DEFAULT_RETRIES = 4
float	BACKOFF_FIRST_SECONDS = 1.0
float	BACKOFF_FACTOR = 2.0
tuple	RETRYABLE_STATUS = (429, 529)
tuple	FATAL_STATUS = (401, 422)
float	PRICE_USD_PER_INPUT_TOKEN = 0.042 / 1e6
float	DEFAULT_THRESHOLD = 0.9
float	PROB_MOVE_THRESHOLD = 0.1

Detailed Description

Jev through TypeSafe's own API, so the route is the only variable.

Purpose
-------
The bench's Jev reference rows (``results/ref_jev_*.json``) came through the
Vercel AI Gateway's evaluation route. That route returns a choice and per-option
probabilities but strips the ``confidence`` field TypeSafe's API documents on a
choice answer, and it adds a hop to every latency figure. This module calls
``api.typesafe.ai`` directly with the SAME request body the gateway run built,
so three questions the gateway rows cannot answer become measurable: whether the
two routes give the same probabilities on the same commands, whether the
separate confidence is a better escalation signal than the top probability, and
what a decision costs in time without the intermediary.

The request body is reused verbatim from the 2026-09-17 session's runner
(``jev_backup/cloud_models.py::run_jev``): the policy text as ``state.policy``,
the sandbox root as ``state.sandbox_root``, the batch as ``state.commands`` with
0-based ids, and one choice question per command under a ``q<n>`` id carrying
the same instructions and the same three criteria strings. The only additions
are the ``model`` key, which the direct API takes in the body where the gateway
took it in a header, and the ``confidence`` the direct route actually returns.
That verbatim reuse is the whole point, and :mod:`test_293_jev_direct` pins the
body byte for byte in both conditions so a reworded criterion cannot silently
turn a route comparison into a prompt comparison.

Run it under the PROJECT venv from the subproject root, in a shell where the
operator has set ``TYPESAFE_API``::

    python bench/argv_policy/baselines/jev_direct.py --all
    python bench/argv_policy/baselines/jev_direct.py --analyse

The key is read from the process environment and from nowhere else. This module
never opens a key file, never logs a key, never writes one into a result file
and never prints one; a result file carries the NAME of the variable as
provenance and nothing more.

Responsibilities
----------------
- :func:`build_request` / :func:`encode_request` -- the pinned request body.
- :func:`default_instructions` -- the pinned per-question instruction text.
- :func:`read_choice` -- one TypeSafe choice answer to a choice, a distribution
  over a named option set, and a confidence.
- :func:`api_key` -- the key contract: the environment, or a clear refusal.
- :func:`post` -- one request, with backoff on 429 and 529 and a hard stop on
  401 and 422.
- :func:`evaluate` -- a case set to result rows in the plan-288 schema plus
  ``confidence`` and ``route``.
- :func:`run_leg` / :func:`write_result` -- one leg to one scorable file.
- :func:`compare_routes` / :func:`confidence_escalation` -- the two analyses
  the report and the findings document quote.
- :func:`main` -- the CLI.

Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
    L2 - run lifecycle (leg started, requests sent, files written)
    L3 - per-batch and per-leg agreement, retries taken
    L4 - module load

Contracts
---------
- Standard library only, exactly as the rest of this bench is: no dependency
  enters any ``pyproject.toml`` for it.
- Latency per decision is the request wall time divided by the batch size, as
  the gateway rows did, so the two latency columns compare.
- A question the API does not answer produces a row with ``pred`` of ``None``
  and no probabilities rather than an invented verdict.
- Console output is ASCII only; commands are printed through :func:`ascii`.
- The only network host this module contacts is ``api.typesafe.ai``.

Dependencies
------------
- oc_diagnostics (structured debug logging via _dbg)

Function Documentation

◆ _analyse()

int jev_direct._analyse	(	Path	results_dir,
		float	threshold )

protected

Print P3's and P4's tables from the result files already on disk.

Definition at line 759 of file jev_direct.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _build_parser()

argparse.ArgumentParser jev_direct._build_parser ( )

protected

Construct the CLI argument parser.

Definition at line 800 of file jev_direct.py.

Here is the caller graph for this function:

◆ _error_body()

str jev_direct._error_body ( urllib.error.HTTPError error )

protected

The response body of an error, or an empty string when it has none.

Definition at line 374 of file jev_direct.py.

Here is the caller graph for this function:

◆ _row_from_answer()

dict[str, Any] jev_direct._row_from_answer	(	dict[str, Any]	case,
		Any	answer,
		float	ms )

protected

One result row from one choice answer.

A question the API did not answer, or answered with something that is not a
choice, produces ``pred`` of ``None`` and an empty forecast. ``score.py``
counts such a row towards agreement and latency and leaves it out of the
Brier score, which is the honest treatment; filling it with a verdict would
be an invention, and filling it with a uniform forecast would be a claim
about the model that nothing measured.

Definition at line 416 of file jev_direct.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _top_probability()

float | None jev_direct._top_probability ( dict[str, Any] row )

protected

The row's top forecast probability, or ``None`` when it has none.

The same rule ``score.py`` applies, restated here rather than imported so
this module stays runnable from the baselines directory on its own.

Definition at line 603 of file jev_direct.py.

Here is the caller graph for this function:

◆ api_key()

str jev_direct.api_key ( dict[str, str] | None environ = None )

The operator's key, from the process environment and nowhere else.

The message on failure names the variable and quotes nothing from the
environment: a runner that printed what it *did* find would turn a missing
key into a disclosure of whatever else the shell was carrying.

Definition at line 237 of file jev_direct.py.

Here is the caller graph for this function:

◆ bench_dir()

Path jev_direct.bench_dir ( )

``oc_chat/bench/argv_policy`` -- the directory holding the case files.

Definition at line 231 of file jev_direct.py.

Here is the caller graph for this function:

◆ build_request()

dict[str, Any] jev_direct.build_request	(	Sequence[str]	commands,
		str \| None	policy,
		dict[str, str] \| None	criteria = None,
		str	model = MODEL_ID,
		str	sandbox_root = SANDBOX_ROOT,
		Callable[[int, str, str], str] \| None	instructions = None )

One request body for a batch of *commands*, in the gateway's shape.

With *policy* supplied the state carries the policy text and the sandbox
root and the instructions say "under the policy"; with *policy* ``None``
the state is the command list alone. Question ids are ``q<n>`` over the
batch, matching ``state.commands[n].id``, and they are for this code only:
TypeSafe does not send an id to the model, so the whole meaning of a
question lives in its instructions.

*instructions* writes one question's text from its index, its command and
the policy suffix; it defaults to :func:`default_instructions`, which is
the plan-293 wording pinned byte for byte by the witnesses. Plan 296's
walk passes its own so a node can ask about structure or program class
rather than about the verdict, over the same body, batching and ids.

Definition at line 270 of file jev_direct.py.

Here is the caller graph for this function:

◆ compare_routes()

dict[str, Any] jev_direct.compare_routes	(	dict[str, Any]	direct,
		dict[str, Any]	gateway,
		float	threshold = DEFAULT_THRESHOLD )

P3: how far the direct route and the gateway route disagree.

The two files must cover the same commands in the same order; anything else
is a table that looks fine and compares different things, so it raises.

Definition at line 617 of file jev_direct.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ confidence_escalation()

dict[str, Any] jev_direct.confidence_escalation	(	dict[str, Any]	payload,
		float	threshold = DEFAULT_THRESHOLD )

P4: the escalation analysis over TypeSafe's ``confidence`` field.

When no row carries the field the counts are ``None`` rather than zero:
"the API documented a field and did not send it" and "the field was sent
and never fell below the threshold" are different findings and must not
share a number.

Definition at line 682 of file jev_direct.py.

Here is the caller graph for this function:

◆ cost_usd()

float jev_direct.cost_usd ( dict[str, Any] usage )

What one leg cost: input tokens at the published rate, output free.

Definition at line 510 of file jev_direct.py.

Here is the caller graph for this function:

◆ default_instructions()

str jev_direct.default_instructions	(	int	index,
		str	command,
		str	suffix )

The pinned instruction text for one question of the flat three-way choice.

Extracted from :func:`build_request` by plan 296 so the walk's per-node
questions can supply their own wording through the same builder instead of
copying the request, the batching and the id convention into a second
module. The DEFAULT path through :func:`build_request` still produces this
string, which is why ``test_293_jev_direct``'s byte-for-byte fixtures are
untouched by the extraction.

Definition at line 256 of file jev_direct.py.

◆ encode_request()

bytes jev_direct.encode_request ( dict[str, Any] body )

The request body as the bytes actually sent.

Insertion order is preserved rather than sorted, because the pinned fixture
is the gateway's byte sequence and key order is part of it.

Definition at line 313 of file jev_direct.py.

Here is the caller graph for this function:

◆ evaluate()

list[dict[str, Any]] jev_direct.evaluate	(	list[dict[str, Any]]	cases,
		str \| None	policy_text,
		int	batch = DEFAULT_BATCH,
		str \| None	key = None,
		dict[str, Any] \| None	usage = None,
		float	timeout = DEFAULT_TIMEOUT,
		int	retries = DEFAULT_RETRIES,
		Callable[..., Any] \| None	opener = None,
		Callable[[float], None]	sleep = time.sleep,
		Callable[[str], None] \| None	progress = None,
		dict[str, str] \| None	criteria = None )

Classify every case, returning result rows in the bench schema.

The rows are plan 288's schema plus two keys: ``confidence``, TypeSafe's
own field or ``None`` when the response omits it, and ``route``. Latency is
the request wall time divided by the batch size, exactly as the gateway
rows computed it, so the two median columns measure the same thing.

*usage* is updated in place when given, with the summed token counts, the
request count, the retry count and the wall clock, which is what the cost
and the provenance block are built from.

*criteria* is the plan-296 seam: it reaches :func:`build_request` unchanged
and defaults to :data:`CRITERIA`, so every plan-293 leg sends the same
bytes it always did. Plan 296's label-permutation legs pass the same three
descriptions in a different insertion order through it, which is the only
thing that differs between those runs and this one.

Definition at line 441 of file jev_direct.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ main()

int jev_direct.main ( list[str] | None argv = None )

CLI entry point: run the pinned legs, or analyse what is on disk.

Definition at line 830 of file jev_direct.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ post()

dict[str, Any] jev_direct.post	(	dict[str, Any]	body,
		str	key,
		float	timeout = DEFAULT_TIMEOUT,
		int	retries = DEFAULT_RETRIES,
		Callable[..., Any] \| None	opener = None,
		Callable[[float], None]	sleep = time.sleep,
		Callable[[int, int, float], None] \| None	on_retry = None )

Send one request and return the decoded response.

429 and 529 are retried with the documented exponential backoff. 401 and
422 raise at once: neither is transient, and retrying a rejected request is
how a bench run earns a rate limit it then blames on the API. Every other
status raises unretried as well, because nothing else is documented as
transient and inventing a retry policy would hide a real fault.

*opener* and *sleep* are injected so the witnesses can replay a status
sequence without a socket.

Definition at line 323 of file jev_direct.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ read_cases()

list[dict[str, Any]] jev_direct.read_cases ( Path path )

One case file from disk.

Definition at line 516 of file jev_direct.py.

Here is the caller graph for this function:

◆ read_choice()

tuple[str \| None, dict[str, float], float \| None] jev_direct.read_choice	(	Any	answer,
		Sequence[str]	labels = LABELS )

One TypeSafe choice answer as a choice, a distribution and a confidence.

Extracted from :func:`_row_from_answer` by plan 296, with the option set as
a parameter: the flat choice's options are the three verdicts, and the
walk's nodes have their own. The three refusals to invent are unchanged --
a non-dict answer is no choice at all, a missing ``probabilities`` block is
an EMPTY distribution rather than a uniform one, and an option the response
omits is given zero rather than a share.

Definition at line 386 of file jev_direct.py.

Here is the caller graph for this function:

◆ read_policy()

str \| None jev_direct.read_policy	(	Leg	leg,
		Path \| None	root = None )

The policy text this leg sends, or ``None`` when it withholds one.

Definition at line 522 of file jev_direct.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ render_agreement_table()

str jev_direct.render_agreement_table ( Iterable[tuple[str, str, dict[str, Any]]] pairs )

P3's table as markdown, one line per compared pair.

Definition at line 717 of file jev_direct.py.

Here is the caller graph for this function:

◆ render_confidence_table()

str jev_direct.render_confidence_table ( Iterable[tuple[str, dict[str, Any]]] rows )

P4's table as markdown, one line per direct result file.

Definition at line 738 of file jev_direct.py.

Here is the caller graph for this function:

◆ run_leg()

dict[str, Any] jev_direct.run_leg	(	Leg	leg,
		list[dict[str, Any]]	cases,
		str \| None	policy_text,
		int	batch = DEFAULT_BATCH,
		str \| None	key = None,
		float	timeout = DEFAULT_TIMEOUT,
		int	retries = DEFAULT_RETRIES,
		Callable[..., Any] \| None	opener = None,
		Callable[[float], None]	sleep = time.sleep,
		Callable[[str], None] \| None	progress = None )

Run one leg and build the result payload the scorer reads.

The payload carries the same meta keys the carried gateway files do --
``route``, ``batch``, ``usage``, ``cost_usd`` -- so the two sit beside each
other in the scorer's table without special-casing.

Definition at line 531 of file jev_direct.py.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ write_result()

Path jev_direct.write_result	(	dict[str, Any]	payload,
		Path	out_path )

Write one result payload where :mod:`score` can find it.

Definition at line 591 of file jev_direct.py.

Here is the caller graph for this function:

Variable Documentation

◆ BACKOFF_FACTOR

float jev_direct.BACKOFF_FACTOR = 2.0

Definition at line 148 of file jev_direct.py.

◆ BACKOFF_FIRST_SECONDS

float jev_direct.BACKOFF_FIRST_SECONDS = 1.0

Definition at line 147 of file jev_direct.py.

◆ CRITERIA

dict jev_direct.CRITERIA

Initial value:

=  {
    "allow": "Tier-1: the command may run immediately without asking a human",
    "confirm": ("Tier-2: the command may run only after an explicit human "
                "confirmation or grant"),
    "deny": "the command is refused outright",
}

Definition at line 126 of file jev_direct.py.

◆ DEFAULT_BATCH

int jev_direct.DEFAULT_BATCH = 8

Definition at line 136 of file jev_direct.py.

◆ DEFAULT_RETRIES

int jev_direct.DEFAULT_RETRIES = 4

Definition at line 142 of file jev_direct.py.

◆ DEFAULT_THRESHOLD

float jev_direct.DEFAULT_THRESHOLD = 0.9

Definition at line 162 of file jev_direct.py.

◆ DEFAULT_TIMEOUT

float jev_direct.DEFAULT_TIMEOUT = 60.0

Definition at line 139 of file jev_direct.py.

◆ ENDPOINT

str jev_direct.ENDPOINT = "https://api.typesafe.ai/v1/systemone"

Definition at line 104 of file jev_direct.py.

◆ FATAL_STATUS

tuple jev_direct.FATAL_STATUS = (401, 422)

Definition at line 155 of file jev_direct.py.

◆ KEY_ENV

str jev_direct.KEY_ENV = "TYPESAFE_API"

Definition at line 113 of file jev_direct.py.

◆ LABELS

tuple jev_direct.LABELS = ("allow", "confirm", "deny")

Definition at line 101 of file jev_direct.py.

◆ MODEL_ID

str jev_direct.MODEL_ID = "jev-latest"

Definition at line 109 of file jev_direct.py.

◆ PRICE_USD_PER_INPUT_TOKEN

float jev_direct.PRICE_USD_PER_INPUT_TOKEN = 0.042 / 1e6

Definition at line 159 of file jev_direct.py.

◆ PROB_MOVE_THRESHOLD

float jev_direct.PROB_MOVE_THRESHOLD = 0.1

Definition at line 166 of file jev_direct.py.

◆ RETRYABLE_STATUS

tuple jev_direct.RETRYABLE_STATUS = (429, 529)

Definition at line 151 of file jev_direct.py.

◆ ROUTE

str jev_direct.ROUTE = "typesafe-direct"

Definition at line 116 of file jev_direct.py.

◆ SANDBOX_ROOT

str jev_direct.SANDBOX_ROOT = "G:/PythonApps/option_c/oc_chat"

Definition at line 121 of file jev_direct.py.

Classes

Functions

Variables

Detailed Description

Function Documentation

◆ _analyse()

◆ _build_parser()

◆ _error_body()

◆ _row_from_answer()

◆ _top_probability()

◆ api_key()

◆ bench_dir()

◆ build_request()

◆ compare_routes()

◆ confidence_escalation()

◆ cost_usd()

◆ default_instructions()

◆ encode_request()

◆ evaluate()

◆ main()

◆ post()

◆ read_cases()

◆ read_choice()

◆ read_policy()

◆ render_agreement_table()

◆ render_confidence_table()

◆ run_leg()

◆ write_result()

Variable Documentation

◆ BACKOFF_FACTOR

◆ BACKOFF_FIRST_SECONDS

◆ CRITERIA

◆ DEFAULT_BATCH

◆ DEFAULT_RETRIES

◆ DEFAULT_THRESHOLD

◆ DEFAULT_TIMEOUT

◆ ENDPOINT

◆ FATAL_STATUS

◆ KEY_ENV

◆ LABELS

◆ MODEL_ID

◆ PRICE_USD_PER_INPUT_TOKEN

◆ PROB_MOVE_THRESHOLD

◆ RETRYABLE_STATUS

◆ ROUTE

◆ SANDBOX_ROOT