Classes
class	Model

Functions
int	feature_index (int size, str gram)
dict[int, float]	features (str command)
Model	train (list[dict[str, str]] rows, int epochs=DEFAULT_EPOCHS, int seed=DEFAULT_SEED, float learning_rate=DEFAULT_LEARNING_RATE, float l2=DEFAULT_L2, int batch=DEFAULT_BATCH)
list[dict[str, object]]	evaluate (Model model, list[dict[str, str]] cases)
float	_accuracy (Model model, list[dict[str, str]] rows)
argparse.ArgumentParser	_build_parser ()
int	main (list[str]\|None argv=None)

Variables
tuple	LABELS = ("allow", "confirm", "deny")
tuple	NGRAM_SIZES = (1, 2, 3, 4)
int	HASH_SPACE = 1 << 16
str	START_SENTINEL = "\x02"
str	END_SENTINEL = "\x03"
int	DEFAULT_EPOCHS = 50
int	DEFAULT_SEED = 7
float	DEFAULT_LEARNING_RATE = 40.0
int	DEFAULT_L2 = 1e-6
int	DEFAULT_BATCH = 32

Detailed Description

The token floor: hashed character n-grams into a multinomial logistic regression.

Purpose
-------
This is the number every later result on this bench is read against. The argv
gate is, at bottom, a set of rules over the characters of a command string, so
before anyone reports that an embedding index or a fine-tuned encoder "learned
the policy", the bench needs to know what a bag of character n-grams and plain
gradient descent already achieve. If the floor sits near the gate, the task is a
token rule and plan 290's encoder has to be read against the floor, not against
chance.

Features are character n-grams for n in 1 to 4 over the command padded with
start and end sentinels, hashed into a fixed space with CRC-32 (Python's own
``hash`` is salted per process, so a model trained in one run would be
meaningless in the next), counted, and L2-normalised so a long command does not
simply outvote a short one. The model is a three-class softmax trained by
minibatch gradient descent with L2 regularisation. No fitting library is used:
the gradient is written out.

Run it under the PROJECT venv from the subproject root::

    python bench/argv_policy/baselines/token_classifier.py

Implementation note (a deliberate departure from plan 288 P4, which reads
"numpy or torch"): this is standard-library only. The project venv has neither
numpy nor torch, installing one is the orchestrator's hard stop, and the scratch
harness venv that does have them cannot import ``oc_diagnostics``, so a module
run there could not be Option C compliant. Pure Python is also the better fit
for the shape of the problem: the features are sparse -- a thirty-character
command touches roughly 120 of 65536 columns -- so a dense design matrix would
be about three gigabytes of mostly zeros, and the sparse inner loop this module
runs is the same arithmetic a sparse numpy implementation would perform.

Responsibilities
----------------
- :func:`feature_index` / :func:`features` -- the hashed n-gram representation.
- :class:`Model` -- the weights and the softmax forecast.
- :func:`train` -- minibatch gradient descent over a corpus.
- :func:`evaluate` -- a case set to result rows in the bench schema.
- :func:`main` -- train, evaluate both case sets, write both result files.

Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
    L2 - training lifecycle (start, per-epoch loss, finish, files written)
    L3 - per-epoch accuracy and per-case-set agreement
    L4 - module load

Contracts
---------
- Deterministic: a corpus, a seed and the hyperparameters fix the weights, so
  the committed result files are reproducible from the committed corpus.
- Read-only with respect to the corpus and the case sets.
- Console output is ASCII only; commands are printed through :func:`ascii`.
- Standard library only, by the reasoning in the implementation note above.

Dependencies
------------
- oc_diagnostics (structured debug logging via _dbg)

Function Documentation

◆ _accuracy()

float token_classifier._accuracy	(	Model	model,
		list[dict[str, str]]	rows )

protected

Share of *rows* the model gets right.

Definition at line 255 of file token_classifier.py.