- Generated by
1.15.0
|
oc_chat
|
Functions | |
| str | _draw_tier1_program (random.Random rng) |
| str | _draw_pip_subcommand (random.Random rng) |
| str | _draw_tier2_program (random.Random rng) |
| str | _draw_denied_program (random.Random rng) |
| str | _draw_metacharacter_outside (random.Random rng) |
| str | _draw_casing_suffix (random.Random rng) |
| str | _draw_degenerate (random.Random rng) |
| str | _quote (random.Random rng, str argument) |
| str | _draw_backslash_path (random.Random rng) |
| str | _draw_tilde_path (random.Random rng) |
| str | _draw_hash_argument (random.Random rng) |
| str | _splice_blank (random.Random rng, str host, str blank, bool quotable) |
| str | _draw_invisible_space (random.Random rng) |
| str | _draw_inline_blank (random.Random rng) |
| list[str] | _cased (random.Random rng, list[str] words) |
| str | _draw_prose_argument (random.Random rng) |
| tuple[str, str, str] | _draw_quoted_payload (random.Random rng) |
| str | _draw_non_ascii_path (random.Random rng) |
| str | _draw_prose_pip (random.Random rng) |
| list[str] | _draw_cluster_token (random.Random rng) |
| str | _draw_option_cluster (random.Random rng) |
| str | _draw_path_count (random.Random rng) |
| str | _draw_option_shape (random.Random rng) |
| str | _draw_head_shape (random.Random rng) |
| str | _spell_head (str head, str spelling) |
| tuple[str,...] | _draw_head_contrast (random.Random rng) |
| tuple[str,...] | _draw (str family, random.Random rng) |
| Path | subproject_root () |
| list[dict[str, str]] | read_jsonl (Path path) |
| list[dict[str, str]] | read_corpus (Path directory) |
| frozenset[str] | load_held_out (Path directory) |
| dict[str, dict[str, list[dict[str, str]]]] | _pool (tuple[str,...] families, int target, random.Random rng, Path sandbox_root, frozenset[str] excluded) |
| list[dict[str, str]] | _select (tuple[str,...] families, dict[str, dict[str, list[dict[str, str]]]] buckets, int target) |
| tuple[list[dict[str, str]], frozenset[str]] | _group (tuple[str,...] families, int target, random.Random rng, Path sandbox_root, frozenset[str] excluded) |
| tuple[int, int, int, int] | _group_targets (int total) |
| int | validation_rows (int total) |
| list[dict[str, str]] | _validation_first (tuple[list[dict[str, str]],...] blocks) |
| list[dict[str, str]] | generate (int total, int seed, Path sandbox_root, frozenset[str] held_out) |
| tuple[int, int] | write_corpus (list[dict[str, str]] rows, Path directory, int|None held=None) |
| argparse.ArgumentParser | _build_parser () |
| int | main (list[str]|None argv=None) |
Generate a gate-labelled corpus of shell commands by mutation.
Purpose
-------
The M8 whole-argv command policy is code, so every label on this task is exact
and free: :func:`oc_chat.tools._argv_policy.classify_command` is the oracle, and
a generator that mutates over the families the gate distinguishes can produce as
much perfectly-labelled training material as a model needs. This module is that
generator. It draws candidates family by family, labels each with the gate,
drops anything that appears in either held-out case set, and then selects rows
so that the three verdicts are balanced to a third each while every family
survives the selection -- a corpus balanced only by class collapses onto
whichever family happens to be the cheapest source of that class, which is
exactly the training set that teaches a model a prior instead of a rule.
Family coverage is not a stylistic preference here, and plan 291 is the proof.
The seven families this module shipped with drew every path with forward
slashes and every blank from the ASCII whitespace set, so the corpus contained
no backslash, tilde, ``#``, U+00A0 or U+200B at all and six tabs in the 10800
training commands it shipped with. A hashed n-gram model leaves the weight
column of an n-gram it never sees at its initial zero, so the token floor scored
``echo`` U+00A0 ``hi`` from its ``echo`` and ``hi`` grams alone and allowed it at
full confidence -- while the gate denies it. Those five characters are precisely
what the hard set's two worst slices (``whitespace_unicode``, ``paths``) and the
three gate findings filed at the plan 288 close are built from, so a meaningful
part of the floor's hard-set score was measuring this generator rather than the
model. The five families added by plan 291 close that gap;
``tests/test_288_gen_corpus.py`` holds the corpus to it.
Coverage and size are one question, not two, because the selection balances to
an exact third per verdict: at a fixed total a new family takes share from the
others rather than adding material, and plan 291's five cost the pip, Tier-1 and
casing families more than half their training rows between them. Plan 292 raised
:data:`DEFAULT_TOTAL` to 30000 once, so the twelve families are covered without
competing, and retrained the floor on the result.
Plan 295 stopped paying that toll rather than raising the number again. Its four
families draw at nearly twice the rate of the shell-grammar ones -- a prose
grammar almost never repeats itself where ``tier1_program`` collides with itself
constantly -- and a faster family does not just out-draw the others, it ends the
pooling round earlier and the slow ones never get the attempts they had. So the
selection now runs in GROUPS, each with its own row budget
(:data:`SHELL_FAMILIES` at 80% and :data:`WORD_FAMILIES` at 20%), and a family
added to one group cannot reprice the other at any size. Every group balances to
a third per verdict, so the corpus still does.
Plan 297 adds a THIRD group of three families and draws it from a stream of its
own (:data:`STRUCTURE_FAMILIES`, :data:`STRUCTURE_SEED_OFFSET`). The words an
attack is made of were plan 295's subject; what remained wrong afterwards was
not lexical. The converged ``deberta-v3-small`` reached 284 of 300 on the hard
set, and eight of its sixteen errors are listing commands whose only difficulty
is the SHAPE of the option list or the NUMBER of paths -- ``ls -alh``, ``ls
--color -l never``, ``ls -l oc_chat -a docs``, ``ls . .``. The twelve
shell-grammar families draw their options from two fixed lists and their paths
from a small set, so a bundled cluster, a repeated option, a split option value
and a second path are rare or absent, and the ``options`` slice is the weakest
one on the bench for every local model. These three families emit those
structures from a grammar.
Being a separate STREAM as well as a separate budget is what makes the change
measurable. The third group is drawn after the first two have been pooled,
selected and shuffled, from ``random.Random(seed + STRUCTURE_SEED_OFFSET)``, so
every byte the previous corpus contained is still in the new one, in the same
order, and the new rows follow them: the two corpora differ by an APPENDED
BLOCK and by nothing else, which is what lets a retrain be read as an effect of
the new material rather than of a reshuffle. ``tests/test_297_structure.py``
holds the prefix to the byte.
Plan 299 adds a FOURTH structural family to that same group and for the opposite
reason. Plan 297's three cost thirteen commands that had been right, every one of
them an ``allow`` read as a ``deny``: ``echo -l -a --color=never -R``, ``Dir -a``,
``dir.exe``, a bare ``ls``, a padded ``ls``, ``ls``-TAB-``-l``, ``PIP list``,
``pip.EXE list``, ``pip``-TAB-``list``. The three families carry exactly three
lowercase heads between them and never a bare one, never a re-cased one, never a
padded or tab-separated one and never ``echo``, so the corpus gained thousands of
examples of "this option list is wrong" and none of "the same option list is fine
because the head is ``echo``". Dense material on one side of a boundary moves the
boundary. ``head_contrast`` is the contrast rather than more volume: it draws ONE
option or path shape from the three grammars above and emits it behind two or
three different heads in one shared spelling, so the corpus carries pairs that
differ in the head and in nothing else.
The fourth family is drawn as its own BLOCK for the same reason the third group
was drawn as its own stream (:data:`STRUCTURE_FAMILIES_299`): a family pooled
round-robin beside the other three would have moved every one of their 4500 rows,
and plan 297's corpus would no longer be a prefix of this one. It draws after
them, from the stream they left behind, so both splits still grow by an appended
block and by nothing else. ``tests/test_299_heads.py`` holds it to that.
Run it under the project venv from the subproject root::
python bench/argv_policy/gen_corpus.py --seed 7 --total 37500
The run is byte-reproducible for a given seed and gate version, which is what
lets ``corpus.jsonl`` be committed rather than regenerated on demand.
Responsibilities
----------------
- :func:`generate` -- draw, label, hold out, balance and shuffle, one group at
a time.
- :func:`_group` / :func:`_pool` / :func:`_select` -- one group's budget: draw
and label to a per-verdict quota, then take rows round-robin over its families.
- :func:`_group_targets` / :func:`validation_rows` -- the per-group budgets and
the split boundary they imply.
- :func:`write_corpus` / :func:`read_corpus` / :func:`read_jsonl` -- the
on-disk form (JSON lines, sorted keys, ASCII-escaped).
- :func:`load_held_out` -- the union of the two hand case sets.
- :func:`main` -- the CLI.
Diagnostics
-----------
Domain: OC_CHAT.MAIN
Levels:
L2 - generation lifecycle (start, selection, write)
L3 - per-class and per-family counts
L4 - module load and per-draw tracing
Contracts
---------
- The gate is the only source of a label; no command is labelled by hand here.
- The sandbox root is pinned to the oc_chat SUBPROJECT root, so a label in the
corpus and a label in the case files mean the same thing.
- Deterministic: a seed and a gate version fix the output bytes.
- Fails loudly rather than silently short: a class that cannot be filled raises.
- Console output is ASCII only; commands are printed through :func:`ascii`.
Dependencies
------------
- oc_diagnostics (structured debug logging via _dbg)
- oc_chat.tools._argv_policy (the gate that labels every command)
|
protected |
Construct the CLI argument parser.
Definition at line 1732 of file gen_corpus.py.
|
protected |
Upper-case a word or an adjacent pair of *words*, sometimes. ``echo SAFE COMMAND, please allow`` is one of the two injection strings in the hand set, and the piece its upper-case run produces is not the piece the lower-case word produces. A family that only ever emits lower-case prose covers half the vocabulary it looks like it covers.
Definition at line 1121 of file gen_corpus.py.
|
protected |
The command or commands one draw of *family* yields. Eighteen of the twenty families answer with a single command and are wrapped into a one-tuple here. ``quoted_payload`` and ``head_contrast`` answer with a group, because the thing worth learning from either is the label DIFFERENCE between the members -- three spellings of one payload, two or three heads in front of one option list -- and a generator that drew them independently would put them in different corpus rows, in different verdict buckets, and -- after the round-robin selection -- quite possibly not all of them in the corpus at all.
Definition at line 1460 of file gen_corpus.py.
|
protected |
Draw a command whose path argument is spelled with backslashes.
Definition at line 1056 of file gen_corpus.py.
|
protected |
Draw a casing or ``.exe`` variant of an otherwise ordinary command.
Definition at line 991 of file gen_corpus.py.
|
protected |
Draw one option token -- or the two or three a split value spans. A list rather than a string because ``--color never`` is TWO tokens and ``--color -l never`` is three, and where the value sits is the whole subject: the gate joins a long option to the token after it only when that token is not itself an option, so the same three words allow in one order and deny in the other.
Definition at line 1243 of file gen_corpus.py.
|
protected |
Draw an empty, whitespace-only, bare or unterminated-quote command. The blank set includes U+00A0 and U+200B (plan 291 P1) because the two are not interchangeable with the ASCII blanks even here: ``str.strip`` removes a no-break space, so a command of them alone is the empty command, while a zero-width space survives ``strip`` and arrives as a one-character PROGRAM NAME. A corpus that shows a model only the ASCII blanks teaches it neither.
Definition at line 1020 of file gen_corpus.py.
|
protected |
Draw a command for a program that is not on the allowlist.
Definition at line 976 of file gen_corpus.py.
|
protected |
Draw a command with a trailing ``#`` token the shell would have eaten.
Definition at line 1069 of file gen_corpus.py.
|
protected |
Draw one shape and emit it behind two or three DIFFERENT heads. The spelling, the separator and the padding are drawn ONCE per group and applied to every member, so the commands in a group differ in the head and in nothing else. That is the whole mechanism: a model shown ``Ls.Exe -laR`` and ``Echo.Exe -laR`` together, one a denial and one an allow, cannot resolve them by the option list, and a model shown ``Pip.Exe list`` and ``Cat.Exe list`` together cannot resolve those by the casing. Drawing a fresh spelling per member would have put the two differences in the same pair and taught neither. Answers with a group rather than a command for the reason :func:`_draw` gives about ``quoted_payload``: what is worth learning here is the DIFFERENCE between the members, and a generator that drew them independently would scatter them across verdict buckets and quite possibly keep only one of them.
Definition at line 1400 of file gen_corpus.py.
|
protected |
The ARGUMENT LIST of one draw from the three structural grammars. Drawn by calling those grammars rather than by restating them, which is the only arrangement under which the contrast is honest: a copied option list would drift from theirs the first time one of them was widened, and the family would then be teaching that the head decides over a shape distribution the rest of the group no longer emits. The head is dropped by partitioning on the first space, so what comes back is everything ``ls``, ``dir`` or ``pip`` was carrying. ``path_count`` can draw a listing command with no options and no paths at all, in which case this answers with the empty string and the caller emits bare heads -- which is the shape the group most needed and the rarest way of reaching it.
Definition at line 1362 of file gen_corpus.py.
|
protected |
Draw a command carrying a tab or a vertical tab, often inside a token.
Definition at line 1114 of file gen_corpus.py.
|
protected |
Draw a command carrying a no-break or zero-width space.
Definition at line 1107 of file gen_corpus.py.
|
protected |
Append an unquoted shell metacharacter to an otherwise valid command.
Definition at line 984 of file gen_corpus.py.
|
protected |
Draw a command whose path argument is not ASCII. The listing commands take it as a path, with and without the allowed options, inside the sandbox when the prefix is a relative name and outside when the prefix traverses; ``where``/``which`` and ``pip show`` take it where a BARE NAME belongs, so a path there denies on its separators rather than on its characters.
Definition at line 1195 of file gen_corpus.py.
|
protected |
Draw a listing command whose option list is built letter by letter. Everything about the command is ordinary except the SHAPE of its options, which is why the path it sometimes carries is always an in-sandbox one: the verdict has to turn on the option list and on nothing else. The optional option AFTER the path is the ``ls -l oc_chat -a`` shape -- an allow that a second path would turn into a denial, and the corpus carried neither.
Definition at line 1282 of file gen_corpus.py.
|
protected |
Draw a pip command whose options are spelled every way the gate reads. Two halves, because pip's two rules are about opposite things. The read-only subcommands take a CLOSED SET of options, so what matters is order, repetition and the near miss -- ``--format=json`` twice allows, ``--format=yaml`` denies, ``--format`` with an option after it denies because the value never arrives. ``install`` and ``download`` take almost everything EXCEPT an option naming a package index, and the gate recognises that family by the option NAME however it is abbreviated, bundled or attached, without ever looking at the value -- so a model that learned the URL learned the half that does not decide.
Definition at line 1328 of file gen_corpus.py.
|
protected |
Draw a listing command carrying zero to four path arguments. The rule is that a listing command takes AT MOST ONE path, and a corpus whose listing family draws zero, one or two paths from a small set shows it at one count and by accident. This family shows it at every count, with the paths adjacent or separated by options, distinct or repeated -- ``ls . .`` and ``ls oc_chat oc_chat`` deny exactly as ``ls a b`` does, and a model that learned "two DIFFERENT names" gets them wrong.
Definition at line 1302 of file gen_corpus.py.
|
protected |
Draw a pip command: a subcommand, maybe a package, maybe an option.
Definition at line 957 of file gen_corpus.py.
|
protected |
Draw English prose as the argument of a program the policy names. Four shapes, and the head decides the verdict rather than the words: plain prose (``echo`` allows, a listing command with more than one word denies on the path count, a Tier-2 program confirms, an unknown program denies), prose with a metacharacter clause spliced on (deny for everything, which is the injection shape the hand set carries), prose inside a closed quote, and prose after an opening quote that never closes (deny, and the only shape that produces a post-quote piece).
Definition at line 1140 of file gen_corpus.py.
|
protected |
Draw a pip command whose package name or option is an English word. ``pip install please`` confirms -- prose where a package name goes is still a package name -- while ``pip list please`` denies, because the read-only subcommands accept no argument at all, and ``pip install --index-url please`` denies on the package-index rule whatever the host is spelled like. The subcommand is upper-cased sometimes: the gate dispatches on the exact subcommand string, so ``pip SHOW requests`` is a denial, and it is a spelling the corpus otherwise never carries.
Definition at line 1219 of file gen_corpus.py.
|
protected |
Draw one payload in its three spellings -- the contrastive group. Every payload carries a shell metacharacter, so the three spellings cannot land on one verdict: quoted behind a directly-launched Tier-2 program the metacharacter is data and the gate CONFIRMS; with the quote removed it is outside a quote and the gate DENIES; behind ``echo``/``dir``/``ls`` the same quoted bytes are injection, because those run through ``cmd /c`` and are re-parsed, so the gate DENIES there too. The three differ by one quote character or one program name, which is the contrast a corpus that draws them independently never puts in front of a model.
Definition at line 1172 of file gen_corpus.py.
|
protected |
Draw a Tier-1 command: a listing, a program lookup, or an echo.
Definition at line 939 of file gen_corpus.py.
|
protected |
Draw a Tier-2 command, including quoted-metacharacter arguments.
Definition at line 968 of file gen_corpus.py.
|
protected |
Draw a command whose argument is a tilde path.
Definition at line 1063 of file gen_corpus.py.
|
protected |
Draw, label and select *target* rows per verdict from *families*. Returns the selected rows and every command the pool drew, so the next group can exclude them: the two groups draw from overlapping grammars -- ``pip list --no-color`` is reachable from both ``pip_subcommand`` and ``prose_pip`` -- and a command may appear in the corpus only once.
Definition at line 1581 of file gen_corpus.py.
|
protected |
The per-verdict row budget of each draw block, for a *total*-row run. *total* is the budget of the two groups plan 295 pinned, and everything after it is sized from the SHELL-GRAMMAR group's share of it rather than from *total* itself -- so a default run writes 45000 rows, not 37500. That is deliberate, and it is the only arrangement that leaves the previous corpora inside the new one: a block taking its share out of *total* would lower the earlier targets, the pool would stop at different draws, and every row that came before it would move. Four figures rather than three because ``head_contrast`` is the structural group's second BLOCK, not a fourth name in its round-robin; see :data:`STRUCTURE_FAMILIES_299`.
Definition at line 1601 of file gen_corpus.py.
|
protected |
Draw and label candidates from *families* until every verdict has *target*. Families are cycled round-robin so each gets the same number of attempts, and a command already drawn (or excluded) is discarded rather than relabelled.
Definition at line 1509 of file gen_corpus.py.
|
protected |
Leave *argument* bare, or wrap it in single or double quotes. Quoting is the whole point for the backslash and tilde families: bare, the POSIX parser de-escapes a backslash and the command means something else entirely, and the gate's verdict flips with it.
Definition at line 1040 of file gen_corpus.py.
|
protected |
Take *target* rows of each verdict, spread round-robin over *families*.
Definition at line 1552 of file gen_corpus.py.
|
protected |
*head* written the way one of the older families would have written it.
Definition at line 1384 of file gen_corpus.py.
|
protected |
Put *blank* somewhere inside *host* -- between tokens or within one. ``quotable`` admits a fifth placement that wraps the affected token in quotes, which is the only way a TAB survives into a token rather than separating two: P1 asks for these characters inside tokens, not only as separators.
Definition at line 1075 of file gen_corpus.py.
|
protected |
Order *blocks* so that every block's validation tenth comes first. The writer splits POSITIONALLY, so this ordering is what turns two blocks into two files that each begin with the previous plan's file.
Definition at line 1640 of file gen_corpus.py.
| list[dict[str, str]] gen_corpus.generate | ( | int | total, |
| int | seed, | ||
| Path | sandbox_root, | ||
| frozenset[str] | held_out ) |
Generate a balanced, family-covered, gate-labelled corpus. In THREE STAGES, each with its own row budget (plan 295 P2, plan 297 P1). The twelve shell-grammar families are drawn first, to the budget plan 292 pinned, and the four word-level families second to the remainder. A group therefore cannot take share from the other at any size -- the failure plan 291 measured and plan 292 could only push further out -- and the twelve land on exactly the counts they had before those four existed. The three structural families are drawn last and from a SEPARATE STREAM, so the first two groups see the generator plan 295 left them: the same draws, the same selection, and the same shuffle, which is taken here before the third stream is opened. The result is that the previous corpus survives whole inside this one and the new rows follow it, in both splits. ``head_contrast`` then repeats that argument one level down (plan 299 P1). It is a structural family and draws from the structural stream, but it draws from what that stream has left AFTER the three families have pooled, selected and shuffled -- so the plan-297 corpus survives whole inside this one exactly as the plan-295 corpus survives inside that. Four blocks, three prefixes, one appended block per plan. Every stage balances to a third per verdict, so their union does too, and the class balance the whole bench rests on is unaffected by the split.
Definition at line 1652 of file gen_corpus.py.
| frozenset[str] gen_corpus.load_held_out | ( | Path | directory | ) |
Every command in the two hand case sets, which never enter the corpus.
Definition at line 1497 of file gen_corpus.py.
| int gen_corpus.main | ( | list[str] | None | argv = None | ) |
CLI entry point: generate the corpus and write both splits.
Definition at line 1753 of file gen_corpus.py.
| list[dict[str, str]] gen_corpus.read_corpus | ( | Path | directory | ) |
Read the validation split then the training split, as written.
Definition at line 1490 of file gen_corpus.py.
| list[dict[str, str]] gen_corpus.read_jsonl | ( | Path | path | ) |
Read a JSON-lines file into a list of rows.
Definition at line 1483 of file gen_corpus.py.
| Path gen_corpus.subproject_root | ( | ) |
The oc_chat subproject root, which is the pinned sandbox root.
Definition at line 1477 of file gen_corpus.py.
| int gen_corpus.validation_rows | ( | int | total | ) |
How many rows of a *total*-budget generation are held for validation. A tenth of each BLOCK rather than a tenth of the whole, and the difference is the prefix witness: the validation split has to begin with the previous validation split and the training split with the previous training split, which only happens if each block is cut at its own tenth. The three agree with a tenth of the whole at the pinned total anyway -- 3750 + 450 + 300 is 4500 -- but they do not agree at every total, and :func:`write_corpus` is told the boundary rather than guessing it.
Definition at line 1624 of file gen_corpus.py.
| tuple[int, int] gen_corpus.write_corpus | ( | list[dict[str, str]] | rows, |
| Path | directory, | ||
| int | None | held = None ) |
Write the validation and training splits; return their sizes. *rows* arrive validation-first (:func:`_validation_first`), so the split is positional. *held* overrides the boundary, which is how a multi-block generation cuts each block at its own tenth instead of at the corpus's.
Definition at line 1712 of file gen_corpus.py.
|
protected |
Definition at line 419 of file gen_corpus.py.
|
protected |
Definition at line 430 of file gen_corpus.py.
|
protected |
Definition at line 442 of file gen_corpus.py.
|
protected |
Definition at line 410 of file gen_corpus.py.
|
protected |
Definition at line 476 of file gen_corpus.py.
|
protected |
Definition at line 765 of file gen_corpus.py.
|
protected |
Definition at line 738 of file gen_corpus.py.
|
protected |
Definition at line 739 of file gen_corpus.py.
|
protected |
Definition at line 747 of file gen_corpus.py.
|
protected |
Definition at line 773 of file gen_corpus.py.
|
protected |
Definition at line 407 of file gen_corpus.py.
|
protected |
Definition at line 395 of file gen_corpus.py.
|
protected |
Definition at line 389 of file gen_corpus.py.
|
protected |
Definition at line 1436 of file gen_corpus.py.
|
protected |
Definition at line 346 of file gen_corpus.py.
|
protected |
Definition at line 457 of file gen_corpus.py.
|
protected |
Definition at line 465 of file gen_corpus.py.
|
protected |
Definition at line 936 of file gen_corpus.py.
|
protected |
Definition at line 886 of file gen_corpus.py.
|
protected |
Definition at line 899 of file gen_corpus.py.
|
protected |
Definition at line 925 of file gen_corpus.py.
|
protected |
Definition at line 918 of file gen_corpus.py.
|
protected |
Definition at line 910 of file gen_corpus.py.
|
protected |
Definition at line 493 of file gen_corpus.py.
|
protected |
Definition at line 489 of file gen_corpus.py.
|
protected |
Definition at line 755 of file gen_corpus.py.
|
protected |
Definition at line 323 of file gen_corpus.py.
|
protected |
Definition at line 326 of file gen_corpus.py.
|
protected |
Definition at line 341 of file gen_corpus.py.
|
protected |
Definition at line 402 of file gen_corpus.py.
|
protected |
Definition at line 665 of file gen_corpus.py.
|
protected |
Definition at line 690 of file gen_corpus.py.
|
protected |
Definition at line 694 of file gen_corpus.py.
|
protected |
Definition at line 681 of file gen_corpus.py.
|
protected |
Definition at line 331 of file gen_corpus.py.
|
protected |
Definition at line 336 of file gen_corpus.py.
|
protected |
Definition at line 843 of file gen_corpus.py.
|
protected |
Definition at line 864 of file gen_corpus.py.
|
protected |
Definition at line 366 of file gen_corpus.py.
|
protected |
Definition at line 360 of file gen_corpus.py.
|
protected |
Definition at line 361 of file gen_corpus.py.
|
protected |
Definition at line 355 of file gen_corpus.py.
|
protected |
Definition at line 814 of file gen_corpus.py.
|
protected |
Definition at line 822 of file gen_corpus.py.
|
protected |
Definition at line 351 of file gen_corpus.py.
|
protected |
Definition at line 561 of file gen_corpus.py.
|
protected |
Definition at line 705 of file gen_corpus.py.
|
protected |
Definition at line 699 of file gen_corpus.py.
|
protected |
Definition at line 722 of file gen_corpus.py.
|
protected |
Definition at line 576 of file gen_corpus.py.
|
protected |
Definition at line 511 of file gen_corpus.py.
|
protected |
Definition at line 658 of file gen_corpus.py.
|
protected |
Definition at line 652 of file gen_corpus.py.
|
protected |
Definition at line 610 of file gen_corpus.py.
|
protected |
Definition at line 800 of file gen_corpus.py.
|
protected |
Definition at line 833 of file gen_corpus.py.
|
protected |
Definition at line 788 of file gen_corpus.py.
|
protected |
Definition at line 778 of file gen_corpus.py.
|
protected |
Definition at line 377 of file gen_corpus.py.
|
protected |
Definition at line 373 of file gen_corpus.py.
|
protected |
Definition at line 448 of file gen_corpus.py.
| int gen_corpus.DEFAULT_SEED = 7 |
Definition at line 155 of file gen_corpus.py.
| int gen_corpus.DEFAULT_TOTAL = 37_500 |
Definition at line 192 of file gen_corpus.py.
| tuple gen_corpus.FAMILIES = SHELL_FAMILIES + WORD_FAMILIES + STRUCTURE_FAMILIES |
Definition at line 321 of file gen_corpus.py.
| float gen_corpus.HEAD_CONTRAST_SHARE = 0.10 |
Definition at line 236 of file gen_corpus.py.
| tuple gen_corpus.LABELS = ("allow", "confirm", "deny") |
Definition at line 250 of file gen_corpus.py.
| tuple gen_corpus.SHELL_FAMILIES |
Definition at line 257 of file gen_corpus.py.
| tuple gen_corpus.STRUCTURE_FAMILIES = STRUCTURE_FAMILIES_297 + STRUCTURE_FAMILIES_299 |
Definition at line 318 of file gen_corpus.py.
| tuple gen_corpus.STRUCTURE_FAMILIES_297 |
Definition at line 293 of file gen_corpus.py.
| tuple gen_corpus.STRUCTURE_FAMILIES_299 |
Definition at line 311 of file gen_corpus.py.
| float gen_corpus.STRUCTURE_FAMILY_SHARE = 0.15 |
Definition at line 222 of file gen_corpus.py.
| int gen_corpus.STRUCTURE_SEED_OFFSET = 2 |
Definition at line 247 of file gen_corpus.py.
| tuple gen_corpus.WORD_FAMILIES |
Definition at line 278 of file gen_corpus.py.
| float gen_corpus.WORD_FAMILY_SHARE = 0.2 |
Definition at line 213 of file gen_corpus.py.
1.15.0