OCT Architecture Document

OCT Architecture Document#

Version: 3.3 Status: Authoritative Reference Scope: Defines the architecture, structure, and design philosophy of the OCT (Option C Tools) ecosystem v0.25.0, depending on oc_diagnostics v2.0.

Version history:

  • 1.1 — Initial release

  • 1.2 — Updated Diagnostics Package section to reflect oc_diagnostics v1.5; added FastAPI middleware, logging handler, full public API surface, and complete debug_config.json schema

  • 2.0 — Major update: added Formatter (v0.7.1), Health Dashboard (v0.7.2), Core module (exclusions.py, compat.py), oct clean command. Updated directory structure, diagrams, module inventory, and test suite documentation. Marked implemented features; updated future items.

  • 2.1 — Added global --root-dir CLI option for overriding project root detection (v0.7.3).

  • 2.2 — Added safe_parse() shared helper and syntax_warnings lint check; captures SyntaxWarning from AST parsing as structured lint findings (v0.7.4).

  • 2.3 — Hardened oct health test reporting: pre-flight pytest detection, actionable error details with --tb=line, test count always shown, = decoration stripped, stderr fallback, status label formatting. Added pytest>=7.0 as core dependency. Added tests/run_tests.py and tests/Tests_README.md to OCT’s own test suite (v0.7.5).

  • 2.4 — Added no_hardcoded_secrets linter rule (§6b Secret Hygiene, DD-05). Added NEVER_EXPORT_FILE_PATTERNS to exclusions.py for file-level export filtering. Removed .env from exporter extension profiles. Added oct docs --doxygen, --all, --clean, --dry-run flags and oct/docs/oct_docs.py module. Updated directory structure, test counts (97 tests across 4 suites), and module inventory.

  • 2.5 — Fixed oct export-source config resolution: uses get_project_root() instead of raw cwd, three-tier config fallback (project → oct bundled → built-in defaults). Added oct test meta-project sub-discovery for monorepo roots. Updated CLI commands table and sequence diagram (v0.9.1).

  • 2.6 — Added oct export-skeleton command (AST-based skeleton extraction with --no-diagnostics, --single-dir, --verbose, --clean). Added _skeleton to WILDCARD_EXCLUDE_DIRS. Updated oct clean to also remove _skeleton_code-* dirs. Updated test counts (123 tests across 5 suites).

  • 3.0 — Added Git integration modules: oct/core/git.py (git primitives), oct/git/ package (quality_gate, policy, audit, conventional, oct_git commands). Added oct git status, oct git check, oct git init, oct git commit, oct git hooks commands. Updated directory structure, diagrams, CLI table, and test counts (477 tests across 10 suites). Version bump v0.9.1 → v0.12.0.

  • 3.1 — Phase 4E: Added oct git changelog command, oct/git/changelog.py module, branch naming enforcement, branch-aware auto-profile selection, oct health git section, oct export-source --diff, oct test --changed, .octrc.json git schema validation. Updated test counts (542 tests, 2 skipped). Version bump v0.12.0 → v0.13.0.

  • 3.2 — Phase 6B+: Added oct/tools/secret_scanner.py (standalone scanner, OI-514), oct/core/waivers.py (shared inline waiver parsing, OI-540). Added entropy-based secret detection (FS-507) with per-context thresholds (code ≥ 4.5, docstring > 5.0, comment > 5.0). Added entropy_exclusions in .octrc.json secret_scanner section (field-name exclusions for namedtuple/dataclass fields, path-based exclusions for test directories). Quality gate honours inline waivers universally. Updated directory structure, module table, configuration schema, test counts (1402 tests, 9 skipped).

  • 3.3 — Phase F1 / v0.25.0: Workspace mode introduced (.option_c_workspace.json manifest). oct git status / commit / ignore / init fan out across subprojects when invoked from the workspace root, with per-subproject quality gates and a workspace bucket for files matched by workspace_files[]. Per-subproject CHANGELOG.md and SECURITY.md introduced at canonical root locations. Updated oc_diagnostics reference to v2.0.0; updated scope line to v0.25.0.

1. Overview#

OCT (Option C Tools) is the official development ecosystem for all Option C-compliant projects. It provides:

  • a linter — static compliance analysis (AST-based)

  • a formatter — auto-fix compliance violations

  • a health dashboard — single-command project compliance report

  • a file generator — create new compliant files

  • a project scaffolder — create full project templates

  • a documentation validator — check required docs

  • a test runner — pytest wrapper

  • a source exporter — consolidate source for review

  • a cleanup tool — remove generated artifacts

  • integration with the standalone oc_diagnostics package

OCT enforces the Option C coding philosophy and provides a predictable, diagnostics-aware development workflow.

2. High-Level Architecture#

+-------------------------------------------------------------+
|                       OCT CLI [--root-dir]                   |
|  lint | format | health | new | scaffold | docs | test      |
|  export-source | export-skeleton | clean | diag | deps     |
|  git (status | check | init | commit | hooks)              |
+-------------------------------------------------------------+
         |                    |                    |
         v                    v                    v
+--------------------+  +-----------+  +-----------------------------+
|    OCT Modules     |  | OCT Core  |  |     oc_diagnostics v2.0     |
|  - Linter          |  | - excl.   |  |  - unified_debug            |
|  - Formatter       |  | - compat  |  |  - instrumentation          |
|  - Health          |  | - root    |  |  - ui_event_logger          |
|  - Generator       |  | - git     |  |  - fastapi_middleware       |
|  - Exporter        |  +-----------+  |  - logging_handler          |
|  - Git Integration |                 |  - self-test suite          |
+--------------------+                 +-----------------------------+
         |
         v
+-----------------------------+
|     Option C Project        |
|  - src/                     |
|  - tests/                   |
|  - docs/                    |
|  - oc_diagnostics/debug_config |
+-----------------------------+

OCT is a development-time tool. oc_diagnostics is a runtime system. Projects consume both.

        flowchart TD

    subgraph CLI["OCT CLI"]
        LINT["oct lint"]
        FORMAT["oct format"]
        HEALTH["oct health"]
        NEW["oct new"]
        SCAFFOLD["oct scaffold"]
        DOCS["oct docs"]
        TEST["oct test"]
        EXPORT["oct export-source"]
        SKELETON["oct export-skeleton"]
        CLEAN["oct clean"]
        DEPS["oct deps"]
        DIAG["oct diag"]
        GIT["oct git"]
    end

    subgraph CORE["OCT Core"]
        EXCLUSIONS["exclusions.py\n(shared exclude lists)"]
        COMPAT["compat.py\n(oc_diagnostics version check)"]
        ROOT["project_root.py\n(root detection)"]
        PARSING["parsing.py\n(safe AST parsing)"]
        GITCORE["git.py\n(git primitives)"]
        WAIVERS["waivers.py\n(inline waiver parsing)"]
    end

    subgraph OCT["OCT Modules"]
        LINTER["Linter\n(oct_lint.py)"]
        FORMATTER["Formatter\n(oct_format.py)"]
        HEALTHMOD["Health Dashboard\n(oct_health.py)"]
        GENERATOR["File Generator\n(new_python_file.py)"]
        SCAFFOLDER["Project Scaffolder\n(cli.py)"]
        EXPORTER["Source Exporter\n(source_exporter.py)"]
        SKELEXP["Skeleton Exporter\n(skeleton_exporter.py)"]
        GITMOD["Git Integration\n(oct_git.py, quality_gate.py,\npolicy.py, audit.py, conventional.py)"]
    end

    subgraph DIAGPKG["oc_diagnostics v2.0\n(Standalone Package)"]
        UDEBUG["unified_debug.py\n(_dbg, _adbg, domains, levels,\nruntime API, profiling)"]
        INSTR["id_instrumentation.py\n(ID rewriting)"]
        MIDDLE["instrumentation_middleware.py\n(callback patching)"]
        LOGGER["ui_event_logger.py\n(event capture)"]
        FMIDDLE["fastapi_middleware.py\n(ASGI middleware)"]
        LHANDLER["logging_handler.py\n(stdlib logging bridge)"]
    end

    subgraph PROJECT["Option C Project"]
        SRC["src/ or ui/ or engine/"]
        TESTS["tests/"]
        DOCS2["docs/"]
        CONFIG["oc_diagnostics/debug_config.json"]
    end

    CLI --> CORE
    CLI --> OCT
    OCT --> CORE
    OCT --> DIAGPKG
    OCT --> PROJECT
    DIAGPKG --> PROJECT
    
        mindmap
  root((OCT Ecosystem))
    CLI
      lint
      format
      health
      new
      scaffold
      docs
      test
      export-source
      export-skeleton
      clean
      deps
      diag
      git
        status
        check
        init
        commit
        hooks
    Core
      exclusions
      compat
      project_root
      git primitives
    Modules
      Linter
      Formatter
      Health Dashboard
      File Generator
      Project Scaffolder
      Source Exporter
      Git Integration
    oc_diagnostics (v2.0)
      _dbg / _adbg
      instrumentation
      middleware
      event logger
      FastAPI middleware
      logging handler
      self-test suite
    Option C Project
      src/
      tests/
      docs/
      oc_diagnostics/debug_config.json

3. Directory Structure of OCT#

oct/
    oct/
        __init__.py              # Package init, exposes __version__
        __main__.py              # python -m oct support
        cli.py                   # Click CLI — all commands
        core/
            project_root.py      # Project root detection (docs/ + tests/ + oc_diagnostics/)
            exclusions.py        # Shared exclusion lists (dirs + NEVER_EXPORT_FILE_PATTERNS)
            compat.py            # oc_diagnostics version compatibility check
            parsing.py           # Shared safe AST parsing
            terminal.py          # Shared terminal color utilities
            git.py               # Git primitives (run, staged files, branch, diff)
            waivers.py           # Shared inline waiver parsing (OCT-LINT: disable=...)
        linter/
            __init__.py
            oct_lint.py          # AST-based Option C linter (incl. no_hardcoded_secrets)
        formatter/
            __init__.py
            oct_format.py        # Auto-fixer for compliance violations
        health/
            __init__.py
            oct_health.py        # Project compliance dashboard
        deps/
            __init__.py
            oct_deps.py          # Dependency graph generation
        diag/
            __init__.py
            oct_diag.py          # Diagnostics config management
        docs/
            __init__.py
            oct_docs.py          # Documentation pipelines (Sphinx, Doxygen, clean)
        git/
            __init__.py
            oct_git.py           # oct git status/check/init/commit/hooks commands
            quality_gate.py      # Unified quality gate (lint + format + secrets)
            policy.py            # Profile resolution, protected branches, scope filtering
            audit.py             # @audited decorator, AuditRecord, JSONL logging
            conventional.py      # Conventional Commits parser, validator, diagnostics impact
        hooks/
            __init__.py
            oct_hooks.py         # Pre-commit hook integration (legacy)
        typecheck/
            __init__.py
            oct_typecheck.py     # Type checking integration
        generator/
            new_python_file.py   # Option C file generator
        tools/
            source_exporter.py   # Source code consolidation exporter
            skeleton_exporter.py # Structural skeleton exporter (AST-based)
            secret_scanner.py    # Standalone hardcoded-secret detection (AST + entropy)
        templates/
            sphinx/              # Sphinx templates (conf.py.template, CSS, logo)
    tests/
        run_tests.py             # pytest runner script
        Tests_README.md          # Test suite documentation
        tests_linter/
            conftest.py          # Shared fixtures
            fixtures/            # Test fixture files
            test_func_pattern.py
            test_header_fixing.py
            test_headers.py
            test_integration.py
            test_no_hardcoded_secrets.py  # §6b secret hygiene rule tests
            test_regression.py
        tests_formatter/
            conftest.py          # Shared fixtures
            fixtures/            # Test fixture files
            test_docstring_fixing.py
            test_func_pattern_fixing.py
            test_header_fixing.py
            test_import_fixing.py
            test_integration.py
        tests_health/
            conftest.py          # Shared fixtures
            test_docs_health.py
            test_integration.py
            test_lint_health.py
        tests_core/
            test_exclusions.py
            test_git.py              # Git primitives (run, staged, branch, diff)
            test_project_root.py
        tests_docs/
            test_docs_clean.py   # oct docs --clean tests
            test_doxygen_cleanup.py
            test_find_packages.py
        tests_git/
            conftest.py          # Shared fixtures (temp_git_project, make_compliant_py, etc.)
            test_audit.py        # Audit record, JSONL writing, @audited decorator
            test_conventional.py # CC parser, validator, error formatting, diagnostics impact
            test_git_check.py    # Quality gate command integration tests
            test_git_commit.py   # Commit command: exit codes, safety gates, secrets invariant
            test_git_hooks.py    # Hooks install/status/remove, deprecation shim
            test_git_init.py     # Init command: .gitignore, .gitattributes, workflow
            test_git_status.py   # Status command integration tests
            test_policy.py       # Profile resolution, protected branches, scope filtering
    docs/
        ARCHITECTURE.md          # This file
        ROADMAP.md
        REFERENCE.md
        README.md
        linter.md
    pyproject.toml
    README.md
    pytest.ini

3.1 Module Dependency Graph#

        classDiagram
    class OCTCLI {
        +lint()
        +format()
        +health()
        +new()
        +scaffold()
        +docs()
        +test()
        +export_source()
        +clean()
        +git()
    }

    class Core {
        +find_project_root()
        +check_oc_diagnostics_compat()
        +LINTER_EXCLUDE_DIRS
        +WILDCARD_EXCLUDE_DIRS
    }

    class Linter {
        +run_linter()
        +validate_header_block()
        +check_docstring()
        +check_diagnostics_profile()
        +check_dbg_usage()
        +check_dbg_import()
        +check_func_pattern()
        +check_no_hardcoded_secrets()
        +find_python_files()
        +LinterContext
    }

    class Formatter {
        +run_formatter()
        +fix_header_block()
        +fix_docstring()
        +fix_dbg_import()
        +fix_func_pattern()
        +FormatterContext
    }

    class HealthDashboard {
        +run_health()
        +check_lint_health()
        +check_fixable_health()
        +check_docs_health()
        +check_test_health()
        +check_compat_health()
    }

    class Generator {
        +create_new_file()
    }

    class Exporter {
        +run_exporter()
        +clean_source_dirs()
        +clean_pycache_dirs()
    }

    class GitIntegration {
        +git_status_cmd()
        +git_check_cmd()
        +git_init_cmd()
        +git_commit_cmd()
        +hooks_install_cmd()
        +hooks_status_cmd()
        +hooks_remove_cmd()
        +run_quality_gate()
        +validate_commit_message()
    }

    OCTCLI --> Core
    OCTCLI --> Linter
    OCTCLI --> Formatter
    OCTCLI --> HealthDashboard
    OCTCLI --> Generator
    OCTCLI --> Exporter
    OCTCLI --> GitIntegration
    Linter --> Core
    Formatter --> Core
    Formatter --> Linter : reuses check functions
    HealthDashboard --> Linter : reuses check functions
    HealthDashboard --> Formatter : reuses fix functions (dry-run)
    HealthDashboard --> Core
    Exporter --> Core
    GitIntegration --> Core
    GitIntegration --> Linter : quality gate
    GitIntegration --> Formatter : quality gate

4. OCT CLI#

The OCT CLI is the primary entry point for all Option C tooling. It uses Click for command routing and delegates all logic to the appropriate module.

Available Commands#

Command

Module

Description

oct lint

oct.linter.oct_lint

Run the Option C linter (AST-based, JSON output, rule profiles)

oct format

oct.formatter.oct_format

Auto-fix compliance violations (headers, docstrings, imports, func patterns)

oct health

oct.health.oct_health

Display project compliance dashboard (lint, fixable, docs, tests, compat)

oct new

oct.generator.new_python_file

Create a new Option C-compliant Python file

oct scaffold

oct.cli (inline)

Create a full Option C project template

oct docs

oct.docs.oct_docs

Validate docs, generate Sphinx/Doxygen HTML, or clean generated artifacts

oct test

oct.cli (_run_pytest)

Run project tests via pytest (single-project or meta-project sub-discovery)

oct export-source

oct.tools.source_exporter

Consolidate source code for review (excludes secret-bearing files)

oct export-skeleton

oct.tools.skeleton_exporter

Export structural skeletons (headers, docstrings, signatures) for AI inspection

oct clean

oct.tools.source_exporter, oct.tools.skeleton_exporter

Remove _source_code-*, _skeleton_code-*, and __pycache__/ directories

oct diag

oct.diag.oct_diag

Validate, inspect, and update debug_config.json

oct deps

oct.deps.oct_deps

Dependency graph generation (JSON, Mermaid, DOT)

oct git status

oct.git.oct_git

Show git status with Option C compliance annotations

oct git check

oct.git.oct_git

Run the unified quality gate (lint + format + secrets)

oct git init

oct.git.oct_git

Initialise or enhance a git repo with Option C conventions

oct git commit

oct.git.oct_git

Commit with quality gate and non-bypassable secrets enforcement

oct git hooks

oct.git.oct_git

Install, verify, or remove Option C git hooks

Each command is implemented as a Click subcommand registered on the cli group.

CLI Design Patterns#

  • Argparse passthrough (lint, format): Uses @click.argument("args", nargs=-1) to forward flags to internal argparse within the module. Allows richer flag handling.

  • Click options (health, new, scaffold, clean): Uses @click.option() directly for simpler command interfaces.

5. OCT Modules#

5.1 Core (oct/core/)#

Shared utilities used by multiple OCT modules.

File

Responsibility

project_root.py

Walk upward from cwd to find the project root (directory containing docs/, tests/, and oc_diagnostics/ or diagnostics/).

exclusions.py

Shared LINTER_EXCLUDE_DIRS, WILDCARD_EXCLUDE_DIRS, and NEVER_EXPORT_FILE_PATTERNS sets, used by linter, formatter, health, exporter, and docs.

compat.py

Check whether oc_diagnostics is installed and version-compatible. Returns a warning message or empty string.

waivers.py

Shared inline waiver parsing (# OCT-LINT: disable=<rule> reason="..."). Extracted from oct_lint.py (OI-540) so both linter and quality gate can import the same parser without the quality gate depending on the linter.

5.2 Linter (oct/linter/oct_lint.py)#

Responsibilities:

  • Enforce Option C header rules (3-line header + blank line)

  • Enforce module docstring structure (Purpose, Responsibilities, Diagnostics, Contracts)

  • Enforce diagnostics profile block (Domain + L2/L3/L4)

  • Enforce _dbg usage for non-trivial modules (AST-verified, > 20 lines)

  • Enforce from oc_diagnostics import _dbg canonical import (AST-verified)

  • Enforce func = "function_name" pattern per function (AST-verified)

  • Detect hardcoded secrets in assignments, dicts, and function defaults (§6b Secret Hygiene, AST-verified) — delegates to oct.tools.secret_scanner (OI-514)

  • Detect high-entropy string literals via Shannon entropy (FS-507) — per-context thresholds: code strings ≥ 4.5 bits/char, docstrings > 5.0, comments > 5.0 (reserved)

  • Support entropy_exclusions from .octrc.json — field-name exclusions for namedtuple/dataclass documentation fields, path-based exclusions for test directories

  • Support inline waivers (# OCT-LINT: disable=<rule> reason="...") — parsed by oct.core.waivers (OI-540)

  • Check for regression tests (heuristic)

  • Generate terminal reports with colour, JSON output, and timestamped log files

  • Support configurable rule profiles via .octrc.json (strict, compact, proto)

The linter is static-analysis only (AST parsing + text analysis). It never imports project modules.

Key exports: run_linter(), validate_header_block(), check_docstring(), check_diagnostics_profile(), check_dbg_usage(), check_dbg_import(), check_func_pattern(), check_no_hardcoded_secrets() (re-exported from oct.tools.secret_scanner), find_python_files(), LinterContext.

5.3 Formatter (oct/formatter/oct_format.py)#

Added in v0.7.1.

Responsibilities:

  • Auto-fix malformed header blocks (shebang, encoding, file identity)

  • Insert or complete module docstrings with required sections

  • Add canonical from oc_diagnostics import _dbg import

  • Insert func = "function_name" in functions that call _dbg()

  • Archive originals to .formatter_archive/<filename>.<timestamp>.bak before modification

  • Support dry-run mode (default), fix mode, JSON output, and verbose output

Design:

  • Report-then-fix model: Default is dry-run (report only). --fix applies changes.

  • Fixer pipeline: header -> docstring -> import -> func_pattern (applied sequentially per file).

  • Each fixer returns Tuple[bool, List[str]] (changed, messages).

  • Reuses linter’s find_python_files() and exclusion lists.

  • Uses FormatterContext dataclass to encapsulate tool state.

5.4 Health Dashboard (oct/health/oct_health.py)#

Added in v0.7.2.

Responsibilities:

  • Aggregate lint check results across all Python files (per-check pass counts, compliance %)

  • Count fixable violations by running formatter checks in dry-run mode

  • Validate presence of required documentation files

  • Run pytest via subprocess and capture pass/fail status with test count and actionable error details

  • Detect missing pytest before subprocess invocation

  • Check oc_diagnostics version compatibility

  • Output results as terminal report or JSON

Design:

  • Composition layer: Reuses existing linter check functions and formatter fix functions directly (no subprocess calls), giving structured data natively.

  • Five independent check functions: check_lint_health(), check_fixable_health(), check_docs_health(), check_test_health(), check_compat_health().

  • All checks are independent; failure of one does not prevent others.

  • Terminal output uses ASCII-safe characters for Windows compatibility.

5.5 File Generator (oct/generator/new_python_file.py)#

Responsibilities:

  • Create new Python files with correct Option C headers

  • Insert from oc_diagnostics import _dbg into generated files

  • Generate module docstring templates with all required sections

  • Ensure correct file identity paths relative to project root

  • Validate target path is inside project root

  • Support --force (overwrite) and --dry-run modes

5.6 Source Exporter (oct/tools/source_exporter.py)#

Responsibilities:

  • Consolidate all source files into _source_code-<timestamp> directories

  • Generate _full_source.<root>.txt rollup

  • Exclude secret-bearing files via NEVER_EXPORT_FILE_PATTERNS (§6b Secret Hygiene)

  • Support --verbose, --clean, and --single-dir modes

  • Handle I/O errors (including Windows MAX_PATH) gracefully: warn and continue

  • Truncate output filenames exceeding 240 characters, appending an MD5 suffix for uniqueness

  • Report skipped directories in the summary rather than crashing

  • Support per-project config via _oct_exporter_config.json with fallback to default configuration.

5.7 Documentation Pipelines (oct/docs/oct_docs.py)#

Responsibilities:

  • Sphinx pipeline (--sphinx): Set up Sphinx config, discover markdown, generate API reference via sphinx-apidoc, generate dependency graphs, write index.rst, run sphinx-build

  • Doxygen pipeline (--doxygen): Locate existing Doxyfile, patch stale INPUT path in temp copy, run doxygen

  • Combined pipeline (--all): Run both Sphinx and Doxygen sequentially

  • Clean pipeline (--clean): Remove generated artifacts (html/, doctrees/, _api/, doxygen_output/, index.rst, dependencies.rst) while preserving user-authored files

  • Exclude .env* from Sphinx via conf.py.template exclude_patterns

Design:

  • Doxygen runs on a temporary copy of the Doxyfile — the original is never modified.

  • Sphinx config (docs/_sphinx/conf.py) is generated once from templates; not overwritten on subsequent builds.

  • Clean preserves _sphinx/, _static/, _templates/, doxygen_work_dir/, and user-authored .md files.

5.8 Skeleton Exporter (oct/tools/skeleton_exporter.py)#

Responsibilities:

  • Extract structural skeletons from Python files via AST: headers, module docstrings, class/function signatures with decorators and type hints.

  • Produce one-line placeholders for non-Python files (filename, line count, size).

  • Strip named docstring sections (--no-diagnostics removes the Diagnostics block).

  • Reuse source exporter infrastructure (config loading, exclusions, file collection, path safety).

Design:

  • Uses safe_parse() from oct.core.parsing for robust AST parsing.

  • Walks tree.body (not flat ast.walk) to preserve structural nesting (classes contain methods).

  • Uses ast.unparse() for faithful signature reconstruction (annotations, defaults, decorators).

  • _strip_docstring_section() is generalised by section name, enabling future --no-contracts or --sections-exclude flags.

  • Output structure mirrors source_exporter: per-directory _skeleton_code-* dirs with _skeleton.<dotpath>.txt files plus _full_skeleton.<root>.txt rollup.

5.9 Git Integration (oct/git/)#

Added in v0.11.0 (Phase 4A/4B), extended in v0.11.1 (4C), v0.12.0 (4D), v0.13.0 (4E).

Responsibilities:

  • Git primitives: run commands, detect repo, list staged files, current branch, diffs

  • Quality gate: unified lint + format + secrets check with profile-based policy

  • Audit trail: @audited decorator for JSONL logging of all git operations

  • Conventional Commits: parser, validator, diagnostics impact analysis

  • Commit workflow: pre-flight checks, secrets enforcement (non-bypassable), protected-branch guards

  • Hooks management: install/status/remove .pre-commit-config.yaml hooks

Architecture:

File

Responsibility

oct/core/git.py

Low-level git primitives (subprocess wrappers, no OCT policy). git_run(), git_staged_files(), current_branch(), is_git_repo().

oct/git/quality_gate.py

Orchestrates lint, format, secrets checks. Returns QualityGateResult with violation counts. Supports entropy_exclusions (field-name + path-based), inline waivers (OI-540).

oct/git/policy.py

Profile resolution (resolve_effective_profile), protected-branch detection, scope filtering, apply_profile_policy.

oct/git/audit.py

AuditRecord dataclass, @audited decorator, JSONL file writer. All oct git commands are audited.

oct/git/conventional.py

ConventionalCommit dataclass, parse_commit_message, validate_commit_message, analyse_diagnostics_impact.

oct/git/oct_git.py

Click commands: status, check, init, commit, hooks subgroup (install/status/remove).

Key design decisions:

  • Zero external dependencies: Git commands use subprocess.run (not gitpython).

  • Non-bypassable secrets (G-D4 invariant): --force and --no-verify never skip secrets detection. The quality gate always runs, and result.secrets_findings is checked unconditionally.

  • Protected branches: main/master (configurable via .octrc.json) force strict profile regardless of CLI override. Direct commits are blocked in strict mode.

  • Audit trail: Every oct git command is decorated with @audited(cmd_name), which writes a JSONL record to logs/oct_git_audit*.jsonl with timestamp, command, exit code, duration, and command-specific metadata.

  • Commit exit codes differ from quality gate exit codes: Quality gate uses 0–5 (per-check granularity). Commit uses 0–4 (outcome-oriented). The commit command maps quality gate results to its own scheme.

6. Configuration#

6.1 Per-project linter configuration (.octrc.json)#

{
  "linter": {
    "profile": "default",
    "exclude_dirs_add": ["vendor"],
    "exclude_dirs_remove": ["data"],
    "wildcard_exclude_add": ["_temp"],
    "rules": {
      "func_pattern": false
    }
  }
}

Profiles: strict (all rules), compact (public functions), proto (headers only). Individual rule overrides via rules map.

Available rules: header, docstring, diagnostics_profile, dbg_usage, dbg_import, func_pattern, no_hardcoded_secrets, tests_exist, type_hints.

6.2 Secret scanner configuration (secret_scanner in .octrc.json)#

{
  "secret_scanner": {
    "code_entropy_threshold": 4.5,
    "docstring_entropy_threshold": 5.0,
    "comment_entropy_threshold": 5.0,
    "entropy_exclusions": [
      {
        "field_name_pattern": "correct_pattern",
        "context": "namedtuple_field",
        "reason": "Documentation code examples in rule definitions"
      },
      {
        "path_pattern": "tests/",
        "context": "test_directory",
        "reason": "Test fixtures exempt from entropy",
        "restrictions": {
          "require_waiver_for_secret_patterns": true,
          "exempt_entropy_only": true
        }
      }
    ]
  }
}

  • code_entropy_threshold — Shannon entropy threshold for code strings (≥ comparison). Set to 0 to disable entropy checking. Default 4.5.

  • docstring_entropy_threshold — threshold for docstrings (strict > comparison). Default 5.0.

  • comment_entropy_threshold — reserved for future AST-level comment scanning. Default 5.0.

  • entropy_exclusions — array of exclusion rules:

    • context: "namedtuple_field" + field_name_pattern — exempt keyword arguments with matching names in Call nodes (e.g. RuleInfo(correct_pattern="...")).

    • context: "test_directory" + path_pattern — exempt files under matching paths. restrictions.exempt_entropy_only disables entropy without disabling name-pattern detection. restrictions.require_waiver_for_secret_patterns requires inline waivers for name-based findings.

Known limitation: Inline waivers (# OCT-LINT: disable=no_hardcoded_secrets reason="...") are matched at file level — a single waiver suppresses all no_hardcoded_secrets findings in the file. Per-line waiver matching is a planned improvement.

6.3 Per-project exporter configuration (_oct_exporter_config.json)#

Supports add/remove/replace semantics for include_extensions, exclude_dirs, and wildcard_exclude_dirs. When missing, oct export-source uses the default configuration defined in oct/core/exporter_config.py.

7. oc_diagnostics Integration#

oc_diagnostics is a standalone runtime package providing:

  • _dbg — synchronous structured debug logging

  • _adbg — async variant for use inside async def functions (FS-16)

  • DEBUG_LEVELS — dict of domain to active level

  • init() — activate stream interception and exception hook; accepts watch_config=True for hot-reload (FS-09)

  • set_debug_level(domain, level) — runtime domain level override (FS-03)

  • set_global_setting(key, value) — runtime global setting override (FS-03)

  • dbg_timed(domain, func, label, level) — context manager: logs ENTER + EXIT with elapsed ms (FS-12)

  • dbg_scope(domain, func, label, level) — context manager: logs ENTER + EXIT without timing (FS-12/FS-19)

  • dbg_profile(domain, func, label, level) — decorator: wraps a function with dbg_timed (FS-12)

  • OcDiagnosticsHandler — Python logging.Handler that routes records through _dbg (FS-08)

  • apply_instrumentation — Dash ID instrumentation (requires pip install oc_diagnostics[dash])

  • DiagnosticsMiddleware — pure ASGI HTTP middleware (requires pip install oc_diagnostics[fastapi])

        flowchart TD

    CONFIG["oc_diagnostics/debug_config.json\n(Project-specific)"]

    UDEBUG["unified_debug.py\n(_dbg, _adbg, domains, levels,\nruntime API, profiling, hot-reload)"]
    LHANDLER["logging_handler.py\n(OcDiagnosticsHandler)"]
    INSTR["id_instrumentation.py\n(ID rewriting)"]
    MIDDLE["instrumentation_middleware.py\n(callback patching)"]
    LOGGER["ui_event_logger.py\n(event capture)"]
    FMIDDLE["fastapi_middleware.py\n(ASGI middleware)"]

    APP["Dash / FastAPI / Application Code"]
    CALLBACKS["Callbacks"]
    OUTPUT["Instrumented Output"]

    CONFIG --> UDEBUG
    UDEBUG --> INSTR
    UDEBUG --> MIDDLE
    UDEBUG --> LOGGER
    UDEBUG --> LHANDLER
    UDEBUG --> FMIDDLE

    APP --> CALLBACKS
    CALLBACKS --> MIDDLE
    MIDDLE --> OUTPUT
    INSTR --> OUTPUT
    LOGGER --> OUTPUT

Project structure#

Projects include only:

oc_diagnostics/debug_config.json

Everything else is imported from oc_diagnostics.

Installation#

Core (always required):

pip install oc_diagnostics

With Dash instrumentation:

pip install oc_diagnostics[dash]

With FastAPI / ASGI middleware:

pip install oc_diagnostics[fastapi]

With JSON Schema config validation (FS-18):

pip install oc_diagnostics[validation]

Import surface#

Top-level imports (always available):

from oc_diagnostics import _dbg                    # synchronous structured logging
from oc_diagnostics import _adbg                   # async variant (FS-16)
from oc_diagnostics import DEBUG_LEVELS
from oc_diagnostics import init
from oc_diagnostics import set_debug_level         # runtime domain level override (FS-03)
from oc_diagnostics import set_global_setting      # runtime global setting override (FS-03)
from oc_diagnostics import dbg_timed               # timing context manager (FS-12)
from oc_diagnostics import dbg_scope               # lifecycle context manager (FS-12/FS-19)
from oc_diagnostics import dbg_profile             # timing decorator (FS-12)
from oc_diagnostics import OcDiagnosticsHandler    # Python logging integration (FS-08)
from oc_diagnostics import apply_instrumentation   # requires: pip install oc_diagnostics[dash]
from oc_diagnostics import DiagnosticsMiddleware   # requires: pip install oc_diagnostics[fastapi]

Sub-module imports for advanced usage:

from oc_diagnostics.id_instrumentation import instrument_ids, MAX_INSTRUMENT_DEPTH
from oc_diagnostics.ui_event_logger import register_callbacks, REQUIRED_STORE_ID
from oc_diagnostics.unified_debug import ADDITIONAL_STRUCTURAL_PREFIXES, ADDITIONAL_STRUCTURAL_EXACT
from oc_diagnostics.unified_debug import HTTP_REDACT_HEADERS, UI_EVENT_PROPERTIES

debug_config.json — full schema#

All keys available in global_settings:

{
  "domains": {
    "GENERAL": { "level": 2, "color": "default" },
    "UI":      { "level": 3, "color": "cyan" },
    "DATA":    { "level": 3, "color": "green" },
    "HTTP":    { "level": 2, "color": "blue" },
    "STDOUT":  { "level": 2, "color": "gray" },
    "STDERR":  { "level": 2, "color": "red" },
    "PYERR":   { "level": 1, "color": "red" }
  },
  "global_settings": {
    "silent_mode": false,
    "enable_timestamps": true,
    "enable_colors": true,
    "include_filename": true,
    "output_mode": "both",
    "instrumentation_enabled": false,
    "max_log_files": 10,
    "max_log_size_mb": 5.0,
    "ui_event_properties": ["n_clicks", "value", "data", "figure", "selectedData"],
    "instrumentation_allow_all": false,
    "mode": "dev",
    "http_redact_headers": ["authorization", "cookie", "set-cookie", "x-api-key"],
    "structural_id_prefixes": [],
    "structural_id_exact": []
  }
}

output_mode options: "terminal", "file", "both", "json" (JSONL structured log, FS-07).

mode options: "dev" (default), "prod" / "production" / "safe" (suppresses stream interception and exception hook, FS-15). Can also be set via OC_DIAGNOSTICS_MODE environment variable.

Environment variables that override config:

  • OC_DIAGNOSTICS_CONFIG — absolute path to debug_config.json

  • OC_DIAGNOSTICS_LOG_DIR — absolute path for log directory

  • OC_DIAGNOSTICS_MODE — set to prod to activate production mode

  • UNIFIED_DEBUG_DISABLE_EXCEPTION_HOOK=1 — disable custom exception hook

  • UNIFIED_DEBUG_DISABLE_STDIO_INTERCEPT=1 — disable stdout/stderr interception

OCT Integration#

  • oct scaffold creates oc_diagnostics/debug_config.json with a valid oc_diagnostics template

  • oct new generates files with from oc_diagnostics import _dbg

  • oct lint checks _dbg import source, usage, and diagnostics profile blocks

  • oct format adds missing _dbg imports and func = "..." patterns

  • oct health checks oc_diagnostics version compatibility

  • oct docs validates diagnostics blocks

  • Future: oct diag will expose diagnostics tools directly

8. Option C Coding Rules (Summary)#

8.1 File headers#

  • shebang (#!/usr/bin/env python3)

  • encoding (# -*- coding: utf-8 -*-)

  • file identity (# project/path.py)

  • blank line

8.2 Module docstrings#

  • Purpose

  • Responsibilities

  • Diagnostics (Domain + L2/L3/L4)

  • Contracts

8.3 Diagnostics#

  • from oc_diagnostics import _dbg (canonical import)

  • func = "function_name" in every function that calls _dbg()

  • _dbg(domain, func, message, level) calls at L2/L3/L4

  • Domain must exist in debug_config.json

8.4 Trivial file exemption#

Files under 20 lines are exempt from _dbg usage, _dbg import, and func = "..." checks. This prevents over-enforcement on scaffolding files like __init__.py.

8.5 Testing#

  • Regression tests required for every feature

  • All tests in <project root>/tests/

  • Tests must be deterministic

  • Documented in Tests_README.md

  • run_tests.py script must exist

8.6 Architecture#

  • docs/ARCHITECTURE.md required

  • Versioned

  • Updated with significant changes

9. Test Suite#

OCT has 1402 tests across its test suites (9 skipped):

Suite

Directory

Tests

Coverage

Core

tests/tests_core/

58+

Project root detection, exclusions, git primitives, octrc loading, path utilities

Linter

tests/tests_linter/

78+

Header validation, func pattern (AST), header fixing, no_hardcoded_secrets (§6b), type hints, config validation, lint –changed, integration, regression

Formatter

tests/tests_formatter/

42+

Header fixing, docstring fixing, import fixing, func pattern fixing, archive rotation, format –changed, integration

Health

tests/tests_health/

22+

Lint aggregation, docs validation, JSON/terminal output, compat check, integration

Docs

tests/tests_docs/

24+

docs –clean, doxygen cleanup, package discovery, Sphinx write safety

Tools

tests/tests_tools/

81

Skeleton extraction (AST, headers, signatures, classes, decorators), source exporter, secret scanner (Shannon entropy, per-context thresholds, field-name exclusions, validation)

Git — Audit

tests/tests_git/

25+

Audit record creation, JSONL writing, @audited decorator

Git — Conventional

tests/tests_git/

35+

Conventional Commits parser, validator, error formatting, diagnostics impact

Git — Commands

tests/tests_git/

118+

Status, check, init, commit, hooks — exit codes, safety gates, integration

Git — Policy

tests/tests_git/

54+

Profile resolution, protected branches, scope filtering

(9 tests skipped: platform-specific or optional-dependency tests.)

All test suites use pytest with shared conftest.py fixtures providing temporary Option C project directories with the required structure.

10. OCT Project Lifecycle#

oct scaffold myproject
cd myproject
oct git init                     # .gitignore, .gitattributes, CI workflow
oct git hooks install            # Install pre-commit hooks
oct new src/module.py
oct docs
oct lint
oct format --fix
oct health
oct test
oct git commit -m "feat: initial implementation"
oct export-source
oct export-skeleton --no-diagnostics

OCT enforces correctness at every step.

        sequenceDiagram
    participant Dev as Developer
    participant OCT as OCT CLI
    participant Project as Option C Project
    participant Diag as oc_diagnostics

    Dev->>OCT: oct scaffold myproject
    OCT->>Project: Create project structure
    OCT->>Project: Add oc_diagnostics/debug_config.json

    Dev->>OCT: oct git init
    OCT->>Project: Create .gitignore, .gitattributes, CI workflow

    Dev->>OCT: oct git hooks install
    OCT->>Project: Generate .pre-commit-config.yaml

    Dev->>OCT: oct new src/module.py
    OCT->>Project: Generate Option C file with headers, import & docstring

    Dev->>OCT: oct lint
    OCT->>Project: Run Option C linter (AST-based)

    Dev->>OCT: oct format --fix
    OCT->>Project: Auto-fix compliance violations

    Dev->>OCT: oct health
    OCT->>Project: Aggregate lint, fixable, docs, tests, compat

    Dev->>OCT: oct test
    OCT->>Project: Run test suite (or discover and run sub-project suites)

    Dev->>OCT: oct git commit -m "feat: initial"
    OCT->>Project: Quality gate + secrets check + commit

    Dev->>OCT: oct export-source
    OCT->>Project: Generate source export

    Dev->>OCT: oct export-skeleton
    OCT->>Project: Generate structural skeletons (headers, signatures, docstrings)

    Project->>Diag: Import _dbg, instrumentation, middleware

11. Future Enhancements#

Implemented:

  • ✅ oct new --test — auto-generate companion test file alongside module (FS-203) — v0.8.0

  • ✅ oct diag — diagnostics tools: validate-config, list-domains, set-level (FS-204) — v0.8.0

  • ✅ oct lint --changed — lint only git-modified files (FS-209) — v0.8.0

  • ✅ oct docs --sphinx — full Sphinx documentation pipeline — v0.8.1

  • ✅ oct docs --doxygen — Doxygen integration with INPUT path patching

  • ✅ oct docs --all — combined Sphinx + Doxygen generation

  • ✅ oct docs --clean — remove generated documentation artifacts

  • ✅ no_hardcoded_secrets — AST-based secret detection linter rule (§6b)

  • ✅ NEVER_EXPORT_FILE_PATTERNS — file-level export exclusion (Layer 2, DD-05)

  • ✅ oct export-skeleton — AST-based skeleton export (headers, docstrings, signatures) with --no-diagnostics — v0.9.1

  • ✅ oct git status — Git status with OCT compliance annotations — v0.11.0

  • ✅ oct git check — Unified quality gate (lint + format + secrets) — v0.11.0

  • ✅ oct git init — Repository initialisation with Option C conventions — v0.11.1

  • ✅ oct format --changed — Format only git-modified files — v0.11.1

  • ✅ oct git commit — Conventional Commits with non-bypassable secrets — v0.12.0

  • ✅ oct git hooks — Hook management (install/status/remove) — v0.12.0

  • ✅ Pre-commit hook integration (FS-206) — v0.12.0

  • ✅ oct git changelog — Keep a Changelog from Conventional Commits — v0.13.0

  • ✅ Branch naming enforcement — advisory/blocking by profile — v0.13.0

  • ✅ Branch-aware auto-profile (pre_commit_profile: "auto") — v0.13.0

  • ✅ oct health git section — 7 git health checks — v0.13.0

  • ✅ oct export-source --diff REF — export changed files only — v0.13.0

  • ✅ oct test --changed — run tests for changed files — v0.13.0

  • ✅ .octrc.json git schema validation — v0.13.0

  • ✅ oct lint --explain <rule> — inline rule documentation with registry (FS-509) — v0.20.0

  • ✅ Shannon entropy-based secret detection (FS-507) — v0.21.0

  • ✅ Incremental lint cache + lint baseline (FS-510, FS-508) — v0.22.0

  • ✅ Standalone secret scanner extraction (OI-514) — oct/tools/secret_scanner.py

  • ✅ Per-context entropy thresholds — code ≥ 4.5, docstring > 5.0, comment > 5.0 (reserved)

  • ✅ entropy_exclusions in .octrc.json — field-name + path-based exclusions (OI-540)

  • ✅ Shared inline waiver module — oct/core/waivers.py (OI-540)

  • ✅ Quality gate honours inline waivers universally (OI-540)

Planned:

  • Per-line waiver matching — waivers matched to specific finding line numbers instead of file-level suppression

  • Defence-in-depth censorship module for exports/docs (--unsafe opt-out) — DD-05 Layer 3

  • Linter plugin system for custom rules (FS-205)

  • oct deps enhancements (FS-208)

Deferred:

  • Export diff mode (FS-207)

  • Debug session recorder

12. Appendix: Example OCT-Generated File#

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# myproject/src/example.py

"""
Example Module

Purpose
-------
Demonstrate Option C file structure.

Responsibilities
----------------
- Provide example function
- Demonstrate diagnostics usage

Diagnostics
-----------
Domain: EXAMPLE
Levels:
    L2 -- lifecycle
    L3 -- semantic details
    L4 -- deep tracing

Contracts
---------
- Must not mutate inputs
"""

from oc_diagnostics import _dbg


def example(x):
    func = "example"
    _dbg("EXAMPLE", func, f"START x={x}", 2)
    return x * 2