oc_diagnostics Architecture Document

oc_diagnostics Architecture Document#

Version: 1.5 Status: Authoritative Reference Scope: Defines the architecture, structure, and design philosophy of the oc_diagnostics runtime package.

Table of Contents#

1. Overview#

oc_diagnostics is the unified debugging, tracing, and instrumentation system for all Option C‑compliant projects.

It provides:

  • _dbg — structured debug logging

  • domain and level system

  • ID instrumentation for Dash applications

  • callback middleware

  • UI event logging

  • a self‑test suite

It is a runtime package. OCT is a development‑time package. Projects consume both.

oc_diagnostics is project‑agnostic and contains no assumptions about any specific application.

2. High-Level Architecture#

        flowchart TD

    subgraph CONFIG["Project Diagnostics Config"]
        CFG["oc_diagnostics/debug_config.json"]
    end

    subgraph CORE["oc_diagnostics Core"]
        UDEBUG["unified_debug.py\n(_dbg, domains, levels,\nstdout/stderr interception,\nexception hook)"]
    end

    subgraph INSTR["Instrumentation System (Dash)"]
        IDR["id_instrumentation.py\n(ID rewriting)"]
        MIDDLE["instrumentation_middleware.py\n(callback patching)"]
        LOGGER["ui_event_logger.py\n(UI event capture)"]
        SELFTEST["Self-Test Suite\n(selftest_cli / run_tests.py)"]
    end

    subgraph FASTAPI["FastAPI / ASGI Integration"]
        FMIDDLE["fastapi_middleware.py\n(ASGI middleware)"]
    end

    CFG --> UDEBUG
    UDEBUG --> IDR
    UDEBUG --> MIDDLE
    UDEBUG --> LOGGER
    IDR --> MIDDLE
    MIDDLE --> LOGGER
    UDEBUG --> FMIDDLE

3. Directory Structure#

oc_diagnostics/
    oc_diagnostics/
        __init__.py
        unified_debug.py
        logging_handler.py
        id_instrumentation.py
        instrumentation_middleware.py
        ui_event_logger.py
        instrumentation_selftest.py
        instrumentation_selftest_cli.py
        instrumentation_selftest_app.py
        fastapi_middleware.py
    tests/
        run_tests.py
        Tests_README.md
    tools/
        migrate_to_oc_diagnostics.py
    docs/
        ARCHITECTURE.md
        CHANGELOG.md
        REFERENCE.md
        INTEGRATION_GUIDE.md
        debug_config.example.json
    README.md
    pyproject.toml

This structure is intentionally flat to keep imports simple:

from oc_diagnostics import _dbg
from oc_diagnostics import apply_instrumentation

4. Core Components#

4.1 unified_debug.py#

Responsibilities:

  • _dbg(domain, func, message, level) — synchronous structured debug logging

  • _adbg() — async variant; captures caller frame synchronously, offloads file I/O via asyncio.to_thread (FS-16)

  • domain‑level filtering

  • colorized output

  • timestamps

  • filename + line number inclusion

  • stdout/stderr interception

  • uncaught exception logging

  • output routing (terminal, file, both, json)

  • JSONL output mode — writes structured JSON records to file, colored text to terminal (FS-07)

  • loading project‑specific debug_config.json; optional validation with jsonschema (FS-18)

  • config hot-reload — daemon thread polls config mtime every 2 s when init(watch_config=True) (FS-09)

  • production mode (PROD_MODE) — suppresses stream interception and exception hook when env var or config "mode" is "prod" (FS-15)

  • runtime level override API: set_debug_level(), set_global_setting() (FS-03)

  • dbg_timed() context manager and dbg_profile() decorator for inline performance measurement (FS-12)

  • log rotation — count-based pruning (max_log_files) and size-based roll-over (max_log_size_mb) (FS-06)

  • HTTP header redaction — HTTP_REDACT_HEADERS frozenset; sensitive headers shown as [REDACTED] at L4 (FS-17)

  • configurable UI event properties (UI_EVENT_PROPERTIES) loaded from config (FS-10)

  • instrumentation_allow_all flag — instrument non-Dash objects without raising debug level (FS-13)

  • thread-safe log file writes (_log_lock protects all file I/O; LOG_FILE is initialized lazily on first write)

  • one-shot RuntimeWarning when _dbg() is called before init() (surfacing misconfiguration)

  • export ADDITIONAL_STRUCTURAL_PREFIXES and ADDITIONAL_STRUCTURAL_EXACT for config-driven structural ID extension

This module is the foundation of the entire diagnostics system.

4.2 id_instrumentation.py#

Responsibilities:

  • rewrite Dash component IDs into pattern‑matching dicts

  • preserve structural IDs

  • recurse through component trees

  • respect instrumentation levels

  • support custom components at higher debug levels

  • enforce MAX_INSTRUMENT_DEPTH (default: 500) to prevent RecursionError on pathologically deep or circular component trees; a L1 diagnostic is emitted when the limit is reached

  • merge project-defined structural ID extensions (ADDITIONAL_STRUCTURAL_PREFIXES, ADDITIONAL_STRUCTURAL_EXACT) loaded from debug_config.json into is_structural_id() checks

  • respect INSTRUMENTATION_ALL flag — instrument non-Dash objects without requiring a higher debug level (FS-13)

Instrumentation is safe, deterministic, and project‑agnostic.

4.3 instrumentation_middleware.py#

Responsibilities:

  • patch Dash callback dispatcher

  • wrap callback outputs with instrumentation

  • apply instrumentation to any component tree — including list and tuple callback returns (each element instrumented individually; container type preserved)

  • integrate with UI event logger

Middleware is opportunistic; explicit instrumentation remains authoritative.

4.4 ui_event_logger.py#

Responsibilities:

  • capture all user interactions

  • listen to instrumented IDs ({"event": "<id>"})

  • log property changes for the set defined by UI_EVENT_PROPERTIES in global_settings (default: n_clicks, value, data, figure, selectedData) (FS-10)

  • route all events through _dbg

  • expose REQUIRED_STORE_ID as a public constant ("debug-settings-data-store") for programmatic use

  • emit a L1 diagnostic warning at registration time when REQUIRED_STORE_ID is absent from app.layout

  • truncate large event values before logging; summarise figure and data properties as type+length when DEBUG_LEVELS["UI"] < 9

This module provides a universal UI event capture mechanism.

4.5 Self-Test Suite#

The self-test suite spans two locations:

oc_diagnostics/ (package-level):

  • instrumentation_selftest_cli.py — headless, deterministic CLI test runner; validates ID rewriting, middleware, structural protection, and debug-level transitions; exits 0 on success, 1 on failure; suitable for CI/CD.

  • instrumentation_selftest_app.py — manual integration harness; launches a Dash development server for visual validation; not for automated testing.

  • instrumentation_selftest.py — lightweight manual smoke-test entry point.

tests/ (project-level):

  • run_tests.py — primary automated test runner; covers pure-Python core tests (no Dash required), FastAPI ASGI contract tests (no optional dependencies required), and optional Dash tests that auto-skip when Dash is absent; invokes instrumentation_selftest_cli as a subprocess; exits 0 on full success, 1 on any failure; suitable for CI/CD integration.

The test suite ensures the entire system behaves consistently across environments.

4.6 fastapi_middleware.py#

Responsibilities:

  • implement DiagnosticsMiddleware as a pure ASGI callable — no dependency on starlette.BaseHTTPMiddleware; works with FastAPI, Starlette, or any ASGI framework

  • log request entry (method, path, client IP) at L2

  • log User-Agent at L3; log full request and response headers at L4

  • capture HTTP status code from the http.response.start ASGI event

  • measure and report elapsed wall-clock time in milliseconds

  • pass non-HTTP scopes (WebSocket, lifespan) through unchanged

  • log application exceptions at L2 and re-raise them

  • redact sensitive request and response headers at L4 using HTTP_REDACT_HEADERS from unified_debug; affected headers are displayed as [REDACTED] (FS-17)

Install the optional extra to bring in FastAPI itself:

pip install oc_diagnostics[fastapi]

Enable HTTP logging by adding the HTTP domain to oc_diagnostics/debug_config.json:

"HTTP": { "level": 2, "color": "blue" }

4.7 logging_handler.py#

Responsibilities:

  • implement OcDiagnosticsHandler — a logging.Handler subclass that routes Python standard-library log records through _dbg() (FS-08)

  • accept a configurable domain (default: "GENERAL") and an optional level_map mapping logging level integers to _dbg diagnostic levels

  • default level map: DEBUG → 4, INFO → 2, WARNING/ERROR/CRITICAL → 1

  • always available — pure Python, no optional dependencies required

This module bridges the standard logging ecosystem with the oc_diagnostics domain-and-level system so third-party libraries and legacy code can participate in the same diagnostic pipeline without modification.

5. Diagnostics Configuration#

Each project must include:

oc_diagnostics/debug_config.json

Example:

{
  "domains": {
    "DATA": { "level": 3, "color": "cyan" },
    "UI":   { "level": 4, "color": "blue" },
    "INSTRUMENTATION": 3,
    "HTTP": { "level": 2, "color": "blue" }
  },
  "global_settings": {
    "silent_mode": false,
    "enable_timestamps": true,
    "enable_colors": true,
    "include_filename": true,
    "output_mode": "both",
    "instrumentation_enabled": true,
    "max_log_files": 10,
    "max_log_size_mb": 5.0,
    "instrumentation_allow_all": false,
    "mode": "dev",
    "http_redact_headers": ["authorization", "cookie", "set-cookie", "x-api-key"]
  }
}

The configuration file is project‑owned, not package‑owned.

6. Import Surface#

oc_diagnostics exposes only the public API:

from oc_diagnostics import _dbg                    # synchronous structured logging
from oc_diagnostics import _adbg                   # async variant (FS-16)
from oc_diagnostics import DEBUG_LEVELS
from oc_diagnostics import init
from oc_diagnostics import set_debug_level         # runtime domain level override (FS-03)
from oc_diagnostics import set_global_setting      # runtime global setting override (FS-03)
from oc_diagnostics import dbg_timed               # timing context manager (FS-12)
from oc_diagnostics import dbg_profile             # timing decorator (FS-12)
from oc_diagnostics import OcDiagnosticsHandler    # Python logging integration (FS-08)
from oc_diagnostics import apply_instrumentation   # requires: pip install oc_diagnostics[dash]
from oc_diagnostics import DiagnosticsMiddleware   # requires: pip install oc_diagnostics[fastapi]

Both optional exports (apply_instrumentation, DiagnosticsMiddleware) raise a clear ImportError at call time when their respective optional dependency is absent. All other exports are always available — no optional dependency required.

Instrumentation constants and helpers are imported explicitly when needed:

from oc_diagnostics.id_instrumentation import instrument_ids, MAX_INSTRUMENT_DEPTH
from oc_diagnostics.ui_event_logger import register_callbacks, REQUIRED_STORE_ID
from oc_diagnostics.unified_debug import ADDITIONAL_STRUCTURAL_PREFIXES, ADDITIONAL_STRUCTURAL_EXACT
from oc_diagnostics.unified_debug import HTTP_REDACT_HEADERS, UI_EVENT_PROPERTIES

This keeps the public surface clean and predictable.

7. Lifecycle & Integration#

A typical Option C project uses diagnostics as follows:

  1. Project defines oc_diagnostics/debug_config.json

  2. Project imports _dbg

  3. Project builds static layout (uninstrumented)

  4. Project registers UI event logger

  5. Project registers callbacks

  6. Project enables instrumentation middleware

  7. Dynamic content is instrumented explicitly

This lifecycle ensures correctness and avoids ID‑mismatch issues.

8. Future Enhancements#

Approved:

  • diagnostics context object (FS-19)

  • integration with OCT’s oct diag command (OI-26, OI-27)

  • richer event metadata

Deferred:

  • debug session recorder

9. Appendix: Mindmap Overview#

        mindmap
  root((oc_diagnostics))
    Debugging
      _dbg
      _adbg (async)
      domains
      levels
      timestamps
      colors
      stdout/stderr interception
      exception hook
      thread-safe file writes
      pre-init RuntimeWarning
      JSONL output mode
    Runtime API
      set_debug_level
      set_global_setting
    Profiling
      dbg_timed context manager
      dbg_profile decorator
    Config
      hot-reload (watch_config)
      production mode (PROD_MODE)
      log rotation (count + size)
      jsonschema validation
    Instrumentation
      id rewriting
      structural ID protection
      project-defined structural IDs (FS-02)
      instrumentation_allow_all (FS-13)
      recursion
      depth limit (MAX_INSTRUMENT_DEPTH)
      custom components
    Middleware
      callback patching
      dynamic instrumentation
      list/tuple output handling
    UI Event Logger
      event capture
      configurable properties (UI_EVENT_PROPERTIES)
      REQUIRED_STORE_ID
      startup store validation
      value truncation
    Self-Test Suite
      CLI tests (selftest_cli)
      Dash app tests (selftest_app)
      automated runner (run_tests.py)
      FastAPI ASGI tests
      debug-level transitions
    FastAPI Integration
      ASGI middleware
      HTTP domain logging
      header redaction (HTTP_REDACT_HEADERS)
      optional extra
    Python logging Integration
      OcDiagnosticsHandler
      domain routing
      level mapping