waivers.py

Go to the documentation of this file.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# oct/core/waivers.py
 
"""
Purpose
-------
OI-540 - shared inline-waiver parsing for OCT-LINT comments.
Plan 193 - waiver SCOPE: a waiver excuses the finding beside it, not the
whole file.
 
Extracted from :mod:`oct.linter.oct_lint` so that both the linter and
the quality gate (:mod:`oct.git.quality_gate`) can import the same
parser without the quality gate depending on the linter.
 
Plan 193 (sec) closed a hole this module sat at the centre of. The
parser always returned line numbers, but BOTH consumers threw them away
and treated any waiver as whole-file suppression - so a real credential
pasted anywhere into one of the ~200 files carrying a
``no_hardcoded_secrets`` waiver committed clean. Scope is now:
 
- ``# OCT-LINT: disable=<rule> reason="…"`` - LINE scope. Covers its own
  line and the immediately following line (:data:`WAIVER_LINE_SPAN`), so
  both trailing-comment and comment-above placements work and the waiver
  travels with the code it excuses.
- ``# OCT-LINT: disable-file=<rule> reason="…"`` - FILE scope. Whole-file
  blindness stays available, but as a deliberate, greppable choice.
- A bare ``disable=`` waiver adjacent to NO finding of its rule keeps its
  historical file-wide power (so the pre-193 tree does not storm the gate
  in one landing) and reports its line via
  :attr:`WaiverOutcome.legacy_lines`, which the linter surfaces as the
  advisory ``legacy_file_scoped_waiver`` burn-down finding. When that
  count reaches zero the legacy branch can be deleted as a no-op.
 
Reasonless waivers remain inert in both forms (pre-existing rule).
 
Responsibilities
----------------
- Host :data:`_WAIVER_RE` - the canonical regex for
  ``# OCT-LINT: disable[-file]=<rule> [reason="..."]`` comments.
- Provide :func:`_comment_tokens` - tokenise source so only real
  comments can carry a waiver.
- Provide :func:`parse_inline_waivers` - returns a mapping of line
  numbers to ``{"rule": str, "reason": str|None, "scope": str}`` dicts.
- Provide :func:`inline_disabled_rules` - the FILE-WIDE answer the
  linter reaches from that mapping, written once so every consumer
  reaches it by the same route (plan 277).
- Provide :func:`split_scanner_findings` - recover per-finding line
  numbers from the secret scanner's ``"; "``-joined message.
- Provide :func:`apply_waivers` - scope waivers over those findings and
  return a :class:`WaiverOutcome`.
 
Diagnostics
-----------
Domain: OCT-CORE
Levels:
    L2 - (none) pure parsing helper; it reports no lifecycle of its own
    L3 - (none) callers report waiver outcomes in their own domains
    L4 - parse and scope entry: input size, and the rule being scoped
 
Contracts
---------
- Must not import :mod:`oct.linter.oct_lint` (no circular dependency).
- Return shape is stable and additive - both linter and quality gate
  depend on it. Plan 193 added the ``"scope"`` key; existing readers of
  ``"rule"`` / ``"reason"`` are unaffected.
- This module decides waiver SCOPE only. It contains no threshold, no
  heuristic, and no secret pattern, and it never changes what the
  scanner detects - only which detected findings are excused.
- :func:`inline_disabled_rules` can never return a member of
  :data:`LINE_SCOPED_RULES`, by construction. Those rules are scoped per
  finding by :func:`apply_waivers` and must never be switched off
  wholesale by a consumer that reads this set (plan 277 - the reason
  G-D4 is untouched by the format pass honouring waivers).
- A waiver is only honoured when it appears in a real ``COMMENT``
  token (193 R1). Waiver-shaped text inside a string literal is inert,
  so a file that documents or tests the waiver syntax cannot waive
  itself. Unparseable input yields NO waivers (fail closed).
- :func:`apply_waivers` is fail-closed: any finding whose line cannot be
  recovered (line ``0``) is unreachable by line scope and survives
  unless an explicit file-scope waiver covers it.
 
Dependencies
------------
- re (waiver + finding-prefix pattern matching)
- tokenize (comment-token extraction - 193 R1)
- io (StringIO line reader fed to tokenize - 193 R1)
- dataclasses (the frozen WaiverOutcome record)
- oct.core.diagnostics (_dbg structured logger)
"""
 
from __future__ import annotations
 
import io
import re
import tokenize
from dataclasses import dataclass
from oct.core.diagnostics import _dbg
 
 
#: A line-scoped waiver covers its own line and the next one. Two lines
#: is what makes both placements work: a trailing comment sits ON the
#: offending line, a standalone comment sits ABOVE it.
WAIVER_LINE_SPAN = 2
 
#: Rules whose findings carry recoverable line numbers, and which are
#: therefore scoped per-line. Every other rule keeps whole-file waiver
#: semantics - its check reports a verdict, not located findings, so
#: there is nothing to scope against.
LINE_SCOPED_RULES = frozenset({"no_hardcoded_secrets"})
 
_WAIVER_RE = re.compile(
    # OCT-LINT: disable=no_hardcoded_secrets reason="Waiver regex pattern, not a secret"
    r'#\s*OCT-LINT:\s*disable(?P<file_scope>-file)?\s*=\s*(?P<rule>[a-z_]+)(?:\s+(?P<reason>.*))?',
    re.IGNORECASE,
)
 
#: The scanner prefixes every violation with ``L<lineno>: ``.
_FINDING_PREFIX_RE = re.compile(r'^L(?P<lineno>\d+):\s')
 
#: Cheap pre-filter for :func:`inline_disabled_rules`. Every waiver must
#: reach :data:`_WAIVER_RE`, which requires this marker, so text without
#: it cannot carry one and need not be tokenised at all. Case-insensitive
#: because ``_WAIVER_RE`` is: the two must agree, or the fast path would
#: silently drop a waiver the slow path honours.
_WAIVER_MARKER_RE = re.compile(r'OCT-LINT', re.IGNORECASE)
 
 
@dataclass(frozen=True)
class WaiverOutcome:
    """The result of scoping a rule's waivers over a scanner message.
 
    ``surviving`` and ``suppressed`` are ``(lineno, text)`` pairs; a
    ``lineno`` of ``0`` means the scanner reported no location (e.g. the
    unparseable-file message), which no line waiver can reach.
 
    ``legacy_lines`` lists the bare ``disable=`` waivers that suppressed
    findings ONLY by falling back to pre-193 whole-file semantics. It is
    empty whenever the legacy branch was not exercised - including when
    a stale waiver had nothing to suppress - so it counts exactly the
    files that would change verdict if the legacy branch were deleted.
    """
 
    surviving: tuple[tuple[int, str], ...]
    suppressed: tuple[tuple[int, str], ...]
    legacy_lines: tuple[int, ...]
 
    @property
    def blocked(self) -> bool:
        """True when at least one finding survived every waiver."""
        return bool(self.surviving)
 
    @property
    def message(self) -> str:
        """The surviving findings re-joined in the scanner's own format."""
        return "; ".join(text for _lineno, text in self.surviving)
 
    @property
    def first_line(self) -> int:
        """Line of the first surviving finding (``0`` when none survive)."""
        return self.surviving[0][0] if self.surviving else 0
 
 
def _comment_tokens(text: str) -> list[tuple[int, str]]:
    """Return ``(lineno, comment_text)`` for every real comment in *text*.
 
    Plan 193 R1 - a waiver is a COMMENT, so only comment tokens are
    offered to the waiver regex. Waiver-shaped text inside a string
    literal (a docstring that documents the syntax, a test fixture that
    exercises it, a rule-registry example) is therefore inert, and can
    no longer silently waive the file that merely talks about waivers.
 
    FAIL CLOSED: if the source cannot be tokenised, this returns no
    comments at all, so the caller honours NO waivers and every finding
    survives. That matches the 170/176 doctrine - an unparseable file
    must never gain suppression it could not earn.
    """
    comments: list[tuple[int, str]] = []
    try:
        for token in tokenize.generate_tokens(io.StringIO(text).readline):
            if token.type == tokenize.COMMENT:
                comments.append((token.start[0], token.string))
    except (tokenize.TokenError, SyntaxError, IndentationError,
            ValueError, UnicodeDecodeError):
        return []
    return comments
 
 
def parse_inline_waivers(text: str) -> dict[int, dict]:
    """Parse ``# OCT-LINT: disable[-file]=<rule> [reason]`` comments from source.
 
    Returns a dict mapping line numbers (1-based) to
    ``{"rule": str, "reason": str|None, "scope": "line"|"file"}``.
 
    ``scope`` is ``"file"`` only for the explicit ``disable-file=`` form;
    the bare ``disable=`` form is ``"line"``. Waivers with no reason are
    still returned (the linter warns about them) but callers must treat
    them as inert.
 
    Plan 193 R1: only real comment tokens are considered - see
    :func:`_comment_tokens` for the rationale and the fail-closed
    behaviour on unparseable input.
    """
    func = "parse_inline_waivers"
    _dbg("OCT-CORE", func, f"scanning {len(text)} chars for waivers", 4)
    waivers: dict[int, dict] = {}
    for lineno, comment in _comment_tokens(text):
        m = _WAIVER_RE.search(comment)
        if m:
            reason = (m.group("reason") or "").strip() or None
            waivers[lineno] = {
                "rule": m.group("rule").lower(),
                "reason": reason,
                "scope": "file" if m.group("file_scope") else "line",
            }
    return waivers
 
 
def inline_disabled_rules(text: str) -> frozenset[str]:
    """Return the rules *text*'s inline waivers switch off for the whole file.
 
    This is the linter's own set, and the ONE place it is written. It is
    :func:`oct.linter.oct_lint._lint_single_file`'s ``waived_rules``
    verbatim - a waiver counts when it carries a reason and names a rule
    outside :data:`LINE_SCOPED_RULES`, and its ``scope`` is not consulted,
    because for a rule whose check reports a verdict rather than located
    findings there is nothing to scope against. That is why the bare
    ``disable=`` form disables its rule file-wide here exactly as the
    linter's ``_rule_active`` gate has always let it.
 
    Plan 277 added this so the quality gate's format pass
    (:func:`oct.git.quality_gate._run_format_pass`) and the health
    dashboard (:mod:`oct.health.oct_health`) can reach the linter's
    answer without restating it. Before, each pass knew a different
    subset of what excuses a file, and a module the operator had ruled
    exempt from ``dbg_import`` linted clean and was refused by the format
    pass on the same rule, in the same commit.
 
    A member of :data:`LINE_SCOPED_RULES` can never appear in the result.
    Those rules are excused per finding by :func:`apply_waivers` and a
    consumer that switched one off file-wide from here would be handing
    back exactly the whole-file blindness plan 193 (sec) closed.
 
    FAIL CLOSED on unparseable input: :func:`parse_inline_waivers` yields
    no waivers when the source cannot be tokenised, so this yields no
    disabled rules and every check runs.
    """
    func = "inline_disabled_rules"
    if not _WAIVER_MARKER_RE.search(text):
        return frozenset()
    disabled = frozenset(
        waiver["rule"] for waiver in parse_inline_waivers(text).values()
        if waiver["reason"] is not None
        and waiver["rule"] not in LINE_SCOPED_RULES
    )
    _dbg("OCT-CORE", func, f"waivers disable {sorted(disabled)}", 4)
    return disabled
 
 
def split_scanner_findings(message: str) -> list[tuple[int, str]]:
    """Split a scanner message into ``(lineno, text)`` findings.
 
    :func:`oct.tools.secret_scanner.check_no_hardcoded_secrets` joins its
    violations with ``"; "`` and prefixes each with ``L<lineno>: ``. A
    truncated preview may itself contain ``"; "``, so a fragment without
    that prefix is folded back into the preceding finding rather than
    being mistaken for a new one. A leading fragment with no prefix (the
    unparseable-file message) is reported at line ``0``, which no line
    waiver can reach.
    """
    findings: list[tuple[int, str]] = []
    if not message:
        return findings
    for fragment in message.split("; "):
        m = _FINDING_PREFIX_RE.match(fragment)
        if m is None and findings:
            lineno, text = findings[-1]
            findings[-1] = (lineno, f"{text}; {fragment}")
            continue
        findings.append((int(m.group("lineno")) if m else 0, fragment))
    return findings
 
 
def apply_waivers(message: str, waivers: dict[int, dict], rule: str) -> WaiverOutcome:
    """Scope *rule*'s waivers over the findings in *message*.
 
    *waivers* is the mapping returned by :func:`parse_inline_waivers`.
    Resolution order:
 
    1. An explicit ``disable-file=`` waiver for *rule* suppresses every
       finding (deliberate whole-file blindness).
    2. Each line waiver suppresses findings on its own line and the next
       (:data:`WAIVER_LINE_SPAN`).
    3. If findings survive AND at least one bare waiver matched nothing,
       that waiver falls back to pre-193 whole-file suppression and is
       reported in :attr:`WaiverOutcome.legacy_lines`.
 
    Reasonless waivers are ignored at every step.
    """
    func = "apply_waivers"
    _dbg("OCT-CORE", func, f"scoping {rule} waivers over the findings", 4)
    findings = tuple(split_scanner_findings(message))
    if not findings:
        return WaiverOutcome((), (), ())
 
    active = {
        lineno: w for lineno, w in waivers.items()
        if w.get("rule") == rule and w.get("reason")
    }
    if any(w.get("scope", "line") == "file" for w in active.values()):
        return WaiverOutcome((), findings, ())
 
    line_waivers = sorted(
        lineno for lineno, w in active.items()
        if w.get("scope", "line") == "line"
    )
 
    covered: set[int] = set()
    matched: set[int] = set()
    for waiver_line in line_waivers:
        for index, (finding_line, _text) in enumerate(findings):
            if finding_line and waiver_line <= finding_line < waiver_line + WAIVER_LINE_SPAN:
                covered.add(index)
                matched.add(waiver_line)
 
    surviving = tuple(f for i, f in enumerate(findings) if i not in covered)
    legacy_lines = tuple(ln for ln in line_waivers if ln not in matched)
 
    # Legacy ramp: a bare waiver that matched nothing keeps its pre-193
    # file-wide power, but only while findings would otherwise survive -
    # so a stale waiver with nothing to suppress is never counted.
    if surviving and legacy_lines:
        return WaiverOutcome((), findings, legacy_lines)
 
    suppressed = tuple(f for i, f in enumerate(findings) if i in covered)
    return WaiverOutcome(surviving, suppressed, ())