commit_observer.py

Go to the documentation of this file.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# oct/git/commit_observer.py
 
"""
Purpose
-------
Detect silent mutations of tracked-but-unstaged files during the
``oct git commit`` pipeline (B1.0). Snapshots the working-tree state
of every tracked file outside the staged set at the start of the
commit, then compares again right before and right after the
underlying ``git commit`` invocation. Any byte-level change is
reported as either a warning or a hard abort, depending on the
``git.observe_unstaged_mutation`` octrc setting.
 
Two detection methods produce those snapshots (plan 242). They agree
on every verdict the control exists to produce; they differ only in
what they read to reach it.
 
``git`` (:data:`METHOD_GIT`)
    Ask git which tracked files differ from the index
    (``git diff --name-only -z``, answered from git's stat cache with
    the racy-git re-hash every ``git commit`` already trusts) and
    content-hash only those, plus the paths named in *also_hash*.
    Every other tracked file carries :data:`_CLEAN_SENTINEL`. About
    0.2 s per pass on a 13k-file tree.
 
``hash`` (:data:`METHOD_HASH`)
    SHA-256 every tracked-but-unstaged file. Byte-identical to the
    pre-242 behaviour apart from the two enumeration fixes below.
    188 s per pass on the same tree with the OS file cache cold.
 
The commit flow resolves the method from the octrc key
``git.observe_unstaged_mutation_method`` and always passes it
explicitly; ``git`` is the default when the key is absent. This
function's own parameter defaults to ``hash``, the more thorough of
the two, so that a caller who names no method never silently gets
less observation than it did before plan 242.
 
Responsibilities
----------------
- Provide :func:`snapshot_unstaged_tracked` - returns a mapping of
  repo-relative path -> sha256 hex digest or sentinel for every
  tracked file *not* in the staged set.
- Provide :func:`diff_snapshots` - returns the list of paths whose
  digest changed (new file, deleted file, or content drift) between
  two snapshots. Unchanged by plan 242; both methods feed it.
- Be defensive: large files (> :data:`_MAX_HASH_BYTES`) are recorded
  with the sentinel digest ``"OVERSIZE"`` so they don't dominate
  commit latency, and a missing/unreadable file maps to ``""`` so the
  diff path treats it consistently.
- Enumerate the index with ``git ls-files -z --stage``, which fixes
  two observation blind spots that were silent before plan 242:
 
  1. Without ``-z``, ``git ls-files`` quotes and octal-escapes any
     name containing a non-ASCII character (``core.quotepath``
     defaults to true). The observer stat-failed on the quoted
     string, recorded the missing sentinel, and never observed the
     file - a mutation of it was invisible to the control.
  2. Gitlink entries (mode ``160000``, a nested repository) came out
     of ``git ls-files`` looking like files. The observer tried to
     open the directory and logged a permission-denied L1 override on
     every snapshot. They are now skipped by mode, not by luck.
 
Diagnostics
-----------
Domain: GIT
Levels:
    L1 - errors: file IO failures during snapshot, unparsable index
         records, path escapes
    L2 - lifecycle: snapshot start / snapshot complete
    L3 - details: file count, hashed count, gitlinks skipped
    L4 - deep trace: per-file hash + size
 
Contracts
---------
- This module is pure: no Click, no subprocess of its own. It uses
  :func:`oct.core.git.git_run` to enumerate tracked files and, under
  :data:`METHOD_GIT`, to ask for the dirty set.
- Snapshot calls must never raise on individual-file IO errors -
  they record the file as unreadable and continue. A failure of git
  *itself* does propagate: the caller's fail-soft wrapper turns the
  observer off for that commit rather than silently switching method,
  which would reintroduce the cost this module exists to remove.
- An unknown *method* is a programming error and raises
  :class:`ValueError`; there is no permissive fallback in either
  direction.
- The hash digest is content-only (no metadata). Renames are detected
  via the path key, not the digest.
- A tracked path that resolves outside *repo_root* is dropped from
  the snapshot rather than hashed, under both methods. Plan 242
  computes that predicate once per distinct parent directory instead
  of once per file (:func:`_make_escape_check`) - the same answer,
  and the difference between a git-native pass costing 0.2 s and one
  costing 2.4 s on a 13k-entry index.
- Residual of :data:`METHOD_GIT`, accepted when the method was ruled
  (FACTORY-01 D3 option (a)): a writer that restores a file's size
  and mtime after writing evades git's stat cache where a SHA-256
  walk would not, and a change visible only through a content filter
  (``core.autocrlf``, a ``.gitattributes`` clean filter) is likewise
  not a difference in git's eyes. Such a writer is adversarial, and
  an adversarial writer can already disable the observer through
  octrc, so it was never in this control's threat model. Run under
  :data:`METHOD_HASH` for a paranoid pass.
- One documented verdict difference between the methods: an oversize
  file that is CLEAN at the pre snapshot is recorded as
  :data:`_CLEAN_SENTINEL` under ``git`` and as
  :data:`_OVERSIZE_SENTINEL` under ``hash``, so a mutation of it is
  flagged under ``git`` and not under ``hash`` (``diff_snapshots``
  cannot compare an OVERSIZE pair). ``git`` is strictly stronger;
  ``hash`` keeps the pre-242 answer by design.
 
Dependencies
------------
- oct.core.diagnostics (_dbg structured logger)
- oct.core.git (lazy; git_run - the only subprocess path out of this module)
- hashlib (SHA-256 content digests of tracked files)
- pathlib (Path for the repo root and tracked-file paths)
"""
 
from __future__ import annotations
 
import hashlib
from pathlib import Path
 
from oct.core.diagnostics import _dbg
 
 
#: Files larger than this are recorded with sentinel ``"OVERSIZE"``
#: rather than hashed. Keeps the snapshot cost bounded on repos with
#: large generated artefacts checked into git.
_MAX_HASH_BYTES: int = 10 * 1024 * 1024  # 10 MB
 
#: Sentinel digest used when a file exceeds :data:`_MAX_HASH_BYTES`.
_OVERSIZE_SENTINEL: str = "OVERSIZE"
 
#: Sentinel digest used when a file is missing or unreadable. Empty
#: string is distinct from any valid sha256 hex digest.
_MISSING_SENTINEL: str = ""
 
#: Sentinel recorded under :data:`METHOD_GIT` for a tracked file whose
#: working tree matches the index, i.e. one git itself calls clean.
#: Distinct from any sha256 hex digest and from the other sentinels.
_CLEAN_SENTINEL: str = "CLEAN"
 
#: Git-native detection: hash only what git reports as differing from
#: the index. The commit flow's default.
METHOD_GIT: str = "git"
 
#: Content-walk detection: SHA-256 every tracked-but-unstaged file.
METHOD_HASH: str = "hash"
 
#: Every accepted value of ``git.observe_unstaged_mutation_method``.
#: The commit flow validates against this tuple and refuses anything
#: else by name rather than falling back in either direction.
OBSERVE_METHODS: tuple[str, ...] = (METHOD_GIT, METHOD_HASH)
 
#: Index mode of a gitlink (nested repository / submodule) entry.
#: Such an entry is a directory on disk, never a blob, and is skipped.
_GITLINK_MODE: str = "160000"
 
#: Index mode of a symlink entry. The only entry kind whose own leaf
#: can resolve outside the repository, so the only one that needs a
#: per-entry resolve in :func:`_make_escape_check`.
_SYMLINK_MODE: str = "120000"
 
 
def _hash_file(path: Path) -> str:
    """Return the sha256 hex digest of *path*, or a sentinel.
 
    Returns :data:`_OVERSIZE_SENTINEL` when the file is larger than
    :data:`_MAX_HASH_BYTES`, and :data:`_MISSING_SENTINEL` when the
    file is missing or unreadable. Never raises.
    """
    func = "_hash_file"
    try:
        size = path.stat().st_size
    except OSError as exc:
        _dbg("GIT", func, f"stat failed: {path} ({exc})", 1, override=True)
        return _MISSING_SENTINEL
 
    if size > _MAX_HASH_BYTES:
        _dbg("GIT", func, f"oversize: {path} ({size} bytes)", 4)
        return _OVERSIZE_SENTINEL
 
    try:
        h = hashlib.sha256()
        with path.open("rb") as fh:
            while True:
                chunk = fh.read(65536)
                if not chunk:
                    break
                h.update(chunk)
        digest = h.hexdigest()
        _dbg("GIT", func, f"hashed {path} -> {digest[:8]}…", 4)
        return digest
    except OSError as exc:
        _dbg("GIT", func, f"read failed: {path} ({exc})", 1, override=True)
        return _MISSING_SENTINEL
 
 
def _tracked_entries(repo_root: Path) -> list[tuple[str, str]]:
    """Return ``[(mode, rel_path)]`` for every entry in the index.
 
    Enumerates with ``git ls-files -z --stage``, whose records are
    ``<mode> <object> <stage>\\t<path>`` separated by NUL. ``-z`` is
    what keeps a non-ASCII path out of ``core.quotepath``'s quoting
    and octal-escaping, and ``--stage`` is what makes a gitlink
    distinguishable from a blob. Raises whatever
    :func:`oct.core.git.git_run` raises - a git failure is the
    caller's fail-soft business, not this function's.
    """
    func = "_tracked_entries"
    from oct.core.git import git_run
 
    proc = git_run(["ls-files", "-z", "--stage"], cwd=Path(repo_root))
    entries: list[tuple[str, str]] = []
    for record in proc.stdout.split("\0"):
        if not record:
            continue
        meta, sep, rel = record.partition("\t")
        if not sep or not rel:
            _dbg(
                "GIT", func,
                f"unparsable ls-files record: {record!r}",
                1, override=True,
            )
            continue
        entries.append((meta.split(" ", 1)[0], rel))
    _dbg("GIT", func, f"index entries={len(entries)}", 4)
    return entries
 
 
def _dirty_paths(repo_root: Path) -> set[str]:
    """Return the repo-relative paths git reports as differing from the index.
 
    ``git diff --name-only -z`` compares the working tree against the
    index - the same question ``git commit`` itself asks - and
    includes deletions. Untracked files are absent by construction.
    """
    func = "_dirty_paths"
    from oct.core.git import git_run
 
    proc = git_run(["diff", "--name-only", "-z"], cwd=Path(repo_root))
    dirty = {p for p in proc.stdout.split("\0") if p}
    _dbg("GIT", func, f"dirty={len(dirty)}", 4)
    return dirty
 
 
def _make_escape_check(repo_root: Path):
    """Return ``is_inside(mode, rel) -> bool`` for index paths.
 
    Answers exactly the question the pre-242 code asked per file -
    "does this tracked path resolve inside *repo_root*?" - without
    paying a ``Path.resolve()`` syscall per file. A tracked path can
    only land outside the repository if one of its parent directories
    is a symlink out, or if its own leaf is one; git names the second
    case in the index (mode :data:`_SYMLINK_MODE`), so the first is
    resolved once per distinct parent directory and cached, and the
    second per entry.
 
    Measured on this monorepo's 13,394-entry index: 2.142 s of
    per-file resolves collapses to 0.048 s across 306 directories,
    which is what brings a git-native pass inside its budget. Only
    :class:`ValueError` is caught, matching the pre-242 behaviour -
    an :class:`OSError` still propagates to the caller's fail-soft.
    """
    func = "_make_escape_check"
    repo_resolved = Path(repo_root).resolve()
    dir_cache: dict[str, bool] = {}
    _dbg("GIT", func, f"escape check built for {repo_resolved}", 4)
 
    def _dir_inside(rel_dir: str) -> bool:
        func = "_make_escape_check._dir_inside"
        cached = dir_cache.get(rel_dir)
        if cached is None:
            try:
                (repo_root / rel_dir).resolve().relative_to(repo_resolved)
                cached = True
            except ValueError:
                _dbg(
                    "GIT", func,
                    f"path-escape directory: {rel_dir}", 1, override=True,
                )
                cached = False
            dir_cache[rel_dir] = cached
        return cached
 
    def is_inside(mode: str, rel: str) -> bool:
        if not _dir_inside(rel.rpartition("/")[0]):
            return False
        if mode != _SYMLINK_MODE:
            return True
        try:
            (repo_root / rel).resolve().relative_to(repo_resolved)
        except ValueError:
            return False
        return True
 
    return is_inside
 
 
def snapshot_unstaged_tracked(
    repo_root: Path,
    staged_set: set[str] | None = None,
    *,
    method: str = METHOD_HASH,
    also_hash: set[str] | None = None,
) -> dict[str, str]:
    """Return ``{rel_path: digest_or_sentinel}`` for tracked-but-unstaged files.
 
    *repo_root* must be the actual git repository root (not the OCT
    project root in monorepo layouts). *staged_set* is a set of
    repo-relative path strings already in the index; those files are
    excluded from the snapshot because their mutations are caller
    intent, not silent regression.
 
    *method* selects how each recorded value is produced -
    :data:`METHOD_HASH` content-hashes every entry, :data:`METHOD_GIT`
    content-hashes only what git reports as dirty (plus *also_hash*)
    and records :data:`_CLEAN_SENTINEL` for the rest. See the module
    docstring for the trade-off and the accepted residual. An unknown
    *method* raises :class:`ValueError`.
 
    *also_hash* is the pre snapshot's set of paths that carried
    anything other than :data:`_CLEAN_SENTINEL`. Passing it on the
    post snapshot is what keeps a hash-to-hash comparison exact for a
    file that was already dirty when the commit started: without it, a
    file reverted to its index content between the two snapshots would
    leave the dirty set and read as unchanged. Ignored under
    :data:`METHOD_HASH`, which hashes everything anyway.
 
    The snapshot includes:
      - Tracked files that are clean (working tree == index).
      - Tracked files modified but not staged (``M`` in
        ``git status -s`` second column).
 
    The snapshot excludes:
      - Files in *staged_set*.
      - Untracked files (``??``).
      - Gitlink entries (mode :data:`_GITLINK_MODE`) and worktree
        symlinks pointing outside *repo_root*.
    """
    func = "snapshot_unstaged_tracked"
    if method not in OBSERVE_METHODS:
        raise ValueError(
            f"unknown commit-observer method {method!r}; expected one "
            f"of {', '.join(OBSERVE_METHODS)}"
        )
    _dbg("GIT", func, f"repo_root={repo_root} method={method}", 2)
 
    if staged_set is None:
        staged_set = set()
 
    is_inside = _make_escape_check(repo_root)
    candidates: dict[str, Path] = {}
    gitlinks = 0
    for mode, rel in _tracked_entries(repo_root):
        if rel in staged_set:
            continue
        if mode == _GITLINK_MODE:
            # A nested repository, not a blob. Reading it as a file is
            # what produced a permission-denied L1 line on every
            # snapshot before plan 242.
            gitlinks += 1
            _dbg("GIT", func, f"gitlink skip: {rel}", 4)
            continue
        if not is_inside(mode, rel):
            # Path escape - drop, don't hash.
            _dbg("GIT", func, f"path-escape skip: {rel}", 1, override=True)
            continue
        candidates[rel] = repo_root / rel
 
    snapshot: dict[str, str] = {}
    if method == METHOD_HASH:
        for rel, full in candidates.items():
            snapshot[rel] = _hash_file(full)
        hashed = len(snapshot)
    else:
        to_hash = _dirty_paths(repo_root)
        if also_hash:
            to_hash = to_hash | set(also_hash)
        hashed = 0
        for rel, full in candidates.items():
            if rel in to_hash:
                snapshot[rel] = _hash_file(full)
                hashed += 1
            else:
                snapshot[rel] = _CLEAN_SENTINEL
 
    _dbg(
        "GIT", func,
        f"snapshot complete: {len(snapshot)} file(s) via {method} "
        f"(hashed={hashed}, staged={len(staged_set)} excluded, "
        f"gitlinks={gitlinks} skipped)",
        3,
    )
    return snapshot
 
 
def diff_snapshots(
    before: dict[str, str],
    after: dict[str, str],
) -> list[str]:
    """Return the list of paths whose digest changed between snapshots.
 
    Includes:
      - Paths present in both with different digests (mutation).
      - Paths present in *before* but missing from *after* (deletion or
        unreadable).
      - Paths newly tracked in *after* (rare; would only appear if
        ``git add`` ran during the commit pipeline).
 
    A path with the same :data:`_OVERSIZE_SENTINEL` on both sides is
    NOT flagged - we cannot tell whether the content of an oversize
    file changed without a full hash. Document this caveat to the
    caller via the per-file payload (omitted from the simple list
    return; see :func:`diff_snapshots_detail` for the verbose form).
 
    :data:`_CLEAN_SENTINEL` needs no special case: it compares as an
    ordinary value, so ``CLEAN`` on both sides is no mutation, and
    ``CLEAN`` against a digest (a file that entered or left the dirty
    set between the snapshots) is one. Plan 242 left this function
    unchanged on purpose - both detection methods feed it.
    """
    func = "diff_snapshots"
    keys = set(before.keys()) | set(after.keys())
    changed: list[str] = []
    for k in keys:
        b = before.get(k, _MISSING_SENTINEL)
        a = after.get(k, _MISSING_SENTINEL)
        if b == _OVERSIZE_SENTINEL and a == _OVERSIZE_SENTINEL:
            continue
        if b != a:
            changed.append(k)
    changed.sort()
    _dbg("GIT", func, f"changed={len(changed)}", 3)
    return changed