stdout_diversion.py

Go to the documentation of this file.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# oct/mcp/stdout_diversion.py
 
"""
Purpose
-------
Keep the stdio transport's protocol channel clean while the ``oct-mcp``
server starts. Under the ``stdio`` transport, standard output (file
descriptor 1) carries JSON-RPC messages, and the MCP specification says
a stdio server must not write anything else there. While
:func:`mcp.server.stdio.stdio_server` serves, the MCP Python SDK already
points fd 1 at standard error and writes the protocol through a private
duplicate. Before that, from process start until
:meth:`mcp.server.MCPServer.run` enters the transport, nothing protected
fd 1, so a ``_dbg`` record from startup (a lifecycle line with the MCP
domain at level 2 or higher, any L1 startup error, the production-mode
downgrade warning, or the FS-25 fallback warning) was written onto the
wire ahead of the ``initialize`` response. This module covers that
startup window with the same mechanism the SDK uses while serving.
 
Responsibilities
----------------
- :func:`divert_stdout`: keep a private duplicate of fd 1 (the wire),
  always above the standard descriptors, and point fd 1 at standard
  error, or at the null device when standard error is not open. Every
  write to fd 1 then misses the wire, whoever makes it: ``_dbg`` in any
  output mode, ``print()``, a third-party library or a child process
  that inherits standard output.
- :func:`restore_stdout`: flush Python's ``sys.stdout`` while fd 1 is
  still diverted, discarding into the null device whatever standard
  error rejects, then point fd 1 back at the wire and close the private
  duplicate. :func:`oct.mcp.server.start_server` calls it immediately
  before ``MCPServer.run(transport="stdio")``, whose ``stdio_server()``
  claims fd 1 again at once.
- :func:`stdout_is_diverted`: report whether a diversion is still
  recorded, so :func:`oct.mcp.server.start_server` can refuse to serve
  when the hand-back failed.
 
Diagnostics
-----------
Domain: MCP
Levels:
    L1 - errors: fd 1 could not be pointed back at the wire
    L2 - lifecycle: diversion started, fd 1 handed back to the transport;
         EXTERNAL_ERROR when standard error rejected buffered output
    L3 - (none): the lifecycle records are the whole trace
    L4 - (none): no deep trace is emitted
 
Every record is written while fd 1 is diverted, so none of them can
reach the wire: the hand-back record is emitted before fd 1 moves.
 
Contracts
---------
- None of the three public functions raises. A failed diversion leaves
  fd 1 carrying the wire. A failed hand-back leaves the diversion
  recorded, with fd 1 still off the wire (on standard error or the null
  device, or closed when a failing Windows ``dup2`` closed it first);
  :func:`stdout_is_diverted` then returns ``True`` and
  :func:`oct.mcp.server.start_server` exits with code 1 instead of
  letting the SDK serve the protocol on standard error.
- :func:`divert_stdout` returns ``True`` only when it started a
  diversion. It returns ``False``, with fd 1 carrying the wire, when a
  diversion is already active, when fd 1 is not an open descriptor, or
  when fd 1 can be pointed neither at standard error nor at the null
  device.
- The private duplicate never lands on descriptor 0 or 2. When one of
  them is closed and the duplicate lands there, that duplicate is held
  while another is taken and then closed again, so the standard
  descriptor stays closed. The SDK reaches the same result with
  ``F_DUPFD_CLOEXEC`` on POSIX; this form works on every platform.
- :func:`restore_stdout` returns ``True`` only when fd 1 carries the
  wire again after a diversion. Without an active diversion it does
  nothing and returns ``False``, so callers such as an in-process test
  of ``start_server`` can call it unconditionally.
- Bytes written into ``sys.stdout`` during the diversion are flushed
  before fd 1 moves back. When standard error rejects that flush (a
  broken or closed pipe), fd 1 is pointed at the null device and the
  flush repeated, so the rejected bytes are discarded instead of
  reaching the wire when the interpreter flushes at exit. Only when the
  null device cannot be opened either can such bytes stay buffered.
- ``sys.stdout`` itself is never replaced or wrapped: only the
  descriptor moves, so ``sys.stdout.buffer.fileno()`` stays 1 and the
  SDK's own claim of fd 1 still applies once the server runs.
- Bytes buffered in ``sys.stdout`` before the diversion starts are not
  flushed first, so they are diverted too; nothing legitimate is
  written to standard output before the transport starts.
- On Windows a child process inherits the Win32 standard-output handle,
  not fd 1. The C runtime of a console interpreter (``python.exe``)
  updates that handle whenever fd 1 changes, so a child started during
  the diversion writes to standard error too. The GUI-subsystem
  ``pythonw.exe`` does not; no child process is started during startup,
  and the SDK rebinds the handle itself while serving (through pywin32,
  when it is installed).
- Module state is not locked: the functions are called from the main
  thread during startup, before any tool call can run.
 
Dependencies
------------
- oct.core.diagnostics (_dbg structured logger)
"""
 
from __future__ import annotations
 
import os
import sys
from contextlib import suppress
 
from oct.core.diagnostics import _dbg
 
#: The standard output descriptor, which carries the protocol under stdio.
_STDOUT_FD: int = 1
 
#: The standard error descriptor, where diverted output goes.
_STDERR_FD: int = 2
 
#: Private duplicate of the original fd 1 while a diversion is active;
#: ``None`` otherwise.
_wire_fd: int | None = None
 
 
def _dup_above_std(fd: int) -> int:
    """Duplicate *fd* onto a descriptor above the standard range.
 
    A duplicate that lands on 0 or 2 (a closed standard descriptor) is
    held while the next one is taken and closed afterwards, so the wire
    never sits behind a standard descriptor and a closed one stays
    closed. With *fd* (fd 1) itself open, at most descriptors 0 and 2
    can be free below 3, so the third duplicate is always above the
    range.
 
    Raises
    ------
    OSError
        When *fd* cannot be duplicated, or when three duplicates in a
        row land in the standard range.
    """
    held: list[int] = []
    try:
        for _attempt in range(_STDERR_FD + 1):
            duplicate = os.dup(fd)
            if duplicate > _STDERR_FD:
                return duplicate
            held.append(duplicate)
        raise OSError(f"no duplicate of fd {fd} above the standard range")
    finally:
        for low in held:
            with suppress(OSError):
                os.close(low)
 
 
def _point_fd1_at(target_fd: int) -> bool:
    """Make fd 1 a duplicate of *target_fd*; return whether that worked."""
    try:
        os.dup2(target_fd, _STDOUT_FD)
    except OSError:
        return False
    return True
 
 
def _point_fd1_at_null_device() -> bool:
    """Make fd 1 a duplicate of the null device; return whether that worked."""
    try:
        null_fd = os.open(os.devnull, os.O_WRONLY)
    except OSError:
        return False
    try:
        return _point_fd1_at(null_fd)
    finally:
        with suppress(OSError):
            os.close(null_fd)
 
 
def _divert_fd1() -> str | None:
    """Point fd 1 at standard error, or at the null device as a fallback.
 
    Returns the name of the target, or ``None`` when neither worked.
    """
    if _point_fd1_at(_STDERR_FD):
        return "standard error"
    if _point_fd1_at_null_device():
        return "the null device"
    return None
 
 
def _flush_python_stdout() -> bool:
    """Flush ``sys.stdout`` and ``sys.__stdout__``.
 
    Returns ``False`` when a flush was rejected with :class:`OSError`,
    which leaves the bytes buffered. A closed stream holds no bytes, so
    its :class:`ValueError` counts as flushed.
    """
    flushed = True
    for stream in (sys.stdout, sys.__stdout__):
        if stream is None:
            continue
        try:
            stream.flush()
        except ValueError:
            continue
        except OSError:
            flushed = False
    return flushed
 
 
def divert_stdout() -> bool:
    """Point fd 1 at standard error and keep the wire in a private descriptor.
 
    Called by :func:`oct.mcp.__main__.main` under the ``stdio`` transport,
    right after argument parsing, so ``--help`` still prints to standard
    output but nothing written during startup reaches the protocol
    channel. :func:`restore_stdout` ends the diversion.
 
    Returns
    -------
    ``True`` when this call started a diversion. ``False`` when a
    diversion was already active or fd 1 could not be diverted; fd 1
    then still carries the wire.
    """
    global _wire_fd
    func = "divert_stdout"
    if _wire_fd is not None:
        return False
    try:
        wire = _dup_above_std(_STDOUT_FD)
    except OSError:
        # No open fd 1 (pythonw.exe without handles, a closed stdout): there
        # is no wire to protect.
        return False
    target = _divert_fd1()
    if target is None:
        # A failed Windows dup2 can close its target first, so make sure fd 1
        # carries the wire again before giving up.
        _point_fd1_at(wire)
        with suppress(OSError):
            os.close(wire)
        return False
    _wire_fd = wire
    _dbg(
        "MCP", func,
        f"standard output diverted to {target} until the stdio transport starts",
        2,
    )
    return True
 
 
def restore_stdout() -> bool:
    """End a diversion started by :func:`divert_stdout`.
 
    Emits its lifecycle record and flushes ``sys.stdout`` while fd 1 is
    still diverted, then points fd 1 back at the wire and closes the
    private duplicate. Called by :func:`oct.mcp.server.start_server`
    immediately before ``MCPServer.run(transport="stdio")``, and again
    by :func:`oct.mcp.__main__.main` on the way out, where it is a
    no-op unless startup failed.
 
    Returns
    -------
    ``True`` when fd 1 carries the wire again. ``False`` when no
    diversion was active, or when fd 1 could not be pointed back; in the
    second case an L1 record says so and the diversion stays recorded,
    which :func:`stdout_is_diverted` reports.
    """
    global _wire_fd
    func = "restore_stdout"
    wire = _wire_fd
    if wire is None:
        return False
    _dbg("MCP", func, "handing standard output back to the stdio transport", 2)
    if not _flush_python_stdout():
        # Standard error rejected the flush (a broken or closed pipe), so the
        # bytes are still buffered and the interpreter would flush them onto
        # the wire later. Discard them into the null device while fd 1 is
        # still off the wire.
        _dbg(
            "MCP", func,
            "EXTERNAL_ERROR: standard error rejected buffered output; "
            "discarding it into the null device",
            2,
        )
        if _point_fd1_at_null_device():
            _flush_python_stdout()
    if not _point_fd1_at(wire):
        _dbg(
            "MCP", func,
            "SYSTEM_ERROR: could not point standard output back at the stdio wire",
            1,
        )
        return False
    with suppress(OSError):
        os.close(wire)
    _wire_fd = None
    return True
 
 
def stdout_is_diverted() -> bool:
    """Return whether a diversion started by :func:`divert_stdout` is still recorded.
 
    ``True`` after a failed hand-back: fd 1 does not carry the wire, so
    the stdio transport must not start.
    """
    return _wire_fd is not None