Web tools for URL fetching and DuckDuckGo search.
Purpose
-------
Provides :func:`fetch_url` (cleaned text extraction via
``trafilatura``) and :func:`web_search` (DuckDuckGo instant answers).
Falls back to clear error messages when optional dependencies are
missing so the agent can decide whether to ask the user to install
or skip the web step.
Responsibilities
----------------
- :func:`fetch_url` — HTTP GET with redirect-follow, then strip
boilerplate via ``trafilatura.extract`` (or fall back to raw HTML
when extraction is unavailable).
- :func:`web_search` — DuckDuckGo instant-answer JSON API; returns
the abstract + related topics.
- ``TOOLS`` and ``SCHEMAS`` for auto-registration.
Requires: ``pip install httpx trafilatura``
Diagnostics
-----------
Domain: OC_CHAT.TOOLS
Levels:
L2 — lifecycle events
L3 — semantic details
L4 — deep tracing
Contracts
---------
- Missing optional deps return ``{"error": "<package> not installed.
Run: pip install <package>"}`` rather than raising.
- ``fetch_url`` honours ``max_chars`` (default 10000) and signals
truncation in the return shape.
- HTTP failures are caught and surfaced as ``{"error": "..."}``
rather than propagating.
- All network I/O has a finite timeout (default 15 s).
Dependencies
------------
- oc_diagnostics (structured debug logging via _dbg)
- oc_chat.core.cancellation (lazy; tool_cancel_requested - skips the request once a Ctrl+C has arrived)
- oc_chat.core.egress_guard (lazy; validate_url_egress, EgressDenied and MAX_REDIRECTS - the public-host egress check on every hop)
- oc_chat.core.external_content_gate (lazy; the external-content trust gate for each fetch hop and the search source)
- oc_chat.core.resource_caps (lazy; clamp_bytes, clamp_results and MAX_BYTES - the size and result-count ceilings)
- oc_chat.tools._local_fetch (lazy; local-mode admission, its named refusal and the bounded response headers)
- oc_chat.tools._sandbox (lazy; redact_secrets - redacts the search query and error text)
- httpx (lazy; the HTTP client for both tools, optional)
- trafilatura (lazy; boilerplate stripping for fetch_url, optional)
See Also
--------
oc_chat/docs/conventions/CONTRACTS.md §3 Tool registry contract; §4 Tool return-
value conventions.
| bool oc_chat.tools.web._validate_hop |
( |
str | url, |
|
|
* | , |
|
|
str | source_label ) |
|
protected |
Validate ONE fetch hop and return ``True`` if it was admitted as LOCAL.
The single place the per-hop destination rule lives, so the initial URL and
every redirect target are judged by exactly the same code:
- a **local candidate** (``http``/``https`` on ``localhost`` / ``127.0.0.1``
/ ``::1``) must be admitted by :func:`~oc_chat.tools._local_fetch.admit_local_url`
— the ``permissive`` profile plus an explicit, allowlisted port — or it
raises :class:`~oc_chat.tools._local_fetch.LocalFetchRefused`. A candidate is
NEVER re-judged as a public target, so its refusal names the real reason
(plan 232 P4) instead of the generic non-public denial;
- everything else goes through the unchanged M8.0 WU-4 egress guard
(§5.1.3, F2/F2a), which raises :class:`~oc_chat.core.egress_guard.EgressDenied`
for a non-``https`` scheme, URL credentials, or any host resolving to a
non-public address — ``127.0.0.2``, RFC1918, link-local and cloud-metadata
included, whatever the port allowlist says.
Definition at line 66 of file web.py.
| dict oc_chat.tools.web.fetch_url |
( |
str | url, |
|
|
int | max_chars = 10_000 ) |
Fetch a URL and return its cleaned text content.
M8.0 WU-4 (§5.1.3, F2/F2a): ``https``-only; URL credentials refused; the host
is resolved and refused if it maps to a non-public address (loopback / RFC1918
/ link-local / cloud-metadata); redirects are followed MANUALLY with
``follow_redirects=False`` and each hop re-validated; ``trust_env=False`` so a
poisoned ``HTTP(S)_PROXY`` / ``SSL_CERT_*`` cannot redirect or MITM the fetch.
Plan 232 / WEB-01 adds the ONE operator-ruled exception: a **local mode** for
a delivery check against a dev server. A hop on ``localhost`` / ``127.0.0.1``
/ ``::1`` is admitted only under the ``permissive`` profile and only when its
explicit port is listed in the visible per-project file
``.oc_chat/local_fetch.json`` (:mod:`oc_chat.tools._local_fetch`); the
admitted result additionally carries ``status``, a bounded ``headers``
allowlist and ``local: True``, and a non-2xx status is returned as DATA
(a 404 is exactly what a delivery check needs to see). Nothing else changes:
every other address stays refused, the public path stays ``https``-only, and
a local candidate that is not admitted refuses BY NAME rather than falling
through to the public path.
Definition at line 95 of file web.py.
1.15.0