Commit Graph
5 Commits
Author SHA1 Message Date
BaaRai 374c985d06 agent: add ingest_scan_output for nmap XML files
A worker can write nmap output with -oX and call ingest_scan_output with
the format and path. The file is parsed deterministically: assets go
through the insert_assets handler, each host becomes one tool_output
fact listing its open ports, and the model gets only counters and at
most 20 notable items per type.

- path is resolved under the run working directory; "..", an absolute
  path outside it and a symlink leading outside are refused; 256 MiB cap
- the XML is streamed host by host with encoding/xml; a file cut short
  keeps the complete hosts and returns truncated: true
- target text (banners, products, hostnames) is kept only as a
  Go-quoted string in the fact detail
- re-ingesting a file upserts the same assets and skips identical facts

Worker-only, like ingest_bbot. nuclei, httpx and ffuf follow separately.
2026-11-13 16:22:37 +01:00
BaaRai 6f3a59e205 Coverage ledger: record examined, ruled-out and untested risk classes
Workers record per (surface, risk_area) outcomes with the new record_coverage
tool (reported, no_issue_found, ruled_out, not_applicable, needs_follow_up).
Gaps are needs_follow_up entries, entries missing their required evidence or
reason, and goal vulnclass values with no entry (unrecorded_risk_class), so an
absent finding never reads as a clean target. Completeness marks timeout,
budget wrap-up, pause, abort and running tasks as incomplete with a caveat.

Surfaced in the planner context (one line per round, full gap list on the
timeout/budget wrap-up round), GET /api/tasks/{id}/coverage, the coverage card
and the Markdown report. Follows coverage_enabled; no new env var.
2026-11-12 20:16:23 +01:00
BaaRai 0f5d93fbee Add ingest_bbot tool (bbot JSONL, typed scope split)
ingest_bbot reads a bbot output file by path and ingests it through
the typed parser from #199 (parseBbotJSONL), one line at a time:

- Lines are read with a fixed cap (bbotMaxLineBytes = 64 KiB, newline
  included). A longer line is skipped and counted, never buffered whole.
- scope_distance == 0 events that are assets (assetItems) go through
  the existing insert_assets handler; no asset SQL is written here.
- Every other event is recorded as a fact with source
  third_party_intel. The summary is built from typed fields only; the
  free text is stored in detail as the inert Go-quoted string.
- Intel never reaches insert_assets or task_scope.

The tool is in WorkerTools, so the existing BuiltinToolSeeds/SeedTool
loop inserts its row (a new key, inserted on fresh and existing DBs).
It is listed in the target gate's noTargetTools: its input is a local
path, not a network target (TestDomainToolsDeclareNoTarget).
2026-11-12 12:35:06 +01:00
BaaRai ca15063db7 fix(intercept): match block rules on the declared targets before any lookup
The target gate resolved each name through the cache before it matched the
asset block rules, so a call to a name that a name rule blocks still cost a
DNS lookup, and that query reached the name servers of the blocked domain.
URL rules (exact_url, fuzzy_url) were not evaluated at dispatch at all: the
gate passed no URL to the matcher, so they only applied to asset and intent
writes.

targetCheck.run now decides in this order, for a gated call:

1. URL rules, once per decision, on the URLs the call declares, as written.
2. Every block rule on the declared form of every target: the name as
   written, the IP literal, the range.
3. Only then resolution (names a derived exemption covers are still never
   resolved), and the block rules again on the resolved addresses.
4. Exemptions, pins and requests, unchanged.

A hit in 1 or 2 blocks with TARGET_RULE_BLOCKED and the same decided row as
any block, costs no lookup for any target of the call, and opens no request.

- intercept.extractDeclared returns the targets (ExtractTargets is now a thin
  wrapper, unchanged for its callers) and the URLs that name a target: the
  WebFetch url, and every URL token the command-string rules read in Bash,
  shell_open, Monitor, shell_send and the string leaves of other tools,
  including ExecuteExtraTool's inner call.
- The custom http tool's rendered URL and the URLs of a rendered command go to
  the same URL rules (gateRendered takes the URLs).
- Known limit, stated in the comments next to the matcher (db.matchOne) and
  at the dispatch match: URL rules match the declared string. exact_url misses
  any other spelling (case, percent-encoding, trailing slash, added query,
  /./), fuzzy_url misses an encoded substring, and a command can build its URL
  at run time. For a host, a domain, IP or CIDR rule is the one to write.

Unchanged: a tool-rule or judge deny still blocks before the gate is reached
(no extraction, no lookup, no request); asset block rules still win over an
allow and an ask; the gate off path does nothing; scripts stay advisory.

Tests:
- intercept: TestBlockRuleOnDeclaredNameNeedsNoLookup (exact_domain and
  fuzzy_domain; zero lookups, also for an unblocked name earlier in the same
  call), TestExtractDeclaredURLs, TestURLBlockRuleAtDispatch (WebFetch, Bash,
  MCP, ExecuteExtraTool, a pinned and an exempt host, a disabled rule, the
  rendered URL and command; zero lookups), TestExactURLRuleMatchesOnlyAsWritten
  (documents the limit).
- guard: TestBlockRuleWinsOverAllowAndAsk (name rule and URL rule, each over a
  rule allow, judge allow, rule ask and judge ask, real interceptor: blocked,
  no allow row, nobody asked, zero lookups). TestDenyRuleWinsBeforeTargetRequest
  now also covers a judge deny and asserts zero lookups with the real gate.
- server: TestBypass_CustomScriptToolTargetNotGated documents that a script
  tool's target, written in its stored code, is not visible to the gate: in a
  gated task the call is allowed and reaches the target with no lookup, no
  request and no row.
2026-11-11 15:47:58 +01:00
BaaRai 78f92f44fc feat(intercept): target extraction, resolver, exemptions and the gate in the hook (S3b 1.5-1.7)
The approval gate for asset targets now decides. It is opt-in per task and
per conversation (target_gate, default off). With the flag off nothing is
extracted, resolved, read or written, and every branch of the hook behaves as
before. No UI.

Extraction and normalisation (1.5)
- intercept.NormalizeTarget: lower case, trailing dot, userinfo, IPv6
  brackets, IPv4-mapped IPv6, decimal / hex / octal / short IPv4, percent-
  encoded host, punycode; port explicit, else the scheme default, else 0.
- intercept.ExtractTargets: WebFetch url exactly; Bash, shell_open, Monitor
  (command) and shell_send (text) with the command-string rules (URL with a
  scheme, IP, IP:port, host:port, CIDR; a bare name is not a target); every
  string of the input for MCP tools, custom tools and any tool the gate does
  not know. ExecuteExtraTool is unwrapped to the inner tool. The platform's
  domain tools, the file tools and web_search have no target.
- The fields are decoded as the tools decode them (key case, last duplicate),
  and a URL authority that reads differently quoted and unquoted yields both
  readings.

Resolver, exemptions, gate decision (1.6)
- Resolver interface, system implementation (A and AAAA), 3 s per lookup,
  positive answers cached 10 s per host with collapsed lookups, failures and
  empty answers never cached, always fresh when a request is raised.
- Exemptions are exact host + port. Derived from the configuration (LLM
  gateways, search backend in use, platform proxies): computed on every call,
  never resolved, never voided. Stored (settings key
  target_gate_exempt_hosts, seeded once with the package sources): resolved,
  and void when any address is private, loopback, link-local, unspecified,
  unique-local, multicast or a metadata address. An entry in both origins
  counts as derived. A stored IP literal that can never be exempt does not
  apply.
- GET / PUT /api/target-gate/exemptions and GET .../history. A change is
  validated (ValidateExemptionHost on the normalised host) and appended to
  target_gate_exemption_log in the transaction that stores it.
- Interceptor.CheckTargets: asset block rules (global and of the task) on the
  name and its addresses, exemptions, resolution, pin lookup. A (host, 0) pin
  covers every port; a (host, 443) pin covers 443 only and never a target
  without a declared port; two pins are never merged. At most 5 unpinned
  targets per call. Codes: TARGET_RULE_BLOCKED, TARGET_RESOLVE_FAILED,
  TARGET_TOO_MANY, TARGET_GATE_UNAVAILABLE; none opens a request.
- With the upstream egress proxy set every pin is name-only at call time,
  whatever name_only_effective says in its row.
- An approval never pins zero addresses: the pin writer refuses a request
  that displays none (db.ErrTargetPinEmpty, 422 on the decide endpoint, the
  row stays pending) and RequestTargets does not open one.
- Every block, and the first use per process of each exemption and of each
  pin version, is a decided row (decided_by system, decision_source target).

The gate in the hook (1.7)
- guard.applyIntercept: a deny by a rule or by the judge blocks first, before
  any target request exists; then CheckTargets; an allow is logged as before
  and does not satisfy the gate; an ask and an unpinned target share one
  request, and the classic ask is raised only when it was not merged. The
  call is checked again after the decisions, at most three passes.
- The gate applies to every tool of a gated session, enabled for rules or
  not, and to the target-only guard of the planner and the main agent, which
  still records blocks only.
- Target blocks have their own model-facing text, built from the code and the
  reason of the result.

Not in this change: rendered targets of custom tools (1.8) and redirect
targets (1.9). Subprocess paths stay advisory; the TestBypass_* tests document
what the gate does not see.
2026-11-11 00:34:55 +01:00