Workers record per (surface, risk_area) outcomes with the new record_coverage
tool (reported, no_issue_found, ruled_out, not_applicable, needs_follow_up).
Gaps are needs_follow_up entries, entries missing their required evidence or
reason, and goal vulnclass values with no entry (unrecorded_risk_class), so an
absent finding never reads as a clean target. Completeness marks timeout,
budget wrap-up, pause, abort and running tasks as incomplete with a caveat.
Surfaced in the planner context (one line per round, full gap list on the
timeout/budget wrap-up round), GET /api/tasks/{id}/coverage, the coverage card
and the Markdown report. Follows coverage_enabled; no new env var.
ingest_bbot reads a bbot output file by path and ingests it through
the typed parser from #199 (parseBbotJSONL), one line at a time:
- Lines are read with a fixed cap (bbotMaxLineBytes = 64 KiB, newline
included). A longer line is skipped and counted, never buffered whole.
- scope_distance == 0 events that are assets (assetItems) go through
the existing insert_assets handler; no asset SQL is written here.
- Every other event is recorded as a fact with source
third_party_intel. The summary is built from typed fields only; the
free text is stored in detail as the inert Go-quoted string.
- Intel never reaches insert_assets or task_scope.
The tool is in WorkerTools, so the existing BuiltinToolSeeds/SeedTool
loop inserts its row (a new key, inserted on fresh and existing DBs).
It is listed in the target gate's noTargetTools: its input is a local
path, not a network target (TestDomainToolsDeclareNoTarget).
doHTTP now connects through intercept.PinnedDialer under the target scope of the request's context, so the probe uses the same pinned-address dial as the target gate. ProbeSite takes the context of the call that asks for the probe. A probe without a scope is refused; with the gate on, the probe bypasses the recording proxy so the dialer decides on the target itself. The path stays dormant: nothing calls ProbeSite on main.
Audit findings F5, F6 and F11 for issue #124.
F5, goal guards:
- goal_met refuses while any goal node of the task is open, and when the
goals cannot be read. A task without goal nodes keeps the old behavior.
- prove_goal accepts a fact as evidence only when its confidence is
"observed"; otherwise it refuses and the goal stays open. Findings are
still accepted.
F6, hint origin:
- add_hint stores the calling ToolSet's worker as the hint origin instead
of "human", and passes it to the planner trigger (TriggerEvent.HintOrigin).
- renderTriggers names the writer: the human via the main agent, a
cross-task orchestration, or an agent, with a neutral fallback.
F11, built-in rules for the planner and the main agent:
- Interceptor.MatchBuiltin evaluates the enabled "[built-in]" deny rules on
the intercept-enabled tools.
- The target-only guard applies them before the target gate, whether the
gate is on or off. User rules and the judge stay out of it.
- SetRulesForTest is a test seam for the store-less guard tests; Log is a
no-op without a store.
In a gated scope, the two HTTP clients the platform owns connect only to
addresses that a pin or a stored exemption covers at the moment of the
connection, on every hop.
- intercept.PinnedDialer(tool, scoped): the DialContext of both clients.
It reads the target scope from the context net/http passes to the dial.
Gate off, or no scope in a client built without one: plain dial, as
today. A client built for a scope whose scope is missing at the dial
fails closed with TARGET_GATE_UNAVAILABLE.
- dialTarget runs the gate decision on the dialled host:port with a fresh
lookup (block rules first, exemptions, pins, port precedence) and
connects only to the addresses it compared, never to the name. Derived
exemptions and name-only pins dial the address as received. An address
no pin covers finds or opens the request (new_ip / no_pin, wait 0) and
fails at once with TARGET_APPROVAL_PENDING. Redirect hops are dialled
against the hop's own (host, port).
- WebFetch (withWebFetch, now given the run context): DialContext set on
every agent kind; DisableDirectRetry when the run's scope is gated.
- Custom http tool: a direct call in a gated scope gets a DefaultTransport
clone with the pinned dialer, no environment proxy and keep-alive off.
Proxied calls are unchanged (the recording proxy is step 2.2).
- Name-only: request reason says the platform does not bind the
connection; the first-use pin_match row records the effective state.
- Derived exemptions: a proxy URL without a port is listed with the port
net/http dials (1080 for socks5) (OPEN-9).
- go.mod: Norma v0.4.8 (DialContext, DisableDirectRetry).
WebFetch and the custom http tool followed redirects with net/http's
default policy, so a pinned host could send the client to a target nobody
approved. RedirectCheck is now the CheckRedirect policy of both clients.
- Gate off, or no scope (operator test endpoint): only the 10-hop limit,
same error text; nothing is parsed, resolved, read or written.
- Gate on: each hop runs through the call's decision code (URL rules on
the hop URL, block rules before any lookup, exemptions, resolution,
pins). A hop no pin covers is not followed and never waits: its request
row is found or created and the call fails with TARGET_APPROVAL_PENDING.
http -> https asks again for port 443 unless a (host, 0) pin exists. An
unreadable hop is blocked with TARGET_GATE_UNAVAILABLE.
- Rows raised by a hop record redirect_from in the audit targets.
- Norma v0.4.7 has WebFetchConfig.CheckRedirect but no agentcore option,
so the four agent kinds build WebFetch themselves (same proxy, CA and
position in the tool list) instead of EnableWebFetch. Norma carries the
policy to the direct retry client.
The target gate resolved each name through the cache before it matched the
asset block rules, so a call to a name that a name rule blocks still cost a
DNS lookup, and that query reached the name servers of the blocked domain.
URL rules (exact_url, fuzzy_url) were not evaluated at dispatch at all: the
gate passed no URL to the matcher, so they only applied to asset and intent
writes.
targetCheck.run now decides in this order, for a gated call:
1. URL rules, once per decision, on the URLs the call declares, as written.
2. Every block rule on the declared form of every target: the name as
written, the IP literal, the range.
3. Only then resolution (names a derived exemption covers are still never
resolved), and the block rules again on the resolved addresses.
4. Exemptions, pins and requests, unchanged.
A hit in 1 or 2 blocks with TARGET_RULE_BLOCKED and the same decided row as
any block, costs no lookup for any target of the call, and opens no request.
- intercept.extractDeclared returns the targets (ExtractTargets is now a thin
wrapper, unchanged for its callers) and the URLs that name a target: the
WebFetch url, and every URL token the command-string rules read in Bash,
shell_open, Monitor, shell_send and the string leaves of other tools,
including ExecuteExtraTool's inner call.
- The custom http tool's rendered URL and the URLs of a rendered command go to
the same URL rules (gateRendered takes the URLs).
- Known limit, stated in the comments next to the matcher (db.matchOne) and
at the dispatch match: URL rules match the declared string. exact_url misses
any other spelling (case, percent-encoding, trailing slash, added query,
/./), fuzzy_url misses an encoded substring, and a command can build its URL
at run time. For a host, a domain, IP or CIDR rule is the one to write.
Unchanged: a tool-rule or judge deny still blocks before the gate is reached
(no extraction, no lookup, no request); asset block rules still win over an
allow and an ask; the gate off path does nothing; scripts stay advisory.
Tests:
- intercept: TestBlockRuleOnDeclaredNameNeedsNoLookup (exact_domain and
fuzzy_domain; zero lookups, also for an unblocked name earlier in the same
call), TestExtractDeclaredURLs, TestURLBlockRuleAtDispatch (WebFetch, Bash,
MCP, ExecuteExtraTool, a pinned and an exempt host, a disabled rule, the
rendered URL and command; zero lookups), TestExactURLRuleMatchesOnlyAsWritten
(documents the limit).
- guard: TestBlockRuleWinsOverAllowAndAsk (name rule and URL rule, each over a
rule allow, judge allow, rule ask and judge ask, real interceptor: blocked,
no allow row, nobody asked, zero lookups). TestDenyRuleWinsBeforeTargetRequest
now also covers a judge deny and asserts zero lookups with the real gate.
- server: TestBypass_CustomScriptToolTargetNotGated documents that a script
tool's target, written in its stored code, is not visible to the gate: in a
gated task the call is allowed and reaches the target with no lookup, no
request and no row.
The hook reads the parameters of a custom tool call, not the URL or the
command they render, and not a target written in the template itself. The
custom command and http tools now run the target gate inside the tool, on what
they are about to run, before anything is sent or started. Gate off (the
default) is unchanged: with no scope or the flag off nothing is parsed,
resolved, read or written.
- intercept.GateRenderedURL: the target is read as net/http dials it (host of
the parsed URL, explicit port, else the scheme default), then goes through
the same decision as at the hook (asset block rules, exemptions, resolution,
pin lookup) and the same request-and-wait, at most three passes. A URL whose
target cannot be read is blocked with TARGET_GATE_UNAVAILABLE and logged as
a decided row; no request, no lookup.
- intercept.GateRenderedCommand: the rendered command is read with the
command-string rules of Bash, and so is the rendering without the shell
quoting of the parameters (a quoted parameter inside a host name hides the
name from the rules). A command that names no target passes, as in Bash.
- A gated scope without the interceptor that owns the gate blocks
(TARGET_GATE_UNAVAILABLE), it never lets the call through.
- runCommandTool and runHTTPTool take the tool key, which names the request
rows. The gate runs before the tool's own timeout starts.
- guard.TargetBlockMessage (was targetBlockMessage) is exported so the tools
return the same model-facing text as the hook.
- Unchanged: the script tool (its target is in stored code; advisory, D9), the
custom-tool test endpoint (no scope: the operator's own action, D14), the
hook order (a rule or judge deny blocks first; the tool then never runs, so
its rendered target is never read), the evaluation of asset rules.
Tests: TestRendered* in intercept (no database); in server, against
PostgreSQL: TestCustomHTTPRenderedTargetGated,
TestCustomCommandRenderedTargetGated, TestCustomToolTestEndpointUngated,
TestTargetGateEndToEnd (a scripted worker's WebFetch waits, a click pins, the
next call passes, revoke, it asks again), TestDenyRuleWinsBeforeRenderedTarget
and TestGateOffTaskIsUnchanged.
Not in this change: redirect targets of WebFetch and of the custom http tool
(1.9).
The approval gate for asset targets now decides. It is opt-in per task and
per conversation (target_gate, default off). With the flag off nothing is
extracted, resolved, read or written, and every branch of the hook behaves as
before. No UI.
Extraction and normalisation (1.5)
- intercept.NormalizeTarget: lower case, trailing dot, userinfo, IPv6
brackets, IPv4-mapped IPv6, decimal / hex / octal / short IPv4, percent-
encoded host, punycode; port explicit, else the scheme default, else 0.
- intercept.ExtractTargets: WebFetch url exactly; Bash, shell_open, Monitor
(command) and shell_send (text) with the command-string rules (URL with a
scheme, IP, IP:port, host:port, CIDR; a bare name is not a target); every
string of the input for MCP tools, custom tools and any tool the gate does
not know. ExecuteExtraTool is unwrapped to the inner tool. The platform's
domain tools, the file tools and web_search have no target.
- The fields are decoded as the tools decode them (key case, last duplicate),
and a URL authority that reads differently quoted and unquoted yields both
readings.
Resolver, exemptions, gate decision (1.6)
- Resolver interface, system implementation (A and AAAA), 3 s per lookup,
positive answers cached 10 s per host with collapsed lookups, failures and
empty answers never cached, always fresh when a request is raised.
- Exemptions are exact host + port. Derived from the configuration (LLM
gateways, search backend in use, platform proxies): computed on every call,
never resolved, never voided. Stored (settings key
target_gate_exempt_hosts, seeded once with the package sources): resolved,
and void when any address is private, loopback, link-local, unspecified,
unique-local, multicast or a metadata address. An entry in both origins
counts as derived. A stored IP literal that can never be exempt does not
apply.
- GET / PUT /api/target-gate/exemptions and GET .../history. A change is
validated (ValidateExemptionHost on the normalised host) and appended to
target_gate_exemption_log in the transaction that stores it.
- Interceptor.CheckTargets: asset block rules (global and of the task) on the
name and its addresses, exemptions, resolution, pin lookup. A (host, 0) pin
covers every port; a (host, 443) pin covers 443 only and never a target
without a declared port; two pins are never merged. At most 5 unpinned
targets per call. Codes: TARGET_RULE_BLOCKED, TARGET_RESOLVE_FAILED,
TARGET_TOO_MANY, TARGET_GATE_UNAVAILABLE; none opens a request.
- With the upstream egress proxy set every pin is name-only at call time,
whatever name_only_effective says in its row.
- An approval never pins zero addresses: the pin writer refuses a request
that displays none (db.ErrTargetPinEmpty, 422 on the decide endpoint, the
row stays pending) and RequestTargets does not open one.
- Every block, and the first use per process of each exemption and of each
pin version, is a decided row (decided_by system, decision_source target).
The gate in the hook (1.7)
- guard.applyIntercept: a deny by a rule or by the judge blocks first, before
any target request exists; then CheckTargets; an allow is logged as before
and does not satisfy the gate; an ask and an unpinned target share one
request, and the classic ask is raised only when it was not merged. The
call is checked again after the decisions, at most three passes.
- The gate applies to every tool of a gated session, enabled for rules or
not, and to the target-only guard of the planner and the main agent, which
still records blocks only.
- Target blocks have their own model-facing text, built from the code and the
reason of the result.
Not in this change: rendered targets of custom tools (1.8) and redirect
targets (1.9). Subprocess paths stay advisory; the TestBypass_* tests document
what the gate does not see.
Storage, flag and request path of the approval gate for asset targets. The
gate itself is not wired into the tool hook yet (steps 1.5 to 1.7): with this
change nothing is extracted, resolved or blocked, whatever the flag says.
Schema and store (1.2)
- tasks.target_gate, conversations.target_gate (default false).
- intercept_pending.target_key and a partial unique index: one open request
per scope (task or conversation) and host:port.
- target_pins (no foreign keys: a pin outlives its task, its conversation and
its request row) and target_gate_exemption_log. All DDL is idempotent; no
CHECK constraints; the cascade on intercept_pending.conversation_id stays.
- db/target_pins.go: pin lookup with port precedence, list, revoke,
find-or-create of a request, and ResolveTargetRequest, which settles the
row, supersedes the old pin and writes the new version in one transaction.
- Deleting a task or an archive stub revokes its pins and settles its open
target requests in the delete transaction. DeleteConversation and
DeleteConversations revoke the pins of the removed conversations in the
same transaction as the delete.
Opt-in flag (1.3)
- Per task through the create and metadata-patch paths, per conversation
through create and patch; exposed in the task DTO. A retest conversation
takes the flag of the finding's task, never its pins. A spawned task starts
with the gate off.
- The task flag is atomic and read on every call through TargetScope.Enabled.
TargetScope now carries the Interceptor that owns the gate.
Request, wait, decide (1.4)
- pendingManager keeps a list of waiters per request; resolve wakes all.
- RequestTargets: find-or-create per target, one card per request, wait on
channel, poll, context and wait bound. A plain request is never settled by
a call that leaves. A request merged with an ask is bound to its creating
call and a timeout never allows it.
- Decide records the approver (subject of the session token) and the remote
address for every human decision, classic asks included. On a target
request "allowed" pins the displayed addresses; that is the only writer of
pins.
- A request raised while the global egress proxy is set says the approval is
name-only and its pin is stored with name_only_effective.
- Routes: GET/POST .../target-pins and .../target-pins/{pid}/revoke under
/api/tasks/{id} and /api/conversations/{id}; decision_source=target filter.
- ValidateExemptionHost: an exemption written as an IP literal may be private
unicast, never loopback, link-local or metadata, unspecified or multicast.
Step 1.1 of the S3b phase 1 plan. Wiring only: the approval gate for asset
targets is not enforced yet and no behaviour changes.
- Planner.Plan and MainAgent.Chat take a hooks parameter, like Worker.Execute.
Their sessions had no hooks, so nothing saw their tool calls before execution.
- guard.NewTargetOnly: the guard for those two agent kinds. It runs the
target-gate seam only: no intercept rule, no fallback judge, and no entry in
the audit ring (Guard.record is untouched and not called for these calls).
- Task.TargetGuard, built beside Task.Guard.
- intercept.WithTargetScope / TargetScopeFromContext: task id or conversation
id, an Enabled func and a wait bound, set at the five session entry points
(planner round, final planner round with wait bound 0, main-agent turn,
worker intent, conversation turn).
The seam allows every call and reads and writes nothing. No task or
conversation has an opt-in flag yet, so Enabled is nil (off) everywhere.
Add intercept_pending.decided_by (human | rule | model | system; empty while
pending or for rows decided before the column existed). Approval semantics are
unchanged: only the actor is recorded.
- rule-matched and model-judged rows written by CreateDecidedIntercept record
rule or model;
- ResolveIntercept takes the actor explicitly and rejects any other value
before touching the database;
- Decide (human allow/deny) records human; timeouts and work-cancelled
denials record system. Work-cancelled is a system refusal, not a human one.
- decided_by is exposed on the pending and approval records and filterable
with decided_by=human|rule|model|system (anything else is a 400).
- Legacy rows show decision_source as their actor when decided_by is empty.
The filter uses the stored column only, so decided_by=model cannot match
older human decisions or timeouts.
- Archive restore fills decided_by for archives that predate the column, and
marks a restored pending row as system.
Since e233c02 the judge writer prefixes its reason with "[model]" while
interceptSource, the approvalDecisionSource SQL fallback and the archive
restore checked "[Model]", so model-decided rows were inserted with
decision_source='unknown' and missed by the decision_source=model filter.
- db: add ModelReasonPrefix ("[model]") and isModelReason, which also
accepts the pre-e233c02 "[模型]" prefix; readers compare
case-insensitively (lower(reason) LIKE in SQL).
- intercept: the writer builds its reason through JudgeReason, using the
shared constant.
- schema.sql: idempotent backfill of 'unknown' rows without a rule whose
reason carries either prefix.
- tests: un-skip TestInterceptFilterHTTP (it failed only because of
this), make the db filter fixture use the constant, and add a
Postgres-free test pinning the writer output to interceptSource.
Fixes#156
Comments, doc comments, UI labels, log and error messages, LLM prompt
strings, markdown docs, test data and assertions. Identifiers, enum
keys, routes, placeholders and code structure are unchanged.
Notes:
- docs/漏洞流量证据.md renamed to docs/vulnerability-traffic-evidence.md
- tests that exercised CJK specific handling (GBK zip names, Chinese
full text search) now use ASCII equivalents; go build passes and the
test suite matches the pre translation baseline (remaining failures
need a PostgreSQL DSN and fail identically on the original)
- frontend mention syntax now uses English kinds (finding, asset, ...)
matching the backend, chat mentions tests pass
BaaRai