RecordExternal wrote method and header maps into req_head/resp_head
with no token or line-break check. Replay already refused those
values. An ingest JSON string with \r\n became extra evidence lines.
Apply the same validHeaderName and CR/LF/NUL checks before storage.
offset+length > size wraps when length is near MaxInt64, so the cap is
skipped and make([]byte, length) can exhaust memory. Compare length
against size-offset instead.
The sink runs the request phase in the Request hook and the response phase
at the start of the Response hook; at SSEStart only status and header
actions apply. WebSocketStart comes after the client is upgraded, so the
handshake is traced "not rewritten: stream" and recorded as sent.
Exchanges keep what the client sent and what the target answered. The
original of a rewritten request lives in a per-flow snapshot slot (entry or
tombstone, 1024 entries / 64 MiB, 4096 tombstones), reserved before the
request is changed, released on Response, RequestError, SSEStart, a 5 MiB
response, or by a sweep of finished flows (ClientDisconnected, 1 s
debounce, 1 min period). Under pressure an entry older than 15 minutes is
turned into a tombstone in place, only when that frees room for the new
reservation; its exchange is recorded "original request unavailable".
record(), recordStreamStart() and insertExchange() clear any trace under
the id before INSERT OR REPLACE; the trace (exchange_rewrites, created
with CREATE TABLE IF NOT EXISTS) is written in the exchange's transaction,
its spilled values registered in blob_refs, and purged with the exchange.
The full-text index covers the originals only. Logs of a rewritten flow
name the host, the step or error class and the rule ids, never the URL or
the error text. A rewritten request's transport error never switches its
host to passthrough. traffic_get shows the rewritten flag and rule ids.
Without a rule that applies, what is recorded is unchanged byte for byte.
The request phase can fail after the engine is done: the rewritten request
does not reach the target. NoteError records that as the rule's last error,
under the new class transport_error.
GET, POST, PUT and DELETE /api/traffic/rewrite-rules and
POST /api/traffic/rewrite-rules/{id}/toggle manage the rules behind
requireAuth. A rule is validated on save (400), created disabled, and
validated again before the toggle enables it. The value of a sensitive action
is never returned; an update sent without it keeps the stored value when the
stored rule has a sensitive action of the same type and header name at the
same position.
The manager compiles the enabled rules after traffic.Open and after every
change, and publishes the snapshot with Traffic.SetRewriteRules, an atomic
swap. An enabled rule the validation refuses is left out and reported as
load_error.
The proxy stores the snapshot and does not apply it: a test sends the same
request with and without matching rules and compares what the target
receives, what the client receives and what is recorded.
traffic/rewrite holds the rule model of the capture proxy's rewrite rules,
their save-time validation, an immutable compiled snapshot and the
transformation of one request or response. It imports the standard library
only, so both server and traffic can use it.
Rules run by priority ascending then id ascending, as a pipeline; within a
rule the actions run in a fixed order. A rule is atomic and fails open: a
failing action or a panic cancels that rule only. A rule that needs a body it
cannot touch (binary, over 1 MiB, compressed, streamed, absent by protocol)
is skipped whole. The engine never writes to the URL, headers or bodies it is
given.
Apply returns the trace lines of the phase: the rule snapshot once per rule,
the body of the last body action only, and no value for a sensitive action or
for a field one wrote earlier in the phase. A sensitive value is fingerprinted
with HMAC-SHA256 truncated to 128 bits, or not at all without a key.
Nothing calls the engine from the proxy yet.
For plain http the client puts the Host in the absolute-form request target, so the proxy dials it; for https go-mitmproxy records the exchange under the inner Host. Comment and error message only.
The Host refusal now explains that go-mitmproxy rebuilds the upstream request from the URL and drops the Host override. ReplayResult gains Recorded (false for an https passthrough host) and the sorted response Headers, both exposed by the route. Headers named in Connection and Proxy-Authorization are dropped. A body over the limit returns ErrReplayTooLarge, mapped to 413.
Traffic.Replay sends an edited request (method, URL, headers, body) through the local proxy, which records the exchange like any other, and returns the response bounded to 2 MiB. POST /api/traffic/replay exposes it to operators only (no agent tool). Invalid input is rejected with ErrReplayInvalid; hop-by-hop headers are dropped, redirects are not followed, and a Host that differs from the URL is refused because the proxy routes by Host.
Stream payloads over 4 KiB now spill to the blob store (spillAt with maxStreamInline; spill keeps maxInlineBody for HTTP bodies) and StreamMessages clips every payload to 4 KiB, reading a spilled binary preview from its blob. The byte cap counts the inline preview. persistStream spills before opening the transaction, as record() does. Without the library lock nothing is read. ts is unix seconds. Adds the final-message-on-close and route tests.
The recording proxy now stores WebSocket and Server-Sent Events streams: one exchange per stream (101 websocket, or the real text/event-stream response) and its messages in stream_messages, with per-message and per-exchange limits and blob spill. Messages are readable through Traffic.StreamMessages, GET /api/traffic/stream and the traffic_messages agent tool; deletion and blob GC cover them.
With traffic capture on, the Bash proxy URL carries a marker userinfo. The recording proxy drops Proxy-Authorization from every request before it is forwarded or recorded; when the value is the marker it marks the client connection, and every later request on it, including those inside its CONNECT tunnel, waits for the per-host token and jitter of the current noise profile (egress.Wait) and is counted, with a bounded wait. WebFetch, the http custom tools and enrich keep the unmarked URL, which egress already paces, so nothing is paced twice. Off paces nothing. Best effort: only clients that honour the proxy are covered, with capture on.
Closes#33. The MITM proxy only sees the agent's own actions, so
anything behind a human gate (signup captcha, email OTP, 2FA, SSO)
left the authenticated surface dark. The operator's browser becomes a
capture source instead:
- traffic: exported RecordExternal storing an externally captured
exchange through the same spill/blob/FTS path as proxy traffic
(insert logic shared with record()); rejects unparsable URLs and
empty methods
- server: POST /api/traffic/ingest inside the authenticated API group,
batch capped at 50 entries, bodies truncated at 256 KiB, per-entry
ingested/failed counts
- extension/: Manifest V3 Chrome/Edge extension driving the
chrome.debugger Network domain on the chosen tab, batching text
exchanges to the ingest endpoint with the operator's URL + JWT in
chrome.storage.local; binary bodies skipped; flush on batch/stop
- tests for both layers, README section
Comments, doc comments, UI labels, log and error messages, LLM prompt
strings, markdown docs, test data and assertions. Identifiers, enum
keys, routes, placeholders and code structure are unchanged.
Notes:
- docs/漏洞流量证据.md renamed to docs/vulnerability-traffic-evidence.md
- tests that exercised CJK specific handling (GBK zip names, Chinese
full text search) now use ASCII equivalents; go build passes and the
test suite matches the pre translation baseline (remaining failures
need a PostgreSQL DSN and fail identically on the original)
- frontend mention syntax now uses English kinds (finding, asset, ...)
matching the backend, chat mentions tests pass
The manual v0.3.6 sync (3675f7e) was incomplete, and the follow-up merge
(fa7b3cd) resolved conflicts in favour of that copy, silently reverting three
shipped changes. main is already an ancestor of this branch, so re-merging
cannot bring them back; restore them explicitly.
- schema.sql: re-add the idempotent llm_profiles_format_check migration.
CREATE TABLE IF NOT EXISTS leaves an existing table alone, so databases
created before openai-responses kept the old two-value CHECK and rejected
every openai-responses profile.
- orchestration.go: re-add the reseedPlannerPrompt/reseedWorkerPrompt calls,
dropped by a comment-alignment hunk while the functions stayed behind as
dead code. Without them existing installs never receive the quantified
acceptance check or the negative-conclusion evidence threshold.
- CHANGELOG.md: restore the 0.3.6 entries, the @neouks credit and the correct
norma version range.
Also close three recovery gaps in the new cold-archive path:
- traffic: write the staging journal even when no legacy host directory
exists. Modern installs keep traffic in SQLite only, so a crash between the
PostgreSQL commit and the SQLite commit left the archived task's exclusive
traffic in the hot store with nothing for recovery to act on.
- recoverTaskArchiveStages: treat a missing journal.json as a discardable
directory instead of an error, which used to set recoveryFailed and stop the
archive worker from starting for the rest of the process lifetime.
- stageTaskArchiveFiles: replay the journal's renames when resuming an
existing staging root. The journal is written before the first rename, so
its presence did not guarantee a complete payload, and a resumed round could
package an archive missing transcripts or workspace files.
Add delete controls to the Traffic and LLM Records pages so users can
purge recorded data without hand-editing the data volume.
Traffic page:
- Host target picker (Popover with checkboxes + select-all toggle)
- Exact-match batch delete for multiple hosts (DELETE /api/traffic/hosts)
- Substring delete for the host-filter input (DELETE /api/traffic?host=)
- Orphaned content-addressed blob garbage collection after deletes
- Distinct-hosts endpoint (GET /api/traffic/hosts) for the picker
LLM Records page:
- Task picker dropdown (distinct task_id + record counts)
- Per-task delete (DELETE /api/llm/records?task=)
Other:
- Add /dist/ to .gitignore (build artifact)
- Extend traffic_test.go with DeleteHost, DeleteHostsExact, Hosts tests
- Fix Popover render bug (replace ScrollArea that broke height calc)
- Remove fade/zoom animation on the host picker Popover