15 Commits
Author SHA1 Message Date
BaaRai d8f47d4359 Traffic ingest: capture operator browsing via a browser extension
Closes #33. The MITM proxy only sees the agent's own actions, so
anything behind a human gate (signup captcha, email OTP, 2FA, SSO)
left the authenticated surface dark. The operator's browser becomes a
capture source instead:

- traffic: exported RecordExternal storing an externally captured
  exchange through the same spill/blob/FTS path as proxy traffic
  (insert logic shared with record()); rejects unparsable URLs and
  empty methods
- server: POST /api/traffic/ingest inside the authenticated API group,
  batch capped at 50 entries, bodies truncated at 256 KiB, per-entry
  ingested/failed counts
- extension/: Manifest V3 Chrome/Edge extension driving the
  chrome.debugger Network domain on the chosen tab, batching text
  exchanges to the ingest endpoint with the operator's URL + JWT in
  chrome.storage.local; binary bodies skipped; flush on batch/stop
- tests for both layers, README section
2026-10-11 14:53:08 +02:00
BaaRai e233c02314 Translate all Chinese text to English
Comments, doc comments, UI labels, log and error messages, LLM prompt
strings, markdown docs, test data and assertions. Identifiers, enum
keys, routes, placeholders and code structure are unchanged.

Notes:
- docs/漏洞流量证据.md renamed to docs/vulnerability-traffic-evidence.md
- tests that exercised CJK specific handling (GBK zip names, Chinese
  full text search) now use ASCII equivalents; go build passes and the
  test suite matches the pre translation baseline (remaining failures
  need a PostgreSQL DSN and fail identically on the original)
- frontend mention syntax now uses English kinds (finding, asset, ...)
  matching the backend, chat mentions tests pass
2026-10-08 23:14:34 +02:00
Autumn-27 eb347d5965 feat(traffic): 流量列表新增「清空全部」,并借空库时机压实索引
一次删除全部流量记录,忽略当前筛选条件。与按 host 删除的两条路径相比多做
两件事:

- 清理索引已不再记录的历史 host 目录。按 host 删除只能从索引反查目录,
  行已消失的孤立目录会一直留着;「清空全部」既然语义是不留残余,就直接
  扫顶层目录(跳过 _index/_blobs/_ca/_delete_staging)。
- 收尾做一次全量压实(optimize + VACUUM + wal_checkpoint(TRUNCATE)),把
  索引占用的磁盘空间还给系统,并把实际释放量返回给前端提示。

VACUUM 的耗时与临时空间只和它保留的内容成正比,所以空库是做全量压实唯一
免费的时机——也正因如此,这一步顺带成为已有实例的转换途径:auto_vacuum
只能通过 VACUUM 生效,清空一次之后旧库即转为增量回收模式,此后日常按
host 删除就能自行回收空间(TestDeleteAllConvertsLegacyIndex 覆盖)。

已绑定到漏洞的流量证据保存在独立的证据库中,不受影响——绑定时已拷出独立
副本,正是为了让热流量可以随时清空而不带走证据。确认弹窗里明确写了这点。

前端把「清空全部」做成 outline 样式而非第二个 destructive 按钮:它忽略一切
筛选条件,不该看起来和「删除该目标」只差一次误点;无流量时禁用。

测试:新增 TestDeleteAllPurgesAndCompacts(清空后索引、正文、blob 引用与
孤立目录全清,空间回收到 1/8 以下,清空后仍可继续录制)与
TestDeleteAllConvertsLegacyIndex(旧库经清空后转为增量回收,随后普通删除
即可回收)。
2026-09-30 02:50:49 -04:00
Autumn-27 e50f5a3083 fix(traffic): 删除流量后把索引空间真正还给操作系统
删除流量只让数据不可见,磁盘占用从不下降。两处原因叠加:

1. SQLite 删行仅把页挂到 freelist,而索引库建库时未启用 auto_vacuum,
   文件永不收缩。256KB 以下的正文全部内联在这个库里,所以体积主体就在
   这里,且只增不减。
2. ex_fts 是 contentless_delete 全文索引,DELETE 只写 tombstone、不回收
   原 postings,从来没有合并过——删流量反而让索引变大(实测 exchanges
   清零后 ex_fts_data 仍占 3.3MB / 840 行)。

trigram 索引约为正文体积的 2 倍,加上内联正文,抓包量大的实例会为早已
删掉的流量长期占用数倍磁盘:实测抓 6MB 文本正文 → 索引 16.2MB,删光后
仍是 16.2MB。

修复:
- 新建索引库在建表前启用 auto_vacuum=incremental。该 pragma 只在库中尚
  无表时可设,且只有紧随的 VACUUM 才把它写入文件头,两步都必须落在同一
  条连接上,故新增 initIndex 固定一条 *sql.Conn 串起 pragma 与建表。
- 新增 reclaim/reclaimChunk:删除提交后在后台分块执行全文索引增量合并、
  incremental_vacuum 与 wal_checkpoint(TRUNCATE)(不做 TRUNCATE 则空间
  留在 WAL 里、文件大小不变,PASSIVE 也清不掉)。接入 DeleteHost、
  HostDeleteStage.Commit 与 RecoverHostDeleteStages 三条路径。
- 回收分块进行并在块间释放写锁:record() 由 go-mitmproxy 在把响应写回
  客户端之前同步调用,一次性回收数 GB 会连带卡住被测请求。Close 先发停止
  信号,剩余预算留给下次删除,不拖住进程退出。
- 终止条件按「是否有进展」判定:incremental_vacuum 只能释放挪得到文件
  末尾的页,残留少量空闲页是正常结果;另加步数上限兜底。fts5 无法查询
  索引是否已合并完(其特殊 INSERT 自身即计一次 changes),故改为固定合并
  预算,未做完的留给后续删除。
- busy_timeout 改由 DSN 提供:它是每连接设置,原先的 db.Exec 只配到池中
  碰巧被选中的那一条,其余连接在有写者时会立即失败而非等待。驱动按第一个
  '?' 切分 DSN,故数据目录路径含 '?' 时退回裸路径 DSN。

已有实例的索引库仍是旧模式(auto_vacuum 无法事后开启),incremental_vacuum
在其上为空操作,启动时打印提示;此类库的 tombstone 增长已止住,已占体积
待后续「存储压缩」入口一次性回收。

测试:新增 reclaim_test.go(空间回收回归、tombstone 合并、新库模式、旧库
不卡死)与 upgrade_test.go(旧装机升级后同库同数据、全文与正文可用、
legacy path 行不受影响;以及降级回旧二进制仍可读写)。
2026-09-30 01:44:44 -04:00
Autumn-27 6d8c50a12e feat(traffic): 流量列表支持内容检索、路径/状态码/长度筛选与列排序 (#177)
- 后端 traffic.Page 改为 PageQuery:新增响应内容(FTS)、路径(url_template)、
  状态码(精确/Nxx 分桶)、响应长度范围筛选,并支持按 ts/status/resp_len 排序
  (尾随 id DESC 保证分页稳定);补 status/resp_len 两个索引(IF NOT EXISTS,旧库自动生效)
- server getTraffic 解析 body/path/status/resp_min/resp_max/sort/order 参数
- 前端流量页新增高级筛选行(响应内容/路径/状态码/长度范围)与可排序表头,
  排序偏好持久化到 localStorage;抽出公共组件 ui/sortable-head
- 仅改动前端页走的 Page,不动 Agent 工具的 query 契约
2026-09-29 21:58:17 -04:00
dingpotian f716739fcb fix(traffic): 流量录制代理默认只监听 127.0.0.1,避免默认开放代理暴露 (#129) (#130) 2026-09-16 09:36:32 +08:00
RuoJi6 fcf33bfe70 fix(traffic): match recorded hosts with port-aware search (#114) 2026-09-15 00:43:20 +08:00
ji ruo e6f8195500 feat: 支持漏洞关联多条流量证据 2026-09-10 13:03:20 +08:00
Autumn-27 87375979af feat(proxy): 新增全局出口代理配置
所有目标流量可经统一的全局代理出网(http/https/socks5,支持 user:pass):
- 开启流量捕获时,作为 MITM 记录代理的上游(流量照录再经代理出网);
  拦截与透传两条路径都走上游,不泄露源 IP。
- 关闭捕获时,直接注入 agent 的 bash 环境与 WebFetch;proxyEnv 增加
  ALL_PROXY 以支持 socks5。

配置存于 settings KV 表(无需迁移),前端系统配置页新增「全局代理」卡片。
与网络搜索代理、LLM 代理相互独立。
2026-09-07 10:43:38 -04:00
Autumn-27 9e8b9e2247 fix: restore changes dropped by the upstream sync merge
The manual v0.3.6 sync (3675f7e) was incomplete, and the follow-up merge
(fa7b3cd) resolved conflicts in favour of that copy, silently reverting three
shipped changes. main is already an ancestor of this branch, so re-merging
cannot bring them back; restore them explicitly.

- schema.sql: re-add the idempotent llm_profiles_format_check migration.
  CREATE TABLE IF NOT EXISTS leaves an existing table alone, so databases
  created before openai-responses kept the old two-value CHECK and rejected
  every openai-responses profile.
- orchestration.go: re-add the reseedPlannerPrompt/reseedWorkerPrompt calls,
  dropped by a comment-alignment hunk while the functions stayed behind as
  dead code. Without them existing installs never receive the quantified
  acceptance check or the negative-conclusion evidence threshold.
- CHANGELOG.md: restore the 0.3.6 entries, the @neouks credit and the correct
  norma version range.

Also close three recovery gaps in the new cold-archive path:

- traffic: write the staging journal even when no legacy host directory
  exists. Modern installs keep traffic in SQLite only, so a crash between the
  PostgreSQL commit and the SQLite commit left the archived task's exclusive
  traffic in the hot store with nothing for recovery to act on.
- recoverTaskArchiveStages: treat a missing journal.json as a discardable
  directory instead of an error, which used to set recoveryFailed and stop the
  archive worker from starting for the rest of the process lifetime.
- stageTaskArchiveFiles: replay the journal's renames when resuming an
  existing staging root. The journal is written before the first rename, so
  its presence did not guarantee a complete payload, and a resumed round could
  package an archive missing transcripts or workspace files.
2026-08-29 01:37:46 -04:00
pphua 809e94f6e4 归档 2026-08-28 12:56:29 +08:00
pphua 0f465825da feat(tasks): add runtime controls and inherited context 2026-08-24 19:29:00 +08:00
pphua 259a0ae965 同步 2026-08-24 14:14:15 +08:00
FileShare Developer 6e623227a6 feat: delete traffic targets and LLM task conversations from UI
Add delete controls to the Traffic and LLM Records pages so users can
purge recorded data without hand-editing the data volume.

Traffic page:
- Host target picker (Popover with checkboxes + select-all toggle)
- Exact-match batch delete for multiple hosts (DELETE /api/traffic/hosts)
- Substring delete for the host-filter input (DELETE /api/traffic?host=)
- Orphaned content-addressed blob garbage collection after deletes
- Distinct-hosts endpoint (GET /api/traffic/hosts) for the picker

LLM Records page:
- Task picker dropdown (distinct task_id + record counts)
- Per-task delete (DELETE /api/llm/records?task=)

Other:
- Add /dist/ to .gitignore (build artifact)
- Extend traffic_test.go with DeleteHost, DeleteHostsExact, Hosts tests
- Fix Popover render bug (replace ScrollArea that broke height calc)
- Remove fade/zoom animation on the host picker Popover
2026-08-10 11:10:58 +08:00
Autumn-27andClaude Opus 4.8 0ef8561411 chore: ARTEX 初始化
ARTEX — AI 自主渗透测试系统(Go 后端 + Next.js 前端,单二进制内嵌)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-26 04:27:45 -04:00