Commit Graph
3 Commits
Author SHA1 Message Date
BaaRai 9d75df335f fix(agent): strip URL userinfo in scan and bbot ingestion
scanURL kept the userinfo of httpx, ffuf and nuclei URLs, so a line such
as http://user:pass@host/path stored the credentials in assets.url, in
the notable items and in the facts. scanURL now drops u.User before
re-serializing; scheme, host, port, path and query are kept.

The nuclei fact detail also embedded the raw host and matched-at values,
the httpx detail the raw host and the ffuf detail the raw redirect
location. They now go through stripUserinfo, which removes only the
userinfo of a URL or bare host:port and keeps every other byte, so a
value without userinfo is unchanged and fact deduplication is the same.

bbotTyped drops the userinfo of the typed url of an in-scope URL event
before it becomes an asset.

Rows already stored with userinfo are not rewritten here.
2026-11-14 01:02:16 +01:00
BaaRai 07fe075c12 agent: ingest nuclei, httpx and ffuf output in ingest_scan_output
Extends ingest_scan_output with three formats:

- nuclei-jsonl: one tool_output fact per result and an asset for the
  scanned target when it is valid; no automatic finding
- httpx-jsonl: one HTTP service asset and one fact per URL
- ffuf-json: one URL asset and one fact per result, read token by token
  with encoding/json; a file cut short keeps the complete results

JSONL lines go through readJSONLines, taken from the bbot ingester
(64 KiB per line; longer lines and invalid JSON are skipped and
counted). bbot now uses the same reader, with no behaviour change.
One file yields at most 50000 assets and facts; records past the cap
are counted in over_cap and the result is marked truncated.
2026-11-13 19:15:59 +01:00
BaaRai 374c985d06 agent: add ingest_scan_output for nmap XML files
A worker can write nmap output with -oX and call ingest_scan_output with
the format and path. The file is parsed deterministically: assets go
through the insert_assets handler, each host becomes one tool_output
fact listing its open ports, and the model gets only counters and at
most 20 notable items per type.

- path is resolved under the run working directory; "..", an absolute
  path outside it and a symlink leading outside are refused; 256 MiB cap
- the XML is streamed host by host with encoding/xml; a file cut short
  keeps the complete hosts and returns truncated: true
- target text (banners, products, hostnames) is kept only as a
  Go-quoted string in the fact detail
- re-ingesting a file upserts the same assets and skips identical facts

Worker-only, like ingest_bbot. nuclei, httpx and ffuf follow separately.
2026-11-13 16:22:37 +01:00