Commit Graph
2 Commits
Author SHA1 Message Date
BaaRai 07fe075c12 agent: ingest nuclei, httpx and ffuf output in ingest_scan_output
Extends ingest_scan_output with three formats:

- nuclei-jsonl: one tool_output fact per result and an asset for the
  scanned target when it is valid; no automatic finding
- httpx-jsonl: one HTTP service asset and one fact per URL
- ffuf-json: one URL asset and one fact per result, read token by token
  with encoding/json; a file cut short keeps the complete results

JSONL lines go through readJSONLines, taken from the bbot ingester
(64 KiB per line; longer lines and invalid JSON are skipped and
counted). bbot now uses the same reader, with no behaviour change.
One file yields at most 50000 assets and facts; records past the cap
are counted in over_cap and the result is marked truncated.
2026-11-13 19:15:59 +01:00
BaaRai 0f5d93fbee Add ingest_bbot tool (bbot JSONL, typed scope split)
ingest_bbot reads a bbot output file by path and ingests it through
the typed parser from #199 (parseBbotJSONL), one line at a time:

- Lines are read with a fixed cap (bbotMaxLineBytes = 64 KiB, newline
  included). A longer line is skipped and counted, never buffered whole.
- scope_distance == 0 events that are assets (assetItems) go through
  the existing insert_assets handler; no asset SQL is written here.
- Every other event is recorded as a fact with source
  third_party_intel. The summary is built from typed fields only; the
  free text is stored in detail as the inert Go-quoted string.
- Intel never reaches insert_assets or task_scope.

The tool is in WorkerTools, so the existing BuiltinToolSeeds/SeedTool
loop inserts its row (a new key, inserted on fresh and existing DBs).
It is listed in the target gate's noTargetTools: its input is a local
path, not a network target (TestDomainToolsDeclareNoTarget).
2026-11-12 12:35:06 +01:00