Commit Graph
848 Commits
Author SHA1 Message Date
BaaRai bc75b5dc43 chore(web): upgrade next to 16.4.6
Closes the critical Next.js advisory reported by npm audit for next <=16.4.0. Only next and its own tree move in the lock (@next/*, @swc/helpers, sharp and its @img packages, postcss and nanoid); react and react-dom are unchanged.
2026-11-15 14:35:21 +01:00
BaaRai cddcd98f3b Merge pull request #238 from BaaRai/chore/npm-prod-deps
chore(web): move the shadcn CLI to devDependencies and drop @antv/graphin
2026-11-15 14:18:15 +01:00
BaaRai d42988b6a8 chore(web): move the shadcn CLI to devDependencies and drop @antv/graphin
shadcn is only used at build time (globals.css imports shadcn/tailwind.css, and the CLI generates components); no web/src file imports it at runtime. @antv/graphin is not imported anywhere. The lock only marks the shadcn dependency tree as dev and removes @antv/graphin; no version changes.
2026-11-15 14:01:02 +01:00
BaaRai 3975e58ce0 Merge pull request #237 from BaaRai/ci/gofmt
ci: run gofmt in the go job, and format the 11 files it flags
2026-11-15 13:55:11 +01:00
BaaRai fdba29f1a9 ci: run gofmt in the go job, and format the 11 files it flags
Mechanical: gofmt -w on the 11 files gofmt -l listed on main. Besides spacing and alignment, gofmt sorted the imports inside one group in two test files and removed a trailing blank line in a third. The go job now fails, listing the files, when gofmt -l reports any tracked Go file.
2026-11-15 13:38:35 +01:00
BaaRai a59d430175 Merge pull request #236 from BaaRai/perf/task-metrics
perf(db): per-task metrics for the task detail and an index-served latest activity
2026-11-15 08:40:39 +01:00
BaaRai c2d1f557c3 perf(db): per-task metrics for the task detail and an index-served latest activity
getTask computed the list metrics of every task to read one; it now calls TaskListMetrics(expID), which runs the same laterals as TaskListMetricsAll with a WHERE on the exploration. listTasks keeps the global query. The latest_activity lateral ordered by its EXTRACT(EPOCH) output alias, so PostgreSQL read and sorted every activity row of each task; it now orders by activity.created_at and idx_act_latest serves the LIMIT 1. Both queries share one SQL constant.
2026-11-15 08:20:08 +01:00
BaaRai e636408f66 Merge pull request #235 from BaaRai/docs/upstream-changelog
docs: credit the upstream and complete the unreleased changelog
2026-11-15 02:01:55 +01:00
BaaRai fcbd87445e docs: credit the upstream and complete the unreleased changelog 2026-11-15 01:54:50 +01:00
BaaRai 59f768f1ef Merge pull request #234 from BaaRai/feat/proxy-pace-bash
feat(traffic): pace Bash requests at the capture proxy
2026-11-15 01:41:56 +01:00
BaaRai 7f3edf9bfe fix(agent): noise prompt states what the capture proxy paces
The low-and-slow block said that commands launched through Bash are not paced. With traffic capture on, Bash commands that use the capture proxy are now paced by rate and jitter; raw scanners and commands that bypass the proxy are not, so the rate flags remain the brake.
2026-11-15 01:23:56 +01:00
BaaRai 2e55615713 feat(traffic): pace Bash requests at the capture proxy
With traffic capture on, the Bash proxy URL carries a marker userinfo. The recording proxy drops Proxy-Authorization from every request before it is forwarded or recorded; when the value is the marker it marks the client connection, and every later request on it, including those inside its CONNECT tunnel, waits for the per-host token and jitter of the current noise profile (egress.Wait) and is counted, with a bounded wait. WebFetch, the http custom tools and enrich keep the unmarked URL, which egress already paces, so nothing is paced twice. Off paces nothing. Best effort: only clients that honour the proxy are covered, with capture on.
2026-11-15 00:57:13 +01:00
BaaRai 251ab680cc Merge pull request #233 from BaaRai/feat/noise-profile-runtime
feat(egress): change the noise profile at runtime
2026-11-15 00:14:23 +01:00
BaaRai ac6e2b3c44 feat(egress): change the noise profile at runtime
The egress governor is now a current governor that SetProfile swaps: new requests use the new one, requests in flight finish on the old one, and one package-level ticker logs the counters of the current governor. WebFetch, the http custom tools and the enrich probe resolve it per request, and the agent prompt block reads the current profile. The noise_profile setting (off, quiet, balanced, aggressive) is applied at startup and wins over ARTEX_NOISE_PROFILE, which stays the fallback; PUT /api/settings validates, stores and applies it without rebuilding agents, and GET /api/settings/noise returns the per-host counters without resetting them. System settings gets a Target noise card.
2026-11-14 23:30:06 +01:00
BaaRai 97b0fdced6 Merge pull request #232 from BaaRai/feat/intercept-risk-batch
feat(intercept): deterministic risk score and batch approval
2026-11-14 23:03:57 +01:00
BaaRai 8f5a67d10c fix(intercept): destructive commands score high; chunk batch decisions to the server limit
The destructive shell signal is worth 60 instead of 50, so a destructive command alone scores 10 + 60 = 70, high. The level thresholds are unchanged. The approvals page sends the selection in sequential calls of at most 100 ids, the server limit (maxBatchDecide), instead of one call that the server would refuse with 400 when more than 100 rows are selected. If a call fails, sending stops: its ids and the later ones stay selected with the error, and ids already decided stay decided. The final toast gives the totals.
2026-11-14 22:46:18 +01:00
BaaRai d8f3c7aec2 feat(intercept): deterministic risk score and batch approval
ScoreRisk scores a pending tool call from fixed, additive signals (no model): shell destructive, privilege, secret, exfiltration, reverse shell, scanner and file-write patterns; file writes with a sensitive-path bonus; web reads with a mutating-method bonus; MCP, unknown tools and unreadable input never score low. The score is computed when a pending row is read and returned as risk next to the unchanged row fields. POST /api/intercept/pending/batch-decide decides up to 100 ids, each through Interceptor.Decide with the same approver and address as a single decision, and reports a result per id. The approvals page gets row selection, a confirmation that counts high-risk rows, per-id errors, and a risk badge with its reasons.
2026-11-14 22:19:09 +01:00
BaaRai c43c69bdd0 Merge pull request #231 from BaaRai/docs/readme-refresh
docs: rewrite README and refresh screenshots
2026-11-14 21:55:11 +01:00
BaaRai a866f3752f docs: rewrite README and refresh screenshots 2026-11-14 21:40:40 +01:00
BaaRai 481dd1c3f2 Merge pull request #230 from BaaRai/perf/coverage-graph
perf(db): batch coverage-graph lookups and index icp scope probes
2026-11-14 21:40:14 +01:00
BaaRai 30c787d690 perf(db): batch coverage-graph lookups and index icp scope probes
BuildCoverageGraph issued one QueryRow per connector root domain and one per company: 221 statements for a task scoped by 800 subdomains over 200 roots and 20 companies. Fetch them with one ANY() query each. The coverage target join probes task_scope once per asset, and the icp branch fell back to uq_task_scope_v2 with only kind as index condition; migration 0003 adds a partial index on task_scope(value) for icp rows. Large-fixture invariants test and benchmark included.
2026-11-14 21:16:49 +01:00
BaaRai 430950570c Merge pull request #152 from xulytiengviet/patch-1
Create SECURITY.md
2026-11-14 20:57:08 +01:00
BaaRai f8bc149872 Merge pull request #225 from err0rgod/docs/fix-stale-postgres-docs
docs: correct stale database and boundary references
2026-11-14 20:57:03 +01:00
BaaRai f3a8e82d29 docs(security): point reports to the BaaRai repository and drop placeholders
Replace the [owner] placeholder and the unfilled email and PGP lines with GitHub Security Advisories as the sole reporting channel. Replace the 0.4/0.5 support table with the latest release line, and turn the response timeframes into best-effort goals.
2026-11-14 20:47:28 +01:00
BaaRai e300f85987 Merge pull request #229 from BaaRai/ci/race-server
ci: run ./server under the race detector
2026-11-14 19:57:20 +01:00
BaaRai dfa0e8a566 ci: run ./server under the race detector
The server suite now passes go test -race against postgres, and its test teardown no longer leaks live servers (#227, #228). Add ./server to the go test -race step; the DSN and -p 1 are unchanged.
2026-11-14 19:11:10 +01:00
BaaRai 9b20a46eba Merge pull request #228 from BaaRai/fix/flaky-server-tests
test(server): fix flaky SkillGaps, notify digest and TempDir cleanup tests
2026-11-14 19:03:16 +01:00
BaaRai 8c8ae97400 test(server): join every live test server before its TempDir cleanup
New restores every unfinished task in the shared database and runs its planner, worker and deadline loops on the server context. Tests that passed context.Background() never stopped or joined them, so they kept writing under the manager TempDir after the test returned and the TempDir cleanup failed with directory not empty. Add stopTestServer, which cancels the context, waits for each task to go quiescent, joins its loops, the archive worker and the side-question writer, and defer it right after New in every test that started a live server.
2026-11-14 18:21:52 +01:00
BaaRai 149151919f test(server): join restored task loops before TestTaskDetailReturnsListMetrics cleanup
New restores every unfinished task in the shared database and runs its planner and worker loops on the server context. They kept writing transcripts under the TempDir after the test returned, so the TempDir cleanup failed with directory not empty. Cancel the context, wait for each task to go quiescent, and join its loops before m.Close and the TempDir cleanup.
2026-11-14 16:21:31 +01:00
BaaRai 5cdb720122 test(server): stop leaked fixtures from polluting later tests
TestMgmtAPI never closed its manager, so the notifier and task-archive workers started by New(context.Background()) kept claiming other tests digest batches and archive jobs. TestTaskOutcomeCard left a skill_usage miss row behind (the table has no foreign keys), which outranked TestSkillGapsReport fixture from the fourth -count iteration on.
2026-11-14 15:44:52 +01:00
BaaRai ae792126dc Merge pull request #227 from BaaRai/fix/agent-hook-race
agent: synchronise package-level hooks (data races under -race)
2026-11-14 15:01:30 +01:00
BaaRai 54ff51177d agent: guard prompt and wrap-up override hooks against concurrent access
PromptOverride, WrapupOverride, WrapupMaxTurnsOverride,
WrapupTaskTimeoutOverride and WrapupTaskTimeoutTurnsOverride are wired
in server.New while restored tasks may already render prompts and build
settlement configs on other goroutines. go test -race ./server reported
races between server.New (server.go:195/207/215) and agent.renderSystem /
resolveWrapup / resolveWrapupTurns.

Same treatment as the tool hooks: unexported variables behind a
sync.RWMutex (one for the prompt hook, one shared by the four wrap-up
hooks), Set* setters and getters, and readers that copy the function
value under the lock and call it outside. A nil hook still means the
built-in default.

TestAgentHooksConcurrentSwap now covers all eight hooks.
2026-11-14 14:19:13 +01:00
BaaRai 37f2573a53 agent: guard ToolAugment/ToolResolve hooks against concurrent access
server wiring (wireAgentAugment, wireTools) assigned the package-level
hooks agent.ToolAugment and agent.ToolResolve while restored tasks could
already be running AugmentTools on other goroutines. go test -race ./server
reported this as a DATA RACE between agent.AugmentTools and
server.wireAgentAugment.

Each hook is now an unexported variable behind a sync.RWMutex, with a
setter (SetToolAugment, SetToolResolve) and a getter (ToolAugment(),
ToolResolve()). AugmentTools copies the function value through the getter
and calls it outside the lock. A nil hook still means no augmentation /
no resolution, as before.

FindingTrafficBindingEnabled is wired in the same server/assembly.go code
and read on the same AugmentTools path, so it gets the same treatment.

Adds TestAugmentToolsHooksConcurrentSwap, which runs AugmentTools while
the setters swap the hooks, so -race catches an unsynchronised access
without depending on test ordering.
2026-11-14 13:46:03 +01:00
BaaRai 5cb44fb969 Merge pull request #226 from BaaRai/feat/versioned-migrations
db: versioned migrations runner with schema_migrations
2026-11-14 13:19:38 +01:00
BaaRai f3b9f6f688 db: refuse a pending migration older than the newest applied one
applyMigrations now fails closed, before any write, when a pending
migration has a version below the highest version recorded in
schema_migrations: migrations must be appended, never inserted.
2026-11-14 13:10:57 +01:00
BaaRai 1da0d07222 db: versioned migrations runner with schema_migrations
Move the llm_records and llm_usage DDL out of Go constants into embedded
files db/migrations/0001_llm_records.sql and 0002_llm_usage.sql (byte-for-byte
the old constants). Open now applies pending migrations after schema.sql and
before seedBuiltins, under the existing schema advisory lock and on the
lock-holding connection. Each migration runs in its own transaction with its
schema_migrations row (version, name, SHA-256 checksum, applied_at).

Open fails closed when the database has a version missing from the binary, or
an applied migration whose file name or checksum changed. A failing migration
is rolled back and blocks startup; earlier ones stay applied.

EnsureLLMRecordsTable and EnsureLLMUsageTable re-run the embedded files and
keep their transaction-level schema lock; Open never calls them.
2026-11-14 12:52:34 +01:00
err0rgod 622b7490e0 docs: finish the post-SQLite data-directory wording
Update the Dockerfile, Compose file and upgrade note to describe what ./data and ./keys actually hold now that the graph stores are PostgreSQL.
2026-11-14 16:58:39 +05:30
BaaRai 8f3f0c54f8 Merge pull request #224 from BaaRai/fix/archive-restore-normalize-url
db: normalise archived asset URLs on restore
2026-11-14 12:27:39 +01:00
err0rgod b6e863c746 docs: correct stale database and boundary references
- describe the graph stores as PostgreSQL in the entrypoint and agent package
- document that -data holds the traffic index, task files and transcripts
- add an Unreleased note for the untrusted-output boundary default flip
- use a placeholder version in the README/build.sh release examples
2026-11-14 16:55:11 +05:30
BaaRai 2e43b423ee db: normalise archived asset URLs on restore
restoreArchiveAssets inserted archived service and endpoint rows with
their raw url, and findArchiveAssetNaturalID matched on that raw value.
An archive taken before normalizeURL dropped userinfo therefore restored
http://user:pass@host in clear and duplicated the existing clean row.

Pass row["url"] through normalizeURL for service and endpoint rows
before the natural-identity lookup, so the lookup reuses the clean row
and an insert stores the normalised URL. Other columns keep their
archived values.
2026-11-14 12:17:33 +01:00
BaaRai 8bb4ca0745 Merge pull request #223 from BaaRai/fix/store-normalize-userinfo
db: drop URL userinfo in normalizeURL before storing assets.url
2026-11-14 12:06:44 +01:00
BaaRai 939f8d4976 db: drop URL userinfo in normalizeURL before storing assets.url
UpsertHTTPService and UpsertEndpoint both key assets.url on normalizeURL,
which lowercased scheme and host but kept u.User. A URL passed with
user:pass@ (insert_assets, server imports, sync) was stored verbatim and
the same URL with different credentials produced separate rows.

Set u.User = nil before serialising. Scheme, host, port, path and query
are unchanged; an @ in the path or query is preserved.

Tests: normalizeURL unit cases, plus PG tests for both upserts covering
credential stripping, dedup across different credentials, @ in path,
and an unchanged URL without userinfo.
2026-11-14 11:56:43 +01:00
BaaRai b5c0bb9f33 Merge pull request #222 from BaaRai/fix/scan-ingest-strip-userinfo
fix(agent): strip URL userinfo in scan and bbot ingestion
2026-11-14 01:12:17 +01:00
BaaRai 9d75df335f fix(agent): strip URL userinfo in scan and bbot ingestion
scanURL kept the userinfo of httpx, ffuf and nuclei URLs, so a line such
as http://user:pass@host/path stored the credentials in assets.url, in
the notable items and in the facts. scanURL now drops u.User before
re-serializing; scheme, host, port, path and query are kept.

The nuclei fact detail also embedded the raw host and matched-at values,
the httpx detail the raw host and the ffuf detail the raw redirect
location. They now go through stripUserinfo, which removes only the
userinfo of a URL or bare host:port and keeps every other byte, so a
value without userinfo is unchanged and fact deduplication is the same.

bbotTyped drops the userinfo of the typed url of an in-scope URL event
before it becomes an asset.

Rows already stored with userinfo are not rewritten here.
2026-11-14 01:02:16 +01:00
BaaRai fe9814c5f6 Merge pull request #221 from BaaRai/chore/db-test-cleanup-order
test(db): close the pool after the row cleanups, report cleanup errors
2026-11-14 00:51:55 +01:00
BaaRai 53fc8478da test(db): close the pool after the row cleanups, report cleanup errors
A `defer d.Close()` runs when the test function returns, before its
t.Cleanup functions. The cleanups that delete the test's rows then ran on
a closed pool, the error was ignored, and the rows stayed in the database
for the next run. PR #220 fixed three files; this applies the same pattern
to every remaining db test that combines `defer d.Close()` with row
cleanups.

- Close the pool with t.Cleanup, registered before the row cleanups so it
  runs last (LIFO).
- Report cleanup errors with t.Errorf through small helpers in
  db/cleanup_helpers_test.go. A profile the test already deleted is not an
  error.

Test files only. No assertion or production change.
2026-11-14 00:38:00 +01:00
BaaRai edc2f4da6d Merge pull request #220 from BaaRai/fix/db-test-isolation
test(db): make profile and company-scope tests repeatable on one database
2026-11-14 00:19:41 +01:00
BaaRai 01a9dae49e test(db): make profile and company-scope tests repeatable on one database
TestPoolProfilesOrder, TestProfileRetryRoundTrip and
TestProfileRetryClampedOnSave closed the pool with defer d.Close(), which
runs before t.Cleanup, so the row-deleting cleanup hit a closed pool and its
error was discarded. The fixed profile names then collided with the
llm_profiles name unique key on the next run. Close via t.Cleanup registered
first, give the profiles per-run names, and report cleanup errors.
TestLLMRetryPolicyRoundTrip had the same ordering and now restores the
setting before the pool closes.

TestCompanyScopeAttribution derived its names from MAX(assets.id) and never
removed its company, so the next pass found the old company holding the same
fixed 198.51.100.0/24 scope and the in-CIDR IP was attributed to it. Use a
time plus random suffix and a per-run /24 from 198.18.0.0/15, delete the
company (its scope cascades) after the assets, and report cleanup errors.
2026-11-13 23:46:42 +01:00
BaaRai 4e126a8524 Merge pull request #219 from BaaRai/feat/release-provenance
release: provenance attestation, CycloneDX SBOM and dry-run trigger
2026-11-13 23:28:15 +01:00
BaaRai abb6cb177d release: add build provenance attestation, CycloneDX SBOM and a dry-run trigger
- release job: attest dist/*.zip with actions/attest-build-provenance
  (id-token: write, attestations: write; contents stays write for the Release).
- SBOM: syft scans the bin-linux-amd64 binary (embedded Go modules), plus
  web/package-lock.json and go.mod/go.sum; artex-<tag>.cdx.json is checked for
  pkg:golang and pkg:npm entries and attached next to SHA256SUMS.
- workflow_dispatch dry run: builds, SBOM, attestations and workflow artifacts;
  the Release step and the docker job only run on push.
- README: Verifying a release section.
2026-11-13 23:17:19 +01:00