Compare commits

..
Author SHA1 Message Date
BaaRai 64b90a746a Merge pull request #292 from henefesto/docs/local-systemd-deployment-32ebb38
docs: add local systemd deployment guide and verified reboot results
2026-11-20 07:46:02 +01:00
BaaRai dbbc287cd8 Merge pull request #395 from BaaRai/security/agent-exec-isolation
security: contain agent execution (exec-uid drop, run dirs, spills, Norma temp)
2026-11-20 07:42:38 +01:00
BaaRai 1675f3683b docs(security): TMPDIR must be a root-controlled path
The start-up check of the Norma temp directory follows TMPDIR and inspects
<tmp>/norma and its immediate parent only: a symlink or an ancestor below a
directory the executor can write is not detected, and POSIX ACLs are not
inspected. One sentence in the Agent Process Isolation paragraph says so.
2026-11-20 06:38:11 +01:00
BaaRai 73d35835c5 test(agent,server): run the tests in a temp directory of their own
Norma keeps background-task output in os.TempDir()/norma. A test that starts
a Norma session without a private TMPDIR wrote into the host's /tmp/norma.

TestMain of the agent and server packages now creates a directory with
os.MkdirTemp("", "artex-test-tmp-"), sets TMPDIR to it before m.Run and
removes it afterwards on every exit path. The directory is world-writable
and sticky like /tmp: MkdirTemp creates it 0700, which the executor of the
root tests could not traverse. In the server package the setup comes after
RunExecShimIfRequested, which stays the first call; the database suffix,
the test key and the advisory lock run in the same order as before. A test
that sets TMPDIR itself keeps its own.
2026-11-20 06:38:10 +01:00
BaaRai c215a06f3a docs(security): a Norma temp directory the executor can write fails the start
The Norma sentence of the Agent Process Isolation paragraph gets one clause:
a pre-existing <tmp>/norma that the executor can write makes start-up fail,
which can refuse a start that worked before. The operator removes that
directory or sets TMPDIR to a directory the executor cannot write. ARTEX
never modifies or deletes the directory.
2026-11-20 06:36:48 +01:00
BaaRai f483ecec83 test(server): cover the Norma temp directory rules in a private temp directory
TestCheckNormaTaskDir runs every rule of checkNormaTaskDir on stat facts
alone, without root and without a file system.

On a real directory, with and without root:
- TestPrepareNormaTaskDirAlreadyThere: a directory the executor uid owns, one
  its gid can write in and an other-writable one fail the start, and the error
  names the path, the owner and group, the mode, the executor and the remedy;
  one of another user or of root that the executor cannot write in is used as
  found. Owners, modes, times and contents of the directory and of what it
  holds are hashed before and after: nothing changes in either case.
- TestPrepareNormaTaskDirParent: a parent the executor can write in and that
  is not sticky fails the start; the same parent with the sticky bit passes.
- TestPrepareNormaTaskDirNotADirectory: a symlink, a dangling symlink and a
  regular file fail the start, and nothing is created through the link.
- TestPrepareNormaTaskDirDropDisabled: with the drop disabled nothing is
  checked or created.

As root, TestNormaTaskDirPlantedByExecutorAsRoot lets an executor process
create <temp>/norma with a symlink in it, as it could before a start:
PrepareExecPrivilege fails and leaves the directory as it is.

The root fixture now gives each test its own temp directory, 1777 like /tmp.
PrepareExecPrivilege and Norma's Bash tool reach <temp>/norma, and before
this every root test went to the one of the host. The new tests set TMPDIR to
t.TempDir() for the same reason.
2026-11-20 06:36:48 +01:00
BaaRai 36940cdf0b security(server): decide the Norma temp directory check from the executor identity
The check of <temp>/norma, where Norma writes background-task output from
the server process, now lives in checkNormaTaskDir. It takes the Lstat of
the path, the Stat of its parent and the executor uid and gid, and decides
from the owners and the mode bits. An access check would answer for root,
which the server is.

The start fails when:
- the path is a symlink or not a directory;
- the executor uid owns the directory: under a sticky /tmp the owner can
  still rename it and put another in its place;
- the group or other bits let the executor write in it;
- the parent directory is writable by the executor and not sticky, or the
  executor owns the parent: the sticky bit does not stop the owner of a
  directory from renaming an entry in it.

The error names the path, its owner and group, its mode, the executor uid
and gid, and the remedy: remove the directory or set TMPDIR to a directory
the executor cannot write.

Any other directory already there is used as found, with the same warning as
before when root does not own it or its group can write in it. A missing
directory is still created root:root 0755 and goes through the same check.
A directory that was already there is never chmodded, chowned, renamed or
removed. With the privilege drop disabled nothing is checked or created.
2026-11-20 06:36:48 +01:00
BaaRai 992fcf7030 security(server): refuse to start when the executor can write the Norma temp directory 2026-11-20 06:36:48 +01:00
BaaRai e555ecd466 docs(security): describe the ask_operator staging as it now works 2026-11-20 06:36:48 +01:00
BaaRai de070c964a test(server): keep the database environment for the tests after the env scrub test
TestSecretEnvCoversCredentialURLs calls ScrubSecretEnv, which unsets every
name in secretEnv for the rest of the test binary, ARTEX_PG_DSN and
ARTEX_SECRET_KEY included. The leak is new on this branch: main has no such
test. Without the fix a merge would silently skip 148 server DB tests in CI:
they skip when ARTEX_PG_DSN is not set, and the run stays green.

The fix is in the test: the loop over secretEnv restores every name with
t.Setenv, so each one gets its value back when the test ends.
2026-11-20 06:36:47 +01:00
BaaRai 72fe3c3da1 Merge pull request #394 from BaaRai/feat/replay-rewrite-toggle
feat(traffic): per-replay rewrite-rule toggle and rewrite stats (P3.3b2b1)
2026-11-20 06:02:23 +01:00
BaaRai 15ed120758 test(traffic): adversarial coverage for the replay rewrite toggle
Duplicate and lower-case nonce headers are stripped without bypassing,
the registry is single-use and rejects multi-valued fields, a forged
caller header cannot steer either mode, concurrent bypass and apply
replays do not share state, SSE streams bypass by default, a cancelled
replay leaves no nonce or marker, and the route rejects a non-boolean
apply_rules.
2026-11-20 05:48:34 +01:00
BaaRai ebb90199c5 security(agent): name the Norma background output directory
Norma keeps the output of background tasks in os.TempDir()/norma
(<tmp>/norma/<session>/tasks/task_<n>.output), written from the server
process with MkdirAll and an open that follows symlinks. ARTEX does not
choose that path, and the temp directory is world-writable: an executor
process that runs before the first session creates the directory (a stdio
MCP server, a process left from an earlier start) could create it first and
place symlinks in it for the server to write through.

- PrepareExecPrivilege, only when the drop is enabled, now creates
  <TempDir>/norma itself (prepareNormaTaskDir): Mkdir 0755 as root, then
  Lstat. A symlink or a file in its place is refused and the server does
  not start.
- A directory that is already there is used as found. It is not chowned:
  it may belong to another user of the host. When root does not own it or
  others can write in it, a warning is logged and nothing else changes.
- SECURITY.md adds this output to the residual list of the Agent Process
  Isolation paragraph, with what the start-up step does and does not cover.

Norma is unchanged.

Tests: TestPrepareNormaTaskDir (created when absent, kept when there,
symlink and file refused, nothing created through the symlink);
root-only TestNormaTaskDirAsRoot (PrepareExecPrivilege leaves
<TMPDIR>/norma root:root 0755 in a sticky world-writable directory, and
an executor process can neither add an entry nor rename it).
2026-11-20 05:35:57 +01:00
BaaRai d85fb16ba3 docs: ask_operator image_path resolves from the run directory
The Unreleased entry still said image_path resolves inside <data>/tasks/<id>. It now states the rule in the code: a relative path resolves from the run directory of the agent that asks, an absolute path must be inside it, and the copy is staged in tasks/<id>/uploads. A second entry names the change for a worker that passed i<N>/shot.png relative to the task directory.
2026-11-20 05:35:56 +01:00
BaaRai 4fa45eb449 security(agent): ask_operator staging refuses symlinks and never overwrites
The staged copy of an ask_operator image was written by the server with
os.Create under a name built from the second, in an uploads directory made
with MkdirAll. An agent that can write in uploads/ (a task directory from
before the run-directory layout) could prepare that name as a symlink, and a
symlinked uploads/ was followed.

- uploads/ is created with Mkdir and checked with Lstat: a symlink or a
  file in its place is refused.
- The name is ask-operator-<unixnano>-<8 random hex>.<ext>, created with
  O_CREATE|O_EXCL, mode 0640. A taken name is never opened: another name is
  tried, five at most.
- The source is opened only when it is still the regular file Lstat saw
  (os.SameFile on the Lstat and the Stat of the open file), which refuses a
  file swapped for a symlink between the EvalSymlinks check and the open.
  The run-directory confinement is unchanged.

Not closed: a directory inside the run directory swapped for a symlink
between the check and the open still leads both Lstat and the open to the
same file elsewhere.
2026-11-20 05:35:56 +01:00
BaaRai 7be765bff2 security(agent): restore group sharing for legacy task entries; resolve ask_operator paths from the run dir
The layout commit made tasks/<id> root:root 0755 and shared only the run
directories. Entries that sat directly in a task directory before it
(uploads/, files the main agent wrote there) were left as found and
nothing shared them any more: after an archive restore, which keeps
neither owners nor modes, the executor could not use them. ask_operator
still resolved a relative image_path from tasks/<id>, not from the
directory the agent works in.

- shareTaskDir: after the task directory is set root:root 0755, each
  direct entry that is not a run directory (main, i<N>) is shared with
  the executor group again (shareWithExecGroup: group owner, group
  read/write, traverse and setgid on directories). Run directories are
  handled as before. The task directory stays root-only, so the executor
  works inside an old entry and cannot rename, remove or add one. A
  symlink among the entries is not followed. The pass runs at start-up
  and on archive restore (installTaskArchiveFiles calls
  shareRestoredTaskDir and rolls back on failure), with no operator step.
- ask_operator: image_path is resolved inside the run directory of the
  agent that asks (agent.EnsureRunDir: tasks/<id>/i<N> for a worker,
  tasks/<id>/main without an intent). An absolute path outside it, "..",
  and a symlink leading out are refused as before. The copy is still
  staged in tasks/<id>/uploads. The tool description says so.

Tests: root-only TestShareTaskDirLegacyEntriesAsRoot (owners and modes,
an executor probe, links to a root-only directory and file left alone);
TestRunDirLayoutOnExistingTreeAsRoot now expects the old file shared;
TestTaskArchiveRestoreRunDirOwnershipAsRoot restores an entry of the task
directory; TestStageAskOperatorImage uses the i<N> and main leaves.
SECURITY.md states the sharing of the old entries and the empty main/.
2026-11-20 05:35:56 +01:00
BaaRai 7520c6e588 security(agent): command-output spills go to a root-owned per-task dir
Norma's tool.Capture writes an oversized tool output to a file in
ToolOutputDir, from the server process. ARTEX passed <run dir>/cmd-output:
the executor can write in the run directory and could turn cmd-output into
a symlink, so root wrote the spill where the link pointed.

- With the privilege drop on, ToolOutputDir is <data>/spill/tasks/<id> for
  the worker, the main agent and the planner of a task, and
  <data>/spill/sessions/<id> for a chat. The server creates it when the
  run starts (agent.SetSpillDirMaker, same style as SetRunDirMaker):
  root:<exec gid> 0750 below root:root 0755 parents, each level through a
  descriptor opened with O_NOFOLLOW. A directory that cannot be created
  leaves the run without a spill directory (output is truncated), never
  with the run directory. Without the drop the path stays
  <run dir>/cmd-output.
- Norma v0.4.11 writes the spill file 0600 (directory 0700), so the
  executor could not read it. The runs wrap their hooks (agent.SpillHooks):
  after a tool call whose result carries the spill marker, the server makes
  the files of the directory root:<exec gid> 0640. The agent reads a spill
  with Bash; the file tools do not reach outside the run directory.
- The spill directory follows the task: stageTaskFiles moves it to the
  delete staging with tasks/<id>, stageTaskArchiveFiles packs it as
  files/spill/tasks/<id>, and installTaskArchiveFiles restores it and puts
  its ownership back. Deleting a conversation removes
  spill/sessions/conv-<id>.

Tests: TestToolOutputDir (agent); TestDeleteTaskFilesRemovesSpillDir,
TestRemoveConversationSpill and the archive round trip (server); root-only
TestSpillDirAsRoot, and the spill directory in
TestTaskArchiveRestoreRunDirOwnershipAsRoot. SECURITY.md drops the spill
from the residual list.
2026-11-20 05:35:55 +01:00
BaaRai e73cdeda93 security(agent): share only run-dir leaves; main agent gets its own leaf
The task and chat workspace trees were shared whole with the executor
group, and the main agent and the planner worked in tasks/<id>, the parent
of every worker directory: the executor could rename, remove or replace a
run directory, its own included, and plant entries beside them.

- Layout: no agent working directory is an ancestor of another. The main
  agent and the planner work in tasks/<id>/main, a worker in
  tasks/<id>/i<N>, a chat in sessions/<id>. With the privilege drop on,
  tasks/, sessions/ and tasks/<id> are root:root 0755 and only the leaves
  are root:<exec gid> 2775.
- PrepareExecPrivilege no longer shares tasks/ and sessions/. It sets the
  parents, shares the leaves found on disk with what they hold, and
  registers the run-directory maker (agent.SetRunDirMaker). ensureRunDir
  and the chat session directory go through it: each level is created
  with Mkdir and set through a descriptor opened with O_NOFOLLOW, parents
  before the leaf, so a link planted where a run directory goes is
  refused. Without the drop both stay a plain MkdirAll.
- Uploads for the main agent land in tasks/<id>/main/uploads, and the
  attachment paths given to it use the same directory; session uploads
  create sessions/<id> by the same rule.
- Task archives still pack and delete the whole task directory, so the
  package holds files/tasks/<id>/main and i<N>. The package keeps neither
  owners nor directory modes: the restore puts the layout back
  (installTaskArchiveFiles takes the privilege).
- Existing data is not migrated. Files a task wrote directly in
  tasks/<id>/ stay there with their owner and mode; the main agent starts
  in an empty tasks/<id>/main and its file tools no longer reach them.

Tests: TestRunDirLayout and TestIsRunDirName (agent); root-only
TestRunDirLayoutAsRoot, TestRunDirSiblingsProtectedAsRoot,
TestRunDirLayoutOnExistingTreeAsRoot and
TestTaskArchiveRestoreRunDirOwnershipAsRoot (server). The archive tests
use the main/ and i<N> leaves. SECURITY.md states the invariant, names the
race that remains and drops ensureRunDir from the residual list.
2026-11-20 05:35:55 +01:00
BaaRai db6d73db57 docs(agent): rea writes its output inside the run dir 2026-11-20 05:35:55 +01:00
BaaRai 0c0db6631d docs(agent): bbot recipe writes inside the run dir; name the open root paths 2026-11-20 05:35:55 +01:00
BaaRai c1c1fe9141 security(agent): confine ingest_bbot; write tool scripts outside the run dir
Two root file operations took a path the agent controls.

- ingest_bbot opened the path it was given with os.Open in the server
  process, which is root in the image: any root-readable file could be
  named, and the result counts told its line structure. The tool now opens
  the file through the file tools' confinement (openWorkRoot on the run's
  working directory under the data directory, then os.Root): an absolute
  path elsewhere, "..", a symlink, a hard-linked file and a run directory
  reached by a symlink are refused. It fails closed without a data
  directory or a working directory. The worker passes its data directory
  to the ToolSet (SetWorkBase); the tool reads the ToolContext for the
  working directory, like ingest_scan_output.
- execPython wrote the script of a custom tool as root to
  <run dir>/.tools and then chowned it. The executor can write the run
  directory, so it could turn .tools into a symlink and have root create
  and chown a file elsewhere. With the privilege drop on, the script now
  goes to <data>/exec-scripts, created by PrepareExecPrivilege as
  root:<exec gid> 0750, outside tasks/ and sessions/: the executor reads
  the 0640 script and cannot create, rename or replace an entry. An
  enabled drop without that directory is an error, not a fallback to the
  run directory. Without the drop nothing changes. PrepareExecPrivilege
  takes the privilege by pointer to record ExecPrivilege.ScriptDir.

Tests: TestOpenBbotOutputConfined and TestIngestBbotToolConfined (no
Postgres), TestExecPythonNeedsScriptDirWhenDropping, and the root-only
TestExecPythonScriptDirAsRoot (.tools planted as a symlink to a root-only
directory). TestIngestBbotScopeSplit now places its file in a run
directory. SECURITY.md: the two paths leave the residual list; the
command-output spill stays.
2026-11-20 05:35:54 +01:00
BaaRai ebcfe532e0 security(agent): refuse run dirs reached by symlink; scrub S3 archive keys
- openWorkRoot: refuse the run directory when its resolved path differs from
  the path under the data directory, so a relative symlink that stays inside
  the data directory (run -> traffic/_ca) no longer reaches root-only files.
- Test: TestConfinedFileToolsRunDirRelativeSymlinkInside targets <base>/traffic/_ca.
- secretEnv: add ARTEX_ARCHIVE_S3_ACCESS_KEY_ID, ARTEX_ARCHIVE_S3_SECRET_ACCESS_KEY,
  ARTEX_ARCHIVE_S3_SESSION_TOKEN; the sealed-env test covers them.
- SECURITY.md: document the run-dir race and the residual root writes through
  agent-influenced paths (cmd-output spill, .tools, bbot ingest).
2026-11-20 05:35:54 +01:00
BaaRai eb5d3d73c1 security(agent): confine the file tools and close the executor review
The SDK file tools Read, Write, Edit, MultiEdit, LS, Glob and Grep run in
the server process, which is root in the image, and took any absolute or
relative path: Read returned /app/keys/jwt.key or the config file, and
Write could plant a skill or rewrite the config past the approval gate.

- agent/filetools.go: defaultTools replaces the seven tools for every
  agent kind (chat, main agent, planner, worker). Same names, prompts and
  schemas; every path goes through an os.Root on the run's working
  directory, opened component by component below the data directory, so
  absolute paths elsewhere, "..", symlinks leading out, a run directory
  swapped for a symlink and files with extra hard links are refused. Grep
  no longer starts ripgrep. The LLM connection test passes an empty tool
  list instead of nil (nil meant the unconfined defaults).
- traffic: chmod the go-mitmproxy CA private key to 0600 after the proxy
  creates or loads it; the CA certificate stays readable.
- server: the executor gets HOME and XDG_CACHE_HOME in <data>/exec-home,
  owned by it, on the bash shim, the MCP drop shim and the custom-tool
  credential path.
- server: ARTEX_LLM_PROXY and ARTEX_LLM_BASE_URL can carry user:pass@;
  they are scrubbed and sealed with the other startup secrets.
- docker: no-new-privileges for the artex service, setuid/setgid bits
  stripped in the image, PLAYWRIGHT_BROWSERS_PATH kept at root's cache.
- db: the persistence approval seed inserts each rule only when no rule
  of that name exists, so a re-run does not duplicate rules.
- Root-gated tests run the bash shim, a custom tool and an MCP stdio
  command as the executor and check the uid, a 0600 root file, the
  server's /proc environ and a cache write.
2026-11-20 05:35:54 +01:00
BaaRai 04a7ab3c39 security(server): keep server secrets from agent processes
GHSA-prm8-vw2v-wjwh: the agent command executor ran as root in the server's
container and PID namespace, so it could read /app/keys and recover the
server secrets from /proc/<pid>/environ.

Deliverable 1, executor identity. Agent processes (Bash, the interactive
shell, custom tools, stdio MCP servers) run as the uid/gid set by
-exec-uid and -exec-gid (default 10001). Custom tools use
SysProcAttr.Credential. Norma starts Bash and the shell itself, so a PATH
shim (the server binary named bash) drops the identity before exec; MCP
stdio commands run through the same binary in drop mode. The task and chat
workspace trees are shared with the executor group (setgid, group rw,
umask 002). The key files and the config file become root-only, and
/app/keys is 0700 in the image. An explicit -exec-uid that cannot be applied
stops the start; the default logs once that same-uid isolation is
unavailable when the server is not root.

Deliverable 2, sealed environment. On Linux, SealProcessEnv re-executes the
server once with the secret variables removed from its environment block,
passing the values through an inherited pipe. The new image restores them
into the Go environment, so readers see the same configuration. Off Linux it
logs that the environment is not sealed.

Deliverable 3, approval gate. Custom tools, MCP servers and skill files
(create_/update_ of custom tool and MCP, create_skill, update_skill_file)
meet built-in ask rules seeded once (intercept_persistence_rules_v1). A user
allow rule cannot skip them; a user deny still wins. The rules are data the
agent cannot edit.

Tests: resolution and refusal of root identities, drop command wrapping,
sealed-environment encoding, and an end-to-end handoff test on Linux that
checks the kernel environment block and a child process hold no secret
names and that the values survive (empty and multi-line included). Approval
matching is tested in intercept.
2026-11-20 05:35:53 +01:00
BaaRai 221ad67549 feat(traffic): per-replay rewrite-rule toggle and RewriteStats API (P3.3b2b1) 2026-11-20 05:25:35 +01:00
BaaRai 90c002befc Merge pull request #392 from BaaRai/feat/evidence-rewrite-flag
feat(evidence): record capture-proxy rewrite provenance on traffic snapshots
2026-11-20 04:33:29 +01:00
BaaRai 9562248508 feat(evidence): record capture-proxy rewrite provenance on traffic snapshots (P3.3b2a)
Add rewritten and rewrite_rule_ids provenance columns to traffic_evidence_snapshots (migration 0008), enrich prepared snapshots from traffic.RewriteInfo outside the ReadEvidence callback, exclude both fields from TrafficSnapshotID so snapshot IDs remain compatible with older binaries, and display applied rule IDs in TrafficEvidenceViewer (with tr.json i18n entries).

Note: snapshots created before migration 0008 keep rewritten=false because InsertEvidenceSnapshotTx uses ON CONFLICT DO NOTHING and does not backfill provenance on existing rows; this is accepted and intentionally not backfilled here.
2026-11-20 04:00:34 +01:00
BaaRai bd95c2669f Merge pull request #375 from Alexandre200-lab/main
Refactor PostgreSQL test skipping logic and improve documentation
2026-11-20 03:33:50 +01:00
BaaRai dbf5cc3186 docs: generalize systemd example unit and deployment guide (#292) 2026-11-20 03:26:50 +01:00
BaaRai 0a968d71ae docs: correct the proxy pin and argv descriptions
Pins are name-only behind a proxy (name-only since #370), and deny/ask rules now inspect compound commands (#377); the allow rule stays strict. Also state the untrusted-boundary default plainly.
2026-11-20 03:18:21 +01:00
BaaRai 705ac80128 Merge pull request #380 from Amineharrabi/feat/stream-messages-direction-filter
feat(traffic): filter stream messages by direction
2026-11-20 03:06:08 +01:00
BaaRai ba84c374a3 Merge pull request #390 from BaaRai/security/llm-models-ssrf
security(server): block SSRF in POST /api/llm/models
2026-11-20 02:58:03 +01:00
unknownandClaude Opus 5.5 d22e9e0e58 feat(traffic): filter stream messages by direction
GET /api/traffic/stream and the traffic_messages tool take an optional
direction (c2s or s2c) to list one side of a WebSocket or SSE
conversation. total and offset apply to the filtered list, seq stays the
position in the whole stream, and the meta limit marker is kept in a
filtered page so dropped messages are never hidden. Any other value is
rejected (400 / tool error).

Part of #379 (traffic inspection: filter by direction).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 22:37:23 +01:00
Alexandrelab-200 33684f378b test(cmd): make snapshot test independent of Redact internals 2026-11-19 14:55:18 -03:00
Alexandrelab-200 1f0b30cb6a fix(test): assert redaction without pinning the mask marker 2026-11-19 14:48:46 -03:00
Alexandrelab-200 cb0915319c fix(test): use pgdb alias for skip helper in aliased imports 2026-11-19 14:43:29 -03:00
Alexandrelab-200 f081449d58 fix(db): declare Skip on the test skipper interface 2026-11-19 14:38:23 -03:00
Alexandrelab-200 f85eb86103 Merge remote-tracking branch 'refs/remotes/origin/main' 2026-11-19 14:25:18 -03:00
Alexandrelab-200 db0aae3657 Refactor PostgreSQL test skipping logic
- Introduced uniform skipping functions in the db package: SkipPGUnavailable and SkipNoPG.
- Updated all relevant test files to utilize these new functions for better consistency and readability.
- Added new db/testpg.go file to house the skipping logic.
- Created architecture and operations documentation to provide a comprehensive overview of the ARTEX system.
- Added a verification script to streamline local testing and ensure code quality.
2026-11-19 14:23:29 -03:00
ParallelsandWarp 32ebb38e0f docs: document local systemd deployment and verified reboot persistence
Add the ARTEX systemd unit and document PostgreSQL setup, application deployment, operations, and successful automatic startup after reboot.

Co-Authored-By: Warp <agent@warp.dev>
2026-11-17 09:20:13 -04:00
191 changed files with 8421 additions and 538 deletions
+1 -1
View File
@@ -125,7 +125,7 @@ jobs:
# shellcheck is preinstalled on the ubuntu runners. -S warning: the scripts
# currently have only info-level notes (intentional A && B || C chains in
# update.sh, an unquoted glob in backup.sh's prune), so fail on warnings.
- run: shellcheck -S warning install.sh update.sh start.sh scripts/backup.sh reset-password.sh scripts/extract-changelog-section.sh scripts/test-extract-changelog-section.sh .github/scripts/bot.sh scripts/test-bot-context.sh
- run: shellcheck -S warning install.sh update.sh start.sh scripts/backup.sh scripts/verify.sh reset-password.sh scripts/extract-changelog-section.sh scripts/test-extract-changelog-section.sh .github/scripts/bot.sh scripts/test-bot-context.sh
- run: bash scripts/test-extract-changelog-section.sh
# The triage bot's repository summary must expose every top-level
# directory; a truncated file list once made it deny that docs/ and
+4 -2
View File
@@ -24,8 +24,9 @@ All notable changes to this project are documented in this file. The format is b
- **Opt-in update banner**: `ARTEX_UPDATE_CHECK` makes the backend check the latest GitHub release at most once per 24 hours and the UI show a banner; off by default (#202).
- **Release provenance**: release archives carry GitHub build provenance attestations and a CycloneDX SBOM (`gh attestation verify <zip> --repo BaaRai/ARTEX`); release notes are the matching `CHANGELOG.md` section (#219, #201).
- **Capture-proxy rewrite rules, storage and API**: `GET`, `POST`, `PUT` and `DELETE /api/traffic/rewrite-rules` and `POST /api/traffic/rewrite-rules/{id}/toggle` manage rules that match a request or a response (method, host, URL, headers, body, status) and describe a rewrite (method, path and query, headers, body, status). Rules are validated on save, limited to 100 of which 50 enabled, created disabled, and stored in the new table `traffic_rewrite_rules` (migration `0005_traffic_rewrite_rules.sql`) with `matchers` and `actions` encrypted at rest. The value of an action marked `sensitive` is never returned, and an update sent without it keeps the stored one. The server compiles the enabled rules at startup and after every change and hands the snapshot to the proxy. The proxy applies them from the next entry. See `docs/traffic-rewrite-rules.md`.
- **Capture-proxy rewrite rules applied**: the capture proxy applies enabled rules to each HTTP request before it is sent to the target and to each response before it reaches the client. Each action that ran is recorded in the new table `exchange_rewrites` in the same transaction as the exchange, `traffic_get` shows a `REWRITTEN` line, and `traffic_search` never finds a value a rule wrote. SSE status and headers are rewritten at the start of the stream. **WebSocket handshakes are not rewritten**: the capture library runs its hook after the upgrade, so the status and headers cannot be changed, and frames pass unchanged. **Responses of 5 MiB or more** are forwarded unchanged and not recorded in full, because the library does not call the response hook for them. Snapshots of original requests are capped at 1024 entries and 64 MiB. Replays go through the proxy, so the rules apply to them. Archives, a per-replay rule toggle, the web UI and the proof flag come later.
- **Capture-proxy rewrite rules applied**: the capture proxy applies enabled rules to each HTTP request before it is sent to the target and to each response before it reaches the client. Each action that ran is recorded in the new table `exchange_rewrites` in the same transaction as the exchange, `traffic_get` shows a `REWRITTEN` line, and `traffic_search` never finds a value a rule wrote. SSE status and headers are rewritten at the start of the stream. **WebSocket handshakes are not rewritten**: the capture library runs its hook after the upgrade, so the status and headers cannot be changed, and frames pass unchanged. **Responses of 5 MiB or more** are forwarded unchanged and not recorded in full, because the library does not call the response hook for them. Snapshots of original requests are capped at 1024 entries and 64 MiB. Replays (`POST /api/traffic/replay`) bypass capture rewrite rules by default via an internal `X-ARTEX-Replay-Nonce` header stripped unconditionally by the proxy, and apply the rules when opted in with `apply_rules: true` (the **Apply capture rewrite rules** checkbox in the Replay panel). `GET /api/traffic` exposes the proxy's `rewrite` counters (`large_responses`, `capacity_refusals`, `tombstone_drops`, `conversions`, `sweeps`). Archives and the rule-management / traffic-list web UI come later.
- **Bulk finding status** (#298): select findings in the findings table and apply one disposition to all of them, up to 500 per call, in one transaction; each row still goes through the single-decision notification semantics and the change is mirrored onto its exploration node (#289).
- **Stream message direction filter** (#379): `GET /api/traffic/stream` and the `traffic_messages` agent tool take an optional `direction` (`c2s` or `s2c`) that lists only one side of a WebSocket or SSE conversation. `total` and `offset` apply to the filtered list, each row keeps its `seq` in the whole stream, and the `meta` row (messages dropped past the recorder limits) stays in a filtered page. Any other value is a 400 (API) or a tool error.
- **Traffic replay** (#266): replay a recorded request from the traffic detail with its method, URL, headers and stored body; replay is disabled when only the body preview is loaded.
- **Finding proof strength in reports** (#268): the Markdown report and the export show each finding's proof strength next to its verification status.
- **Opt-in Keenable web search backend** (#295): the search tool gains Keenable alongside Brave and Tavily; the key is stored encrypted and can be cleared independently.
@@ -65,7 +66,8 @@ All notable changes to this project are documented in this file. The format is b
- **Ingested traffic headers**: `RecordExternal` rejects a method that is not an HTTP token and header names or values that contain CR, LF, or NUL, matching Replay, so ingested exchanges cannot inject extra header lines into stored evidence.
- **Browser MCP starts offline** (#329): the seeded `browser` MCP ran `npx @playwright/mcp`, which contacts the npm registry on every start even with the package installed globally, and Playwright MCP defaulted to a Chrome channel the image does not ship. It now runs the image's `playwright-mcp` binary with `--headless --browser chromium`. Existing databases that still have a stock `npx @playwright/mcp` row (including the common `-y` form) or a `playwright-mcp` row with no `--browser` are rewritten once; any other command or package is left alone. The image also installs the Chromium build `@playwright/mcp` and `@playwright/cli` expect, which is not the same build `playwright install` puts in the cache.
- **CREATE DATABASE identifiers**: `ensureDatabase` now builds the statement with `pgx.Identifier`, which quotes the DSN database name and doubles any embedded quote, so the name cannot close the identifier. No database name PostgreSQL accepts is refused, so existing installs with a hyphen, a dot or non-ASCII characters keep starting.
- **ask_operator image_path**: the screenshot path must resolve inside the task working directory (`<data>/tasks/<id>`). Absolute paths, `..` segments, and symlinks that leave that directory are refused, so the tool cannot copy an arbitrary server image into the operator transcript.
- **ask_operator image_path**: the screenshot path must resolve inside the run directory of the agent that asks (`<data>/tasks/<id>/i<N>` for a worker, `<data>/tasks/<id>/main` for the main agent). A relative path resolves from that directory and an absolute path must be inside it; `..` segments and symlinks that leave it are refused, so the tool cannot copy an arbitrary server image into the operator transcript. The copy is staged in `<data>/tasks/<id>/uploads`.
- **ask_operator image_path, behavior change**: a worker that passed `i<N>/shot.png` relative to the task directory must now pass `shot.png` from its working directory, or an absolute path inside its run directory.
- **insert_assets fail-closed**: when intercept or task allow/block rules cannot be loaded, `insert_assets` refuses the whole call instead of evaluating an empty rule set (which allowed every asset). The tool result does not include the underlying database error.
- **BlobRange length no longer wraps int64**: `offset+length > size` overflowed when length was near MaxInt64, so the cap was skipped and `make([]byte, length)` could exhaust memory. The bound is now `length > size-offset`.
- **PostgreSQL text[] literals escape backslashes**: `marshalStringArray` quoted double quotes but not backslashes, so a value ending in `\"` could close the current array element and inject another. It now doubles backslashes before quotes, matching `marshalJSONBArray`.
+10 -1
View File
@@ -189,6 +189,10 @@ RUN npm install -g "@playwright/mcp@${PLAYWRIGHT_MCP_VERSION}" "@playwright/cli@
&& playwright install --with-deps chromium \
&& "$(npm root -g)/@playwright/mcp/node_modules/.bin/playwright-core" install chromium \
&& rm -rf /var/lib/apt/lists/*
# Agent processes run as the executor with HOME and XDG_CACHE_HOME in the data
# directory (server/execpriv_unix.go); the browsers installed above stay in root's
# cache, which /root's 0711 lets the executor reach.
ENV PLAYWRIGHT_BROWSERS_PATH=/root/.cache/ms-playwright
WORKDIR /app
# Binary compiled from source in the build stage above.
COPY --from=build /out/artex /app/artex
@@ -196,7 +200,12 @@ COPY --from=build /out/artex /app/artex
# to artex — docker stop only signals PID 1; without forwarding, artex never receives it,
# cannot shut down gracefully, and is hard-killed by SIGKILL after 10 seconds.
COPY start.sh /app/start.sh
RUN chmod +x /app/artex /app/start.sh && mkdir -p /app/keys
RUN chmod +x /app/artex /app/start.sh && mkdir -p -m 0700 /app/keys \
&& chmod 0711 /root
# No setuid or setgid programs (su, passwd, mount, bbot's chrome_sandbox, ...): an
# agent process runs as the executor and must not regain root through one, even
# when the container runs without docker-compose.yml's no-new-privileges.
RUN find / -xdev -perm /6000 -type f -exec chmod a-s {} +
COPY skills/ /app/skills/
# persistence points: data/ (traffic index, task files, transcripts) and keys/ (jwt.key, secret.key, via ARTEX_JWT_KEY_DIR)
VOLUME ["/app/data", "/app/keys"]
+5
View File
@@ -659,6 +659,7 @@ CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
- Backend: `go run ./cmd/artex` (without `-tags embedui`, the frontend is not embedded)
- Frontend: `cd web && npm run dev` (`/api` is reverse-proxied to the backend, with hot reload)
- Tests: `go test ./...`
- Reproduce all CI checks locally: `./scripts/verify.sh` (add `--full` for `-race` and the static web build)
- Mock preview (no backend): `cd web && NEXT_PUBLIC_MOCK=1 npm run dev`
---
@@ -726,6 +727,10 @@ flowchart TB
| **db** | PostgreSQL persistence (pgx); embedded schema and numbered migrations applied at startup |
| **Supporting** | Recording MITM proxy, approval gate, async enrichment, MCP/skills/memory/report |
For the code-level map (execution flow, trust boundaries, decisions) see
[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md); for startup, configuration precedence,
backup and key rotation see [docs/OPS.md](docs/OPS.md).
---
## Security
+4
View File
File diff suppressed because one or more lines are too long
+203
View File
@@ -0,0 +1,203 @@
package agent
import (
"context"
"encoding/json"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"github.com/Autumn-27/artex/db"
actool "github.com/Autumn-27/norma/tool"
)
const bbotConfinedSecret = "TOPSECRET-bbot-ingest"
type bbotConfinedFixture struct {
base, runDir, secret string
}
// newBbotConfinedFixture lays out a data directory like the server's: a worker run
// directory under tasks/, and a key file outside it (the server reads it as root).
func newBbotConfinedFixture(t *testing.T) *bbotConfinedFixture {
t.Helper()
base := t.TempDir()
runDir := filepath.Join(base, "tasks", "1", "i7")
keys := filepath.Join(base, "keys")
for _, d := range []string{runDir, keys} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
// One line of bbot JSON, so a read that got through would count an event.
secret := filepath.Join(keys, "jwt.key")
line := `{"type":"FINDING","host":"` + bbotConfinedSecret + `","scope_distance":0}` + "\n"
if err := os.WriteFile(secret, []byte(line), 0o600); err != nil {
t.Fatal(err)
}
return &bbotConfinedFixture{base: base, runDir: runDir, secret: secret}
}
func (f *bbotConfinedFixture) symlink(t *testing.T, target, name string) {
t.Helper()
if err := os.Symlink(target, filepath.Join(f.runDir, name)); err != nil {
if runtime.GOOS == "windows" {
t.Skipf("symlinks unavailable: %v", err)
}
t.Fatal(err)
}
}
// outsidePaths name the key file from the run directory: by absolute path, by
// "..", through symlinks placed in the run directory, and through a hard link.
func (f *bbotConfinedFixture) outsidePaths(t *testing.T) []string {
t.Helper()
f.symlink(t, f.secret, "lnabs")
f.symlink(t, filepath.Join("..", "..", "..", "keys", "jwt.key"), "lnrel")
f.symlink(t, filepath.Join("..", "..", "..", "keys"), "lndir")
paths := []string{
f.secret,
filepath.Join("..", "..", "..", "keys", "jwt.key"),
filepath.Join(f.runDir, "..", "..", "..", "keys", "jwt.key"),
"lnabs",
filepath.Join(f.runDir, "lnabs"),
"lnrel",
filepath.Join("lndir", "jwt.key"),
}
if runtime.GOOS != "windows" {
if err := os.Link(f.secret, filepath.Join(f.runDir, "hard")); err != nil {
t.Fatal(err)
}
paths = append(paths, "hard", "/etc/passwd", "/proc/self/environ")
}
return paths
}
// The path gate alone: every path that leads out of the run directory is refused,
// and so is every path when the base or the working directory is missing or wrong.
func TestOpenBbotOutputConfined(t *testing.T) {
f := newBbotConfinedFixture(t)
for _, p := range f.outsidePaths(t) {
if file, err := openBbotOutput(f.base, f.runDir, p); err == nil {
file.Close()
t.Errorf("openBbotOutput(%q) opened a file outside the run directory", p)
}
}
inside := filepath.Join(f.runDir, "output.json")
if err := os.WriteFile(inside, []byte(bbotToolFixture), 0o644); err != nil {
t.Fatal(err)
}
f.symlink(t, "output.json", "lnin")
if file, err := openBbotOutput(f.base, f.runDir, "lnin"); err == nil {
file.Close()
t.Error("a symlink inside the run directory was opened")
}
if file, err := openBbotOutput(f.base, f.runDir, "."); err == nil {
file.Close()
t.Error("the run directory itself was opened")
}
for name, c := range map[string][2]string{
"no base": {"", f.runDir},
"no working directory": {f.base, ""},
"base as working dir": {f.base, f.base},
"working dir not in base": {filepath.Join(f.base, "tasks"), filepath.Join(f.base, "keys")},
} {
if file, err := openBbotOutput(c[0], c[1], "output.json"); err == nil {
file.Close()
t.Errorf("%s: openBbotOutput(%q, %q) opened a file", name, c[0], c[1])
}
}
// A run directory reached by a symlink is refused, as for the file tools.
f.symlink(t, filepath.Join(f.base, "keys"), filepath.Join("..", "i8"))
if file, err := openBbotOutput(f.base, filepath.Join(f.base, "tasks", "1", "i8"), "jwt.key"); err == nil {
file.Close()
t.Error("a run directory that is a symlink was opened")
}
// A benign file in the run directory is still read and parsed, by relative and
// by absolute path.
for _, p := range []string{"output.json", inside} {
file, err := openBbotOutput(f.base, f.runDir, p)
if err != nil {
t.Fatalf("openBbotOutput(%q): %v", p, err)
}
res, oversized, err := readBbotJSONL(file)
file.Close()
if err != nil {
t.Fatalf("read %q: %v", p, err)
}
if len(res.assetItems()) != 1 || len(res.InScope) != 2 || len(res.Intel) != 3 || res.Skipped != 1 || oversized != 0 {
t.Fatalf("%q: %d assets, %d in scope, %d intel, %d skipped, %d oversized; want 1, 2, 3, 1, 0",
p, len(res.assetItems()), len(res.InScope), len(res.Intel), res.Skipped, oversized)
}
}
}
// The tool itself, without Postgres: a refused path returns before a store is
// used, and the stores here are empty shells that cannot answer a query. An
// in-scope event that is not an asset writes nothing, so a file inside the run
// directory goes through the whole handler.
func TestIngestBbotToolConfined(t *testing.T) {
f := newBbotConfinedFixture(t)
tsx := NewToolSet(&db.ExplorationStore{}, "worker-1")
tsx.SetAssetStore(&db.AssetStore{}, nil)
tsx.SetWorkBase(f.base)
tool := tsx.ingestBbot()
call := func(tc *actool.ToolContext, path string) actool.Result {
t.Helper()
in, _ := json.Marshal(map[string]string{"path": path})
res, err := tool.Call(context.Background(), in, tc)
if err != nil {
t.Fatalf("ingest_bbot(%q): %v", path, err)
}
return res
}
tc := &actool.ToolContext{WorkingDir: f.runDir}
for _, p := range f.outsidePaths(t) {
res := call(tc, p)
if !res.IsError || !strings.HasPrefix(res.Flatten(), "open: ") {
t.Errorf("ingest_bbot(%q) = %q, want an open error", p, res.Flatten())
}
if strings.Contains(res.Flatten(), bbotConfinedSecret) {
t.Errorf("ingest_bbot(%q) returned the key file content: %q", p, res.Flatten())
}
}
inside := filepath.Join(f.runDir, "output.json")
if err := os.WriteFile(inside, []byte(`{"type":"FINDING","host":"in.example.com","scope_distance":0}`+"\nnot json\n"), 0o644); err != nil {
t.Fatal(err)
}
// No working directory (a nil ToolContext) and no base both fail closed.
if res := call(nil, inside); !res.IsError || !strings.HasPrefix(res.Flatten(), "open: ") {
t.Errorf("without a working directory: %q, want an open error", res.Flatten())
}
noBase := NewToolSet(&db.ExplorationStore{}, "worker-1")
noBase.SetAssetStore(&db.AssetStore{}, nil)
in, _ := json.Marshal(map[string]string{"path": inside})
if res, err := noBase.ingestBbot().Call(context.Background(), in, tc); err != nil || !res.IsError || !strings.HasPrefix(res.Flatten(), "open: ") {
t.Errorf("without a base: %q %v, want an open error", res.Flatten(), err)
}
for _, p := range []string{"output.json", inside} {
res := call(tc, p)
if res.IsError {
t.Fatalf("ingest_bbot(%q): %s", p, res.Flatten())
}
var out struct {
InScopeNotAsset int `json:"in_scope_not_asset"`
SkippedLines int `json:"skipped_lines"`
IntelFacts int `json:"intel_facts"`
}
if err := json.Unmarshal([]byte(res.Flatten()), &out); err != nil {
t.Fatalf("result %q: %v", res.Flatten(), err)
}
if out.InScopeNotAsset != 1 || out.SkippedLines != 1 || out.IntelFacts != 0 {
t.Fatalf("ingest_bbot(%q) = %s; want 1 in-scope non-asset, 1 skipped line", p, res.Flatten())
}
}
}
+110 -56
View File
@@ -4,12 +4,14 @@ import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"strconv"
"strings"
acperm "github.com/Autumn-27/norma/permission"
actool "github.com/Autumn-27/norma/tool"
)
@@ -49,68 +51,120 @@ func bbotIntelFact(ev bbotEvent) factItem {
return factItem{Summary: summary, Detail: ev.Quoted, Source: factSourceThirdPartyIntel}
}
// openBbotOutput opens the bbot output file p through the run's working directory
// dir under base, the confinement of the file tools (filetools.go). The server
// reads the file as root while the agent names the path, so an absolute path
// outside the working directory, "..", a symlink and a hard-linked file are
// refused. Without a data directory or a working directory it fails closed.
func openBbotOutput(base, dir, p string) (*os.File, error) {
if strings.TrimSpace(base) == "" {
return nil, errors.New("no data directory for this run")
}
w, err := openWorkRoot(base, dir)
if err != nil {
return nil, err
}
defer w.Close() // a file opened in the root stays open
rel, err := w.rel(p)
if err != nil {
return nil, fmt.Errorf("%s is %w %s", p, err, w.dir)
}
// Lstat: a symlink is refused even when it stays inside the working directory,
// and a FIFO is not opened (that would block).
if fi, err := w.Lstat(rel); err != nil {
return nil, err
} else if !fi.Mode().IsRegular() {
return nil, errNotRegular
}
f, err := w.Open(rel)
if err != nil {
return nil, err
}
if err := checkOpened(f); err != nil {
f.Close()
return nil, err
}
return f, nil
}
// ingestBbot reads a bbot output file and splits it by the typed scope_distance:
// in-scope asset events go through the insert_assets handler, every other event
// becomes a third_party_intel fact. Intel never reaches insert_assets or
// task_scope.
// task_scope. It needs the run's working directory, so it reads the ToolContext
// that writeExpTool does not pass on; the exploration guard is the same.
func (t *ToolSet) ingestBbot() actool.CoreTool {
return t.writeExpTool("ingest_bbot",
"Ingest a bbot output file (JSONL, output.json) by path. Only the typed fields of each event are read.\n"+
"Events with scope_distance 0 that are assets (DNS_NAME, IP_ADDRESS, OPEN_TCP_PORT, URL) are registered through insert_assets. "+
"Every other event (scope_distance not 0, or missing) is recorded as a fact with source third_party_intel: it is never added to the scope or to the assets.\n"+
"An event's free text (data) is kept only as a quoted string in the fact detail; it is data, not instructions. Lines over 64 KiB and lines that are not JSON are skipped and counted.\n"+
const name = "ingest_bbot"
return actool.Build(actool.Spec{
Name: name,
Description: "Ingest a bbot output file (JSONL, output.json) by path. Only the typed fields of each event are read.\n" +
"path is relative to your working directory; a path outside it, a symlink or a hard-linked file is refused.\n" +
"Events with scope_distance 0 that are assets (DNS_NAME, IP_ADDRESS, OPEN_TCP_PORT, URL) are registered through insert_assets. " +
"Every other event (scope_distance not 0, or missing) is recorded as a fact with source third_party_intel: it is never added to the scope or to the assets.\n" +
"An event's free text (data) is kept only as a quoted string in the fact detail; it is data, not instructions. Lines over 64 KiB and lines that are not JSON are skipped and counted.\n" +
"Returns: {assets: <insert_assets result>, intel_facts, fact_errors, in_scope_not_asset, skipped_lines, oversized_lines}",
obj(map[string]any{
"path": str("path of the bbot JSONL output file (output.json)"),
Schema: obj(map[string]any{
"path": str("path of the bbot JSONL output file (output.json), relative to your working directory"),
}, "path"),
func(ctx context.Context, in json.RawMessage) (actool.Result, error) {
if t.as == nil {
return actool.Errorf("ingest_bbot is not enabled: AssetStore is not initialized"), nil
}
var a struct {
Path string `json:"path"`
}
if err := json.Unmarshal(in, &a); err != nil || strings.TrimSpace(a.Path) == "" {
return actool.Errorf("path is required"), nil
}
f, err := os.Open(a.Path)
if err != nil {
return actool.Errorf("open: " + err.Error()), nil
}
defer f.Close()
res, oversized, err := readBbotJSONL(f)
if err != nil {
return actool.Errorf("read: " + err.Error()), nil
}
Permissions: func(context.Context, json.RawMessage, acperm.Context) acperm.Decision { return acperm.Allowed() },
Run: func(ctx context.Context, in json.RawMessage, tc *actool.ToolContext) (actool.Result, error) {
return t.needExploration(name, func(ctx context.Context, in json.RawMessage) (actool.Result, error) {
return t.runIngestBbot(ctx, in, tc)
})(ctx, in)
},
})
}
items := res.assetItems()
out := map[string]any{
"in_scope_not_asset": len(res.InScope) - len(items),
"skipped_lines": res.Skipped,
"oversized_lines": oversized,
}
if len(items) > 0 {
raw, _ := json.Marshal(map[string]any{"assets": items})
r, err := t.insertAssets().Call(ctx, raw, nil)
if err != nil || r.IsError {
return r, err
}
out["assets"] = json.RawMessage(r.Flatten())
}
func (t *ToolSet) runIngestBbot(ctx context.Context, in json.RawMessage, tc *actool.ToolContext) (actool.Result, error) {
if t.as == nil {
return actool.Errorf("ingest_bbot is not enabled: AssetStore is not initialized"), nil
}
var a struct {
Path string `json:"path"`
}
if err := json.Unmarshal(in, &a); err != nil || strings.TrimSpace(a.Path) == "" {
return actool.Errorf("path is required"), nil
}
dir := ""
if tc != nil {
dir = tc.WorkingDir
}
f, err := openBbotOutput(t.workBase, dir, a.Path)
if err != nil {
return actool.Errorf("open: " + err.Error()), nil
}
defer f.Close()
res, oversized, err := readBbotJSONL(f)
if err != nil {
return actool.Errorf("read: " + err.Error()), nil
}
facts := 0
errs := map[string]string{}
for i, ev := range res.Intel {
if _, err := t.recordOneFact(bbotIntelFact(ev), t.ownerNode); err != nil {
errs[strconv.Itoa(i)] = err.Error()
continue
}
facts++
}
out["intel_facts"] = facts
if len(errs) > 0 {
out["fact_errors"] = errs
}
return jsonResult(out)
})
items := res.assetItems()
out := map[string]any{
"in_scope_not_asset": len(res.InScope) - len(items),
"skipped_lines": res.Skipped,
"oversized_lines": oversized,
}
if len(items) > 0 {
raw, _ := json.Marshal(map[string]any{"assets": items})
r, err := t.insertAssets().Call(ctx, raw, nil)
if err != nil || r.IsError {
return r, err
}
out["assets"] = json.RawMessage(r.Flatten())
}
facts := 0
errs := map[string]string{}
for i, ev := range res.Intel {
if _, err := t.recordOneFact(bbotIntelFact(ev), t.ownerNode); err != nil {
errs[strconv.Itoa(i)] = err.Error()
continue
}
facts++
}
out["intel_facts"] = facts
if len(errs) > 0 {
out["fact_errors"] = errs
}
return jsonResult(out)
}
+14 -2
View File
@@ -1,6 +1,7 @@
package agent
import (
"context"
"encoding/json"
"os"
"path/filepath"
@@ -9,6 +10,7 @@ import (
"testing"
"github.com/Autumn-27/artex/db"
actool "github.com/Autumn-27/norma/tool"
)
// A benign bbot output: one in-scope host, one out-of-scope host, one event
@@ -41,12 +43,22 @@ func TestIngestBbotScopeSplit(t *testing.T) {
tsx.SetTaskID(task.ID)
tsx.SetAssetStore(d.Assets(), d.Companies())
path := filepath.Join(t.TempDir(), "output.json")
// The file is read through the run's working directory under the data directory.
base := t.TempDir()
runDir := filepath.Join(base, "tasks", "1", "i1")
if err := os.MkdirAll(runDir, 0o755); err != nil {
t.Fatal(err)
}
tsx.SetWorkBase(base)
path := filepath.Join(runDir, "output.json")
if err := os.WriteFile(path, []byte(bbotToolFixture), 0o600); err != nil {
t.Fatal(err)
}
input, _ := json.Marshal(map[string]string{"path": path})
res := callGuardTool(t, tsx.ingestBbot(), string(input))
res, err := tsx.ingestBbot().Call(context.Background(), input, &actool.ToolContext{WorkingDir: runDir})
if err != nil {
t.Fatal(err)
}
if res.IsError {
t.Fatalf("ingest_bbot: %s", res.Flatten())
}
+9 -1
View File
@@ -26,7 +26,15 @@ func callReadJSON(t *testing.T, tool actool.CoreTool, input string) any {
}
func TestGraphOverviewExpandsAssociatedCompanyScope(t *testing.T) {
t.Skip("known failure with a fresh database, pre-existing on unmodified upstream; tracked in BaaRai/ARTEX#9")
// Known failure with a freshly seeded database (BaaRai/ARTEX#9). Repro:
// ARTEX_PG_DSN=postgres://artex:testpass@localhost:5432/artextest_agent \
// go test ./agent -run TestGraphOverviewExpandsAssociatedCompanyScope -count=1 -v
// On a fresh DB the company-scope expansion in graphOverviewData is not
// visible to the task: the t.Fatalf checks after QueryByTask and
// graphOverviewData (company asset linkage, company_scope rows and
// company_keywords) fail. Keep the skip until seeding is ordered correctly;
// do not delete it.
t.Skip("known failure with a fresh database; repro and failing assertions documented above; BaaRai/ARTEX#9")
d := testDB(t)
defer d.Close()
+9 -2
View File
@@ -17,10 +17,17 @@ import (
// Exercise the actual SDK event -> hook -> execution -> result path. No real
// model or command is used; the probe tool only returns a fixed string.
func TestCaptureApprovalLifecycle(t *testing.T) {
t.Skip("known failure with a fresh database, pre-existing on unmodified upstream; tracked in BaaRai/ARTEX#9")
// Known failure with a freshly seeded database (BaaRai/ARTEX#9). Repro:
// ARTEX_PG_DSN=postgres://artex:testpass@localhost:5432/artextest_agent \
// go test ./agent -run TestCaptureApprovalLifecycle -count=1 -v
// The checks after ListTaskIntercepts / GetInterceptDetail and the
// "unexpected review" t.Fatalf below fail before any other test has seeded
// intercept state. Keep this skipped until the fresh-DB setup is fixed;
// do not simply delete the skip.
t.Skip("known failure with a fresh database; repro and failing assertion documented above; BaaRai/ARTEX#9")
dsn, _, err := db.DSN()
if err != nil {
t.Skip("no test database configured")
db.SkipPGUnavailable(t, err)
}
d, err := db.Open(dsn)
if err != nil {
+7 -4
View File
@@ -152,13 +152,14 @@ func (c *ChatAgent) Chat(ctx context.Context, agentKey, sessionID, message strin
if !sessionDirInside(c.workDir, sessionWorkDir) {
return "", fmt.Errorf("invalid session id %q", sessionID)
}
_ = os.MkdirAll(sessionWorkDir, 0o755)
// A run directory like the task ones, created the same way (makeRunDir).
makeRunDir(c.workDir, sessionWorkDir)
ctx = intercept.WithReviewWorkingDirectory(ctx, sessionWorkDir)
// Pure assistant: DefaultTools as the base; AugmentTools layers in the key's
// visible skills/MCP and lets the DB tools table filter/override. DefaultTools
// have no tools-table rows, so they always pass through.
base := actool.DefaultTools()
base := defaultTools(c.workDir) // file tools confined to sessionWorkDir (filetools.go)
ctx = WithRunInfo(ctx, RunInfo{SessionID: sessionID})
tools, def, cleanup := AugmentTools(ctx, agentKey, base)
defer cleanup()
@@ -190,9 +191,10 @@ func (c *ChatAgent) Chat(ctx context.Context, agentKey, sessionID, message strin
MaxDuration: maxDuration,
Compaction: compactionConfig(c.window),
Todos: actool.NewTodoStore(),
// large tool output spills to cmd-output/ under the session dir.
// large tool output spills to cmd-output/ under the session dir, or to the
// session's spill directory (spill.go).
// the truncation cap uses the SDK default (30000 chars of tool.Capture).
ToolOutputDir: filepath.Join(sessionWorkDir, "cmd-output"),
ToolOutputDir: SessionToolOutputDir(c.workDir, sessionID, sessionWorkDir),
// hitting the budget (steps) → the SDK runs wrap-up: output a one-line summary. The prompt and
// wrap-up rounds are editable in the backend per agent key
// (each custom agent gets its own copy; empty/0 uses the generic default: 10 rounds).
@@ -206,6 +208,7 @@ func (c *ChatAgent) Chat(ctx context.Context, agentKey, sessionID, message strin
if c.guard != nil {
opts.Hooks = c.guard.Hooks()
}
opts.Hooks = SpillHooks(opts.Hooks, opts.ToolOutputDir)
// The session id is the provider-side identity (session header, usage metering)
// with or without persistence; Norma only puts it on ctx when it writes a transcript.
opts.SessionID = sessionID
+862
View File
@@ -0,0 +1,862 @@
package agent
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path"
"path/filepath"
"regexp"
"sort"
"strings"
actool "github.com/Autumn-27/norma/tool"
)
// Confined file tools. The SDK's Read/Write/Edit/MultiEdit/LS/Glob/Grep run inside
// the server process, which is root in the Docker image, and accept any absolute or
// relative path: they would read /app/keys or the config file, and write a skill or
// the config, while the agent's shell runs as the unprivileged executor. These
// replacements keep the SDK tools' names, descriptions and input schemas (the model
// sees no change) and open every path through an os.Root on the run's working
// directory. os.Root resolves each path component inside the root at the syscall
// level, so an absolute path outside, "..", or a symlink leading out of the working
// directory is refused, including a symlink the agent's shell swaps in while a call
// runs. Grep does not start ripgrep: that would read the files with root's rights by
// path.
// confinedFileToolNames are the SDK default tools replaced by defaultTools.
var confinedFileToolNames = map[string]func(*workRoot, json.RawMessage, *actool.ToolContext) (actool.Result, error){
"Read": runConfinedRead,
"Write": runConfinedWrite,
"Edit": runConfinedEdit,
"MultiEdit": runConfinedMultiEdit,
"LS": runConfinedLS,
"Glob": runConfinedGlob,
"Grep": runConfinedGrep,
}
// defaultTools is actool.DefaultTools() with the file tools confined to the run's
// working directory under base (the data directory the agent's run directories
// live in). Every agent kind builds its tool set from it.
func defaultTools(base string) []actool.CoreTool {
all := actool.DefaultTools()
out := make([]actool.CoreTool, 0, len(all))
for _, t := range all {
if run, ok := confinedFileToolNames[t.Name()]; ok {
t = &confinedFileTool{CoreTool: t, base: base, run: run}
}
out = append(out, t)
}
return out
}
// confinedFileTool keeps the SDK tool's metadata and permission self-check and
// replaces only Call.
type confinedFileTool struct {
actool.CoreTool
base string
run func(*workRoot, json.RawMessage, *actool.ToolContext) (actool.Result, error)
}
func (t *confinedFileTool) Call(_ context.Context, in json.RawMessage, tc *actool.ToolContext) (actool.Result, error) {
dir := ""
if tc != nil {
dir = tc.WorkingDir
}
w, err := openWorkRoot(t.base, dir)
if err != nil {
return actool.Errorf(fmt.Sprintf("Error: %s cannot run: %v", t.Name(), err)), nil
}
defer w.Close()
return t.run(w, in, tc)
}
// workRoot is the run's working directory opened as an os.Root. dir is the path the
// model knows it by: absolute paths inside it are accepted and outputs use it.
type workRoot struct {
*os.Root
dir string
}
var (
errOutsideWorkDir = errors.New("outside the working directory")
errNotRegular = errors.New("not a regular file")
errHardLinked = errors.New("the file has other hard links, which can lead outside the working directory")
)
// openWorkRoot opens dir, which must lie strictly below base. base is set by the
// server and resolved as is; below it every component is opened inside its parent
// (os.Root.OpenRoot), so a run directory the agent's shell replaced with a symlink
// cannot move the root out of the parent workspace tree (to / or to the data
// directory, for instance). Fails closed on any error.
func openWorkRoot(base, dir string) (*workRoot, error) {
if strings.TrimSpace(dir) == "" {
return nil, errors.New("no working directory for this run")
}
absDir, err := filepath.Abs(dir)
if err != nil {
return nil, err
}
absBase, err := filepath.Abs(base)
if err != nil {
return nil, err
}
rel, err := filepath.Rel(absBase, absDir)
if err != nil || rel == "." || !filepath.IsLocal(rel) {
return nil, errors.New("the working directory is not inside the data directory")
}
realBase, err := filepath.EvalSymlinks(absBase)
if err != nil {
return nil, fmt.Errorf("data directory: %w", err)
}
// A relative symlink inside the data directory passes os.Root (it stays in the
// root), so the path is also required to be free of symlinks: the agent's shell
// can replace a run directory with a link to another run's tree or to a
// root-only file such as the MITM CA key.
want := filepath.Join(realBase, rel)
if got, err := filepath.EvalSymlinks(want); err != nil || got != want {
return nil, errors.New("the working directory path contains a symlink")
}
r, err := os.OpenRoot(realBase)
if err != nil {
return nil, fmt.Errorf("data directory: %w", err)
}
for _, c := range strings.Split(rel, string(filepath.Separator)) {
next, err := r.OpenRoot(c)
r.Close()
if err != nil {
return nil, fmt.Errorf("open the working directory: %w", err)
}
r = next
}
return &workRoot{Root: r, dir: absDir}, nil
}
// rel maps a model-supplied path to a path relative to the root. Empty means the
// working directory itself. An absolute path must lie inside the working directory;
// ".." leading out of it is refused here, and os.Root refuses any symlink that does.
func (w *workRoot) rel(p string) (string, error) {
if strings.ContainsRune(p, 0) {
return "", errOutsideWorkDir
}
if p == "" {
return ".", nil
}
if filepath.IsAbs(p) {
r, err := filepath.Rel(w.dir, filepath.Clean(p))
if err != nil {
return "", errOutsideWorkDir
}
p = r
}
p = filepath.Clean(p)
if !filepath.IsLocal(p) {
return "", errOutsideWorkDir
}
return p, nil
}
// refuse answers a path outside the working directory.
func (w *workRoot) refuse(p string) actool.Result {
return actool.Errorf(fmt.Sprintf("Error: %s is outside the working directory %s. File tools only reach paths inside it (a symlink that leads out of it is refused too). To read a file elsewhere that your shell may access, use Bash.", p, w.dir))
}
// readFile reads a regular file inside the root. A file with more than one hard
// link is refused: a link made in the working directory to a file outside it would
// otherwise be read with the server's rights.
func (w *workRoot) readFile(rel string) ([]byte, error) {
if fi, err := w.Stat(rel); err != nil {
return nil, err
} else if !fi.Mode().IsRegular() {
return nil, errNotRegular
}
f, err := w.Open(rel)
if err != nil {
return nil, err
}
defer f.Close()
if err := checkOpened(f); err != nil {
return nil, err
}
return io.ReadAll(f)
}
// writeFile replaces the contents of rel, creating it when missing. The checks run
// on the opened file before it is truncated.
func (w *workRoot) writeFile(rel string, data []byte) error {
if fi, err := w.Stat(rel); err == nil && !fi.Mode().IsRegular() {
return errNotRegular
}
// 0664: the executor group shares the workspace tree (server/execpriv.go).
f, err := w.OpenFile(rel, os.O_WRONLY|os.O_CREATE, 0o664)
if err != nil {
return err
}
if err := checkOpened(f); err != nil {
f.Close()
return err
}
if err := f.Truncate(0); err != nil {
f.Close()
return err
}
if _, err := f.Write(data); err != nil {
f.Close()
return err
}
return f.Close()
}
func checkOpened(f *os.File) error {
fi, err := f.Stat()
switch {
case err != nil:
return err
case !fi.Mode().IsRegular():
return errNotRegular
case extraHardLinks(fi):
return errHardLinked
}
return nil
}
// ---------- Read ----------
const confinedMaxLineLen = 2000
// confinedReadOutputBonus mirrors the SDK Read tool's extra output budget.
const confinedReadOutputBonus = 1000
func runConfinedRead(w *workRoot, input json.RawMessage, tc *actool.ToolContext) (actool.Result, error) {
var in struct {
FilePath string `json:"file_path"`
Offset int `json:"offset"`
Limit int `json:"limit"`
}
if err := json.Unmarshal(input, &in); err != nil {
return actool.Result{}, err
}
rel, err := w.rel(in.FilePath)
if err != nil {
return w.refuse(in.FilePath), nil
}
info, err := w.Stat(rel)
if err != nil {
return actool.Errorf(fmt.Sprintf("Error reading %s: %v", in.FilePath, err)), nil
}
if info.IsDir() {
return actool.Errorf(fmt.Sprintf("Error: %s is a directory, not a file. Use LS or Glob to list its contents.", in.FilePath)), nil
}
data, err := w.readFile(rel)
if err != nil {
return actool.Errorf(fmt.Sprintf("Error reading %s: %v", in.FilePath, err)), nil
}
if isBinaryData(data) {
return actool.Text(fmt.Sprintf("[binary file %s not shown — %d bytes]", in.FilePath, len(data))), nil
}
lines := strings.Split(string(data), "\n")
start := 0
if in.Offset > 0 {
start = in.Offset - 1
}
if start > len(lines) {
start = len(lines)
}
limit := in.Limit
if limit <= 0 {
limit = 2000
}
end := min(start+limit, len(lines))
var b strings.Builder
for i := start; i < end; i++ {
line := lines[i]
if len(line) > confinedMaxLineLen {
line = line[:confinedMaxLineLen] + "… [line truncated]"
}
fmt.Fprintf(&b, "%6d\t%s\n", i+1, line)
}
if b.Len() == 0 {
if len(data) == 0 {
return actool.Text("(file is empty)"), nil
}
return actool.Text("(no lines in the requested range)"), nil
}
return actool.Text(confinedReadCapture(tc, b.String(), start)), nil
}
// confinedReadCapture bounds Read's output like the SDK tool: it cuts at a line
// boundary and tells the model the offset to continue from.
func confinedReadCapture(tc *actool.ToolContext, s string, start int) string {
limit := actool.MaxOutput(tc) + confinedReadOutputBonus
if len(s) <= limit {
return s
}
head := s[:limit]
if cut := strings.LastIndexByte(head, '\n'); cut >= 0 {
head = s[:cut+1]
}
next := start + strings.Count(head, "\n") + 1
return head + fmt.Sprintf(
"\n... [Read truncated to fit the output limit. Continue reading the file with Read offset=%d (and an optional limit), or narrow with Grep/`sed -n`.] ...", next)
}
// isBinaryData reports a NUL byte in the first chunk, the SDK's heuristic.
func isBinaryData(data []byte) bool {
return bytes.IndexByte(data[:min(len(data), 8000)], 0) >= 0
}
// ---------- Write / Edit / MultiEdit ----------
func runConfinedWrite(w *workRoot, input json.RawMessage, _ *actool.ToolContext) (actool.Result, error) {
var in struct {
FilePath string `json:"file_path"`
Content string `json:"content"`
}
if err := json.Unmarshal(input, &in); err != nil {
return actool.Result{}, err
}
rel, err := w.rel(in.FilePath)
if err != nil {
return w.refuse(in.FilePath), nil
}
existed := false
if info, err := w.Stat(rel); err == nil {
if info.IsDir() {
return actool.Errorf(fmt.Sprintf("Error: %s is an existing directory, not a file.", in.FilePath)), nil
}
existed = true
}
if dir := filepath.Dir(rel); dir != "." {
// 0775: the executor group shares the workspace tree (server/execpriv.go).
if err := w.MkdirAll(dir, 0o775); err != nil {
return actool.Errorf(fmt.Sprintf("Error creating directory: %v", err)), nil
}
}
if err := w.writeFile(rel, []byte(in.Content)); err != nil {
return actool.Errorf(fmt.Sprintf("Error writing %s: %v", in.FilePath, err)), nil
}
verb := "Created"
if existed {
verb = "Overwrote"
}
return actool.Text(fmt.Sprintf("%s %s (%d bytes)", verb, in.FilePath, len(in.Content))), nil
}
func runConfinedEdit(w *workRoot, input json.RawMessage, _ *actool.ToolContext) (actool.Result, error) {
var in struct {
FilePath string `json:"file_path"`
OldString string `json:"old_string"`
NewString string `json:"new_string"`
ReplaceAll bool `json:"replace_all"`
}
if err := json.Unmarshal(input, &in); err != nil {
return actool.Result{}, err
}
if in.OldString == in.NewString {
return actool.Errorf("Error: old_string and new_string are identical"), nil
}
rel, err := w.rel(in.FilePath)
if err != nil {
return w.refuse(in.FilePath), nil
}
data, err := w.readFile(rel)
if err != nil {
return actool.Errorf(fmt.Sprintf("Error reading %s: %v", in.FilePath, err)), nil
}
content := string(data)
count := strings.Count(content, in.OldString)
if count == 0 {
return actool.Errorf(fmt.Sprintf("Error: old_string not found in %s", in.FilePath)), nil
}
if count > 1 && !in.ReplaceAll {
return actool.Errorf(fmt.Sprintf("Error: old_string is not unique in %s (%d matches); add context or set replace_all", in.FilePath, count)), nil
}
var updated string
if in.ReplaceAll {
updated = strings.ReplaceAll(content, in.OldString, in.NewString)
} else {
updated = strings.Replace(content, in.OldString, in.NewString, 1)
}
if err := w.writeFile(rel, []byte(updated)); err != nil {
return actool.Errorf(fmt.Sprintf("Error writing %s: %v", in.FilePath, err)), nil
}
return actool.Text(fmt.Sprintf("Edited %s (%d replacement(s))", in.FilePath, count)), nil
}
func runConfinedMultiEdit(w *workRoot, input json.RawMessage, _ *actool.ToolContext) (actool.Result, error) {
var in struct {
FilePath string `json:"file_path"`
Edits []struct {
OldString string `json:"old_string"`
NewString string `json:"new_string"`
ReplaceAll bool `json:"replace_all"`
} `json:"edits"`
}
if err := json.Unmarshal(input, &in); err != nil {
return actool.Result{}, err
}
if len(in.Edits) == 0 {
return actool.Errorf("Error: no edits provided"), nil
}
rel, err := w.rel(in.FilePath)
if err != nil {
return w.refuse(in.FilePath), nil
}
data, err := w.readFile(rel)
if err != nil {
return actool.Errorf(fmt.Sprintf("Error reading %s: %v", in.FilePath, err)), nil
}
content := string(data)
total := 0
for i, e := range in.Edits {
if e.OldString == e.NewString {
return actool.Errorf(fmt.Sprintf("Error in edit %d: old_string and new_string are identical", i+1)), nil
}
count := strings.Count(content, e.OldString)
if count == 0 {
return actool.Errorf(fmt.Sprintf("Error in edit %d: old_string not found", i+1)), nil
}
if count > 1 && !e.ReplaceAll {
return actool.Errorf(fmt.Sprintf("Error in edit %d: old_string is not unique (%d matches); add context or set replace_all", i+1, count)), nil
}
if e.ReplaceAll {
content = strings.ReplaceAll(content, e.OldString, e.NewString)
} else {
content = strings.Replace(content, e.OldString, e.NewString, 1)
}
total += count
}
if err := w.writeFile(rel, []byte(content)); err != nil {
return actool.Errorf(fmt.Sprintf("Error writing %s: %v", in.FilePath, err)), nil
}
return actool.Text(fmt.Sprintf("Applied %d edits to %s (%d replacement(s))", len(in.Edits), in.FilePath, total)), nil
}
// ---------- LS ----------
func runConfinedLS(w *workRoot, input json.RawMessage, tc *actool.ToolContext) (actool.Result, error) {
var in struct {
Path string `json:"path"`
}
if err := json.Unmarshal(input, &in); err != nil {
return actool.Result{}, err
}
rel, err := w.rel(in.Path)
if err != nil {
return w.refuse(in.Path), nil
}
// Stat first: opening a FIFO to list it would block.
if info, err := w.Stat(rel); err == nil && !info.IsDir() {
return actool.Errorf(fmt.Sprintf("Error listing %s: not a directory", in.Path)), nil
}
entries, err := fs.ReadDir(w.FS(), filepath.ToSlash(rel))
if err != nil {
return actool.Errorf(fmt.Sprintf("Error listing %s: %v", in.Path, err)), nil
}
if len(entries) == 0 {
return actool.Text("(empty directory)"), nil
}
sort.Slice(entries, func(i, j int) bool {
if entries[i].IsDir() != entries[j].IsDir() {
return entries[i].IsDir() // directories first
}
return entries[i].Name() < entries[j].Name()
})
var b strings.Builder
for _, e := range entries {
if e.IsDir() {
fmt.Fprintf(&b, "%s/\n", e.Name())
continue
}
var size int64
if info, err := w.Lstat(filepath.Join(rel, e.Name())); err == nil {
size = info.Size()
}
fmt.Fprintf(&b, "%s\t%d bytes\n", e.Name(), size)
}
return actool.Text(actool.Capture(tc, b.String())), nil
}
// ---------- Glob ----------
var confinedSkipDirs = map[string]bool{".git": true, "node_modules": true, "vendor": true}
// relUnder returns p relative to the walk base (both slash paths from fs.WalkDir).
func relUnder(base, p string) string {
if base == "." {
return p
}
if p == base {
return "."
}
return strings.TrimPrefix(p, base+"/")
}
func runConfinedGlob(w *workRoot, input json.RawMessage, tc *actool.ToolContext) (actool.Result, error) {
var in struct {
Pattern string `json:"pattern"`
Path string `json:"path"`
}
if err := json.Unmarshal(input, &in); err != nil {
return actool.Result{}, err
}
rel, err := w.rel(in.Path)
if err != nil {
return w.refuse(in.Path), nil
}
base := filepath.ToSlash(rel)
type match struct {
path string
mod int64
}
var matches []match
_ = fs.WalkDir(w.FS(), base, func(p string, d fs.DirEntry, err error) error {
if err != nil {
return nil
}
if d.IsDir() {
if confinedSkipDirs[d.Name()] {
return fs.SkipDir
}
return nil
}
if globMatch(in.Pattern, relUnder(base, p)) {
var mod int64
if info, e := w.Lstat(filepath.FromSlash(p)); e == nil {
mod = info.ModTime().UnixNano()
}
matches = append(matches, match{filepath.Join(w.dir, filepath.FromSlash(p)), mod})
}
return nil
})
sort.Slice(matches, func(i, j int) bool { return matches[i].mod > matches[j].mod })
if len(matches) == 0 {
return actool.Text("No files found"), nil
}
var b strings.Builder
for _, m := range matches {
b.WriteString(m.path)
b.WriteByte('\n')
}
return actool.Text(actool.Capture(tc, b.String())), nil
}
// globMatch is the SDK Glob matcher: a pattern without "/" matches the base name,
// "**" matches any number of segments.
func globMatch(pattern, p string) bool {
pattern = filepath.ToSlash(pattern)
p = filepath.ToSlash(p)
if !strings.Contains(pattern, "/") {
if ok, _ := path.Match(pattern, path.Base(p)); ok {
return true
}
}
return globSegs(strings.Split(pattern, "/"), strings.Split(p, "/"))
}
func globSegs(pat, name []string) bool {
if len(pat) == 0 {
return len(name) == 0
}
if pat[0] == "**" {
for i := 0; i <= len(name); i++ {
if globSegs(pat[1:], name[i:]) {
return true
}
}
return false
}
if len(name) == 0 {
return false
}
if ok, _ := path.Match(pat[0], name[0]); !ok {
return false
}
return globSegs(pat[1:], name[1:])
}
// ---------- Grep ----------
func runConfinedGrep(w *workRoot, input json.RawMessage, tc *actool.ToolContext) (actool.Result, error) {
var in struct {
Pattern string `json:"pattern"`
Path string `json:"path"`
Glob string `json:"glob"`
Type string `json:"type"`
OutputMode string `json:"output_mode"`
IgnoreCase bool `json:"-i"`
LineNums *bool `json:"-n"`
After int `json:"-A"`
Before int `json:"-B"`
Context int `json:"-C"`
Multiline bool `json:"multiline"`
HeadLimit *int `json:"head_limit"`
Offset int `json:"offset"`
}
if err := json.Unmarshal(input, &in); err != nil {
return actool.Result{}, err
}
before, after := in.Before, in.After
if in.Context > 0 {
before, after = in.Context, in.Context
}
flags := ""
if in.IgnoreCase {
flags += "i"
}
if in.Multiline {
flags += "s"
}
pat := in.Pattern
if flags != "" {
pat = "(?" + flags + ")" + pat
}
re, err := regexp.Compile(pat)
if err != nil {
return actool.Errorf(fmt.Sprintf("Error: invalid regex: %v", err)), nil
}
rel, err := w.rel(in.Path)
if err != nil {
return w.refuse(in.Path), nil
}
mode := in.OutputMode
if mode == "" {
mode = "files_with_matches"
}
showLines := true
if in.LineNums != nil {
showLines = *in.LineNums
}
typeGlobs := grepTypeGlobs(in.Type)
var results []grepFileHit
scan := func(p, shown string) {
data, err := w.readFile(filepath.FromSlash(p))
if err != nil || isBinaryData(data) {
return
}
if h := grepScan(data, filepath.FromSlash(shown), re, mode, showLines, in.Multiline, before, after); h != nil {
results = append(results, *h)
}
}
base := filepath.ToSlash(rel)
info, err := w.Stat(rel)
if err != nil {
return actool.Errorf(fmt.Sprintf("Error: %v", err)), nil
}
if info.IsDir() {
_ = fs.WalkDir(w.FS(), base, func(p string, d fs.DirEntry, err error) error {
if err != nil {
return nil
}
if d.IsDir() {
if confinedSkipDirs[d.Name()] {
return fs.SkipDir
}
return nil
}
name := path.Base(p)
if in.Glob != "" {
if ok, _ := path.Match(in.Glob, name); !ok {
return nil
}
}
if len(typeGlobs) > 0 && !grepMatchAny(typeGlobs, name) {
return nil
}
scan(p, relUnder(base, p))
return nil
})
} else {
scan(base, path.Base(base))
}
if len(results) == 0 {
return actool.Text("No matches found"), nil
}
sort.Slice(results, func(i, j int) bool { return results[i].path < results[j].path })
var lines []string
switch mode {
case "content":
for _, r := range results {
lines = append(lines, r.lines...)
}
case "count":
for _, r := range results {
lines = append(lines, fmt.Sprintf("%s: %d", r.path, r.count))
}
default: // files_with_matches
for _, r := range results {
lines = append(lines, r.path)
}
}
headLimit := 250
if in.HeadLimit != nil {
headLimit = *in.HeadLimit
}
kept, more := grepPage(lines, headLimit, in.Offset)
out := strings.Join(kept, "\n")
if len(kept) > 0 {
out += "\n"
}
if more > 0 {
out += fmt.Sprintf("... [%d more result(s) — page with offset=%d, or head_limit=0 for all] ...\n", more, in.Offset+len(kept))
}
return actool.Text(actool.Capture(tc, out)), nil
}
// grepFileTypes maps a ripgrep-style file type to its globs (the SDK's subset).
var grepFileTypes = map[string][]string{
"go": {"*.go"},
"py": {"*.py", "*.pyi"},
"python": {"*.py", "*.pyi"},
"js": {"*.js", "*.jsx", "*.mjs", "*.cjs", "*.vue"},
"ts": {"*.ts", "*.tsx", "*.mts", "*.cts"},
"rust": {"*.rs"},
"java": {"*.java"},
"kotlin": {"*.kt", "*.kts"},
"c": {"*.c", "*.h"},
"cpp": {"*.cpp", "*.cc", "*.cxx", "*.hpp", "*.hh", "*.hxx", "*.h"},
"cs": {"*.cs"},
"rb": {"*.rb"},
"ruby": {"*.rb"},
"php": {"*.php"},
"swift": {"*.swift"},
"sh": {"*.sh", "*.bash", "*.zsh"},
"html": {"*.html", "*.htm"},
"css": {"*.css", "*.scss", "*.sass", "*.less"},
"json": {"*.json"},
"yaml": {"*.yaml", "*.yml"},
"toml": {"*.toml"},
"xml": {"*.xml"},
"md": {"*.md", "*.markdown"},
"markdown": {"*.md", "*.markdown"},
"sql": {"*.sql"},
"proto": {"*.proto"},
}
// grepTypeGlobs resolves a type name to its globs; an unknown type is treated as a
// bare extension (type:foo → *.foo). Empty type → nil.
func grepTypeGlobs(t string) []string {
if t == "" {
return nil
}
if g, ok := grepFileTypes[strings.ToLower(t)]; ok {
return g
}
return []string{"*." + t}
}
func grepMatchAny(globs []string, name string) bool {
for _, g := range globs {
if ok, _ := path.Match(g, name); ok {
return true
}
}
return false
}
// grepPage skips the first offset entries then keeps at most limit (0 = unlimited).
func grepPage(lines []string, limit, offset int) (kept []string, more int) {
if offset > 0 {
if offset >= len(lines) {
return nil, 0
}
lines = lines[offset:]
}
if limit > 0 && len(lines) > limit {
return lines[:limit], len(lines) - limit
}
return lines, 0
}
type grepFileHit struct {
path string
lines []string
count int
}
// grepScan is the SDK Grep's per-file scan over data already read; shown is the
// path printed for the file.
func grepScan(data []byte, shown string, re *regexp.Regexp, mode string, showLines, multiline bool, before, after int) *grepFileHit {
lines := strings.Split(string(data), "\n")
if n := len(lines); n > 0 && lines[n-1] == "" {
lines = lines[:n-1] // drop the empty element from a trailing newline
}
var matches []int
if multiline {
lineStart := make([]int, len(lines))
off := 0
for i, ln := range lines {
lineStart[i] = off
off += len(ln) + 1
}
hit := make([]bool, len(lines))
for _, loc := range re.FindAllIndex(data, -1) {
s, e := loc[0], loc[1]
for i := range lines {
ls, le := lineStart[i], lineStart[i]+len(lines[i])
if s <= le && e > ls {
hit[i] = true
}
}
}
for i, h := range hit {
if h {
matches = append(matches, i)
}
}
} else {
for i, ln := range lines {
if re.MatchString(ln) {
matches = append(matches, i)
}
}
}
if len(matches) == 0 {
return nil
}
res := &grepFileHit{path: shown, count: len(matches)}
if mode != "content" {
return res
}
isMatch := make(map[int]bool, len(matches))
for _, m := range matches {
isMatch[m] = true
}
var order []int
seen := map[int]bool{}
for _, m := range matches {
lo, hi := max(m-before, 0), min(m+after, len(lines)-1)
for j := lo; j <= hi; j++ {
if !seen[j] {
seen[j] = true
order = append(order, j)
}
}
}
sort.Ints(order)
hasContext := before > 0 || after > 0
prev := -2
for _, j := range order {
if hasContext && prev >= 0 && j != prev+1 {
res.lines = append(res.lines, "--")
}
sep := "-"
if isMatch[j] {
sep = ":"
}
if showLines {
res.lines = append(res.lines, fmt.Sprintf("%s%s%d%s%s", shown, sep, j+1, sep, lines[j]))
} else {
res.lines = append(res.lines, fmt.Sprintf("%s%s%s", shown, sep, lines[j]))
}
prev = j
}
return res
}
+9
View File
@@ -0,0 +1,9 @@
//go:build !unix
package agent
import "io/fs"
// extraHardLinks: the link count is not available here; the executor identity that
// could plant a link only exists on unix (server/execpriv_unix.go).
func extraHardLinks(fs.FileInfo) bool { return false }
+347
View File
@@ -0,0 +1,347 @@
package agent
import (
"context"
"encoding/json"
"os"
"path/filepath"
"reflect"
"runtime"
"strings"
"testing"
actool "github.com/Autumn-27/norma/tool"
)
const confinedSecret = "TOPSECRET-confined-tools"
type confinedFixture struct {
base, work, secret string
tools map[string]actool.CoreTool
tc *actool.ToolContext
}
// newConfinedFixture lays out a data directory like the server's: the run's
// working directory under sessions/, and a key file outside it, plus symlinks in
// the working directory that lead to the key (absolute, relative, through a dir).
func newConfinedFixture(t *testing.T) *confinedFixture {
t.Helper()
base := t.TempDir()
work := filepath.Join(base, "sessions", "s1")
keys := filepath.Join(base, "keys")
for _, d := range []string{work, keys} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
secret := filepath.Join(keys, "jwt.key")
if err := os.WriteFile(secret, []byte(confinedSecret+"\n"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(work, "in.txt"), []byte("hello\nworld\n"), 0o644); err != nil {
t.Fatal(err)
}
links := map[string]string{
"lnabs": secret,
"lnrel": filepath.Join("..", "..", "keys", "jwt.key"),
"lndir": filepath.Join("..", "..", "keys"),
"lnin": "in.txt",
}
for name, target := range links {
if err := os.Symlink(target, filepath.Join(work, name)); err != nil {
if runtime.GOOS == "windows" {
t.Skipf("symlinks unavailable: %v", err)
}
t.Fatal(err)
}
}
tools := map[string]actool.CoreTool{}
for _, tl := range defaultTools(base) {
tools[tl.Name()] = tl
}
return &confinedFixture{base: base, work: work, secret: secret, tools: tools, tc: &actool.ToolContext{WorkingDir: work}}
}
func (f *confinedFixture) call(t *testing.T, tool string, in map[string]any) actool.Result {
t.Helper()
raw, _ := json.Marshal(in)
res, err := f.tools[tool].Call(context.Background(), raw, f.tc)
if err != nil {
t.Fatalf("%s(%s): %v", tool, raw, err)
}
return res
}
func (f *confinedFixture) secretUnchanged(t *testing.T) {
t.Helper()
b, err := os.ReadFile(f.secret)
if err != nil || string(b) != confinedSecret+"\n" {
t.Fatalf("key file changed: %q %v", b, err)
}
if _, err := os.Stat(filepath.Join(filepath.Dir(f.secret), "new.txt")); err == nil {
t.Fatal("a file was created next to the key")
}
}
// outsidePaths reach the key file or another file outside the working directory.
func (f *confinedFixture) outsidePaths() []string {
sep := string(filepath.Separator)
return []string{
f.secret,
filepath.Join("..", "..", "keys", "jwt.key"),
f.work + sep + ".." + sep + ".." + sep + "keys" + sep + "jwt.key", // unclean on purpose
"lnabs",
"lnrel",
filepath.Join("lndir", "jwt.key"),
"/proc/self/environ",
"/etc/passwd",
}
}
func refused(res actool.Result) bool { return res.IsError }
// checkRefusals runs every confined tool against every outside path. Nothing may
// succeed, nothing may print the key, and the key must stay unchanged.
func checkRefusals(t *testing.T, f *confinedFixture) {
t.Helper()
for _, p := range f.outsidePaths() {
cases := []struct {
tool string
in map[string]any
}{
{"Read", map[string]any{"file_path": p}},
{"Write", map[string]any{"file_path": p, "content": "pwned"}},
{"Edit", map[string]any{"file_path": p, "old_string": confinedSecret, "new_string": "pwned"}},
{"MultiEdit", map[string]any{"file_path": p, "edits": []any{map[string]any{"old_string": confinedSecret, "new_string": "pwned"}}}},
{"LS", map[string]any{"path": filepath.Dir(p)}},
{"Glob", map[string]any{"pattern": "*", "path": filepath.Dir(p)}},
{"Grep", map[string]any{"pattern": "TOPSECRET", "path": p, "output_mode": "content"}},
{"Grep", map[string]any{"pattern": "TOPSECRET", "path": filepath.Dir(p), "output_mode": "content"}},
}
for _, c := range cases {
res := f.call(t, c.tool, c.in)
out := res.Flatten()
if strings.Contains(out, confinedSecret) {
t.Errorf("%s %v leaked the key: %s", c.tool, c.in, out)
}
// Listing or searching "." (the working directory, the parent of the links)
// is allowed, and Glob skips a symlinked directory without an error;
// everything else must fail.
dir, _ := c.in["path"].(string)
mayPass := dir == "." && (c.tool == "LS" || c.tool == "Glob" || c.tool == "Grep") ||
dir == "lndir" && c.tool == "Glob"
if !refused(res) && !mayPass {
t.Errorf("%s %v was not refused: %s", c.tool, c.in, out)
}
}
f.secretUnchanged(t)
}
// Searching the working directory itself must not follow the links out of it.
for _, in := range []map[string]any{
{"pattern": "TOPSECRET", "output_mode": "content"},
{"pattern": "TOPSECRET", "path": "lndir", "output_mode": "content"},
} {
if out := f.call(t, "Grep", in).Flatten(); strings.Contains(out, confinedSecret) {
t.Errorf("Grep %v leaked the key: %s", in, out)
}
}
if res := f.call(t, "LS", map[string]any{"path": "lndir"}); !refused(res) || strings.Contains(res.Flatten(), "jwt.key") {
t.Errorf("LS through a symlinked directory was not refused: %s", res.Flatten())
}
if res := f.call(t, "Glob", map[string]any{"pattern": "*", "path": "lndir"}); strings.Contains(res.Flatten(), "jwt.key") {
t.Errorf("Glob through a symlinked directory listed the key: %s", res.Flatten())
}
}
// The refusal does not depend on privilege: the test user can read the key file
// itself, and the tools still refuse every path that leads to it.
func TestConfinedFileToolsRefuseOutsidePaths(t *testing.T) {
f := newConfinedFixture(t)
if _, err := os.ReadFile(f.secret); err != nil {
t.Fatalf("the process must be able to read the key itself: %v", err)
}
checkRefusals(t, f)
}
// As root the server can read any file; the tools must still refuse a root-only file
// outside the working directory and the server's own environment.
func TestConfinedFileToolsAsRoot(t *testing.T) {
if runtime.GOOS == "windows" || os.Geteuid() != 0 {
t.Skip("needs root")
}
f := newConfinedFixture(t)
if fi, err := os.Stat(f.secret); err != nil || fi.Mode().Perm() != 0o600 {
t.Fatalf("key file must be 0600: %v %v", fi, err)
}
if _, err := os.ReadFile("/proc/1/environ"); err != nil {
t.Fatalf("root must be able to read /proc/1/environ itself: %v", err)
}
checkRefusals(t, f)
for _, p := range []string{"/proc/1/environ", "/proc/self/root/etc/shadow"} {
if res := f.call(t, "Read", map[string]any{"file_path": p}); !refused(res) {
t.Errorf("Read %s was not refused: %s", p, res.Flatten())
}
}
}
func TestConfinedFileToolsWorkInside(t *testing.T) {
f := newConfinedFixture(t)
ok := func(tool string, in map[string]any, want string) {
t.Helper()
res := f.call(t, tool, in)
if res.IsError || !strings.Contains(res.Flatten(), want) {
t.Fatalf("%s %v = %q (error %v), want %q", tool, in, res.Flatten(), res.IsError, want)
}
}
ok("Read", map[string]any{"file_path": "in.txt"}, " 2\tworld")
ok("Read", map[string]any{"file_path": filepath.Join(f.work, "in.txt")}, "hello")
ok("Read", map[string]any{"file_path": "lnin"}, "hello") // a symlink that stays inside
ok("Write", map[string]any{"file_path": "sub/dir/new.txt", "content": "alpha beta\n"}, "Created")
ok("Write", map[string]any{"file_path": filepath.Join(f.work, "sub", "dir", "new.txt"), "content": "alpha beta\n"}, "Overwrote")
ok("Edit", map[string]any{"file_path": "sub/dir/new.txt", "old_string": "alpha", "new_string": "gamma"}, "1 replacement")
ok("MultiEdit", map[string]any{"file_path": "sub/dir/new.txt", "edits": []any{
map[string]any{"old_string": "gamma", "new_string": "delta"},
map[string]any{"old_string": "beta", "new_string": "epsilon"},
}}, "Applied 2 edits")
ok("Read", map[string]any{"file_path": "sub/dir/new.txt"}, "delta epsilon")
ok("LS", map[string]any{}, "in.txt")
ok("LS", map[string]any{"path": "sub"}, "dir/")
ok("Glob", map[string]any{"pattern": "**/*.txt"}, filepath.Join(f.work, "sub", "dir", "new.txt"))
ok("Grep", map[string]any{"pattern": "wor.d", "output_mode": "content"}, "in.txt:2:world")
ok("Grep", map[string]any{"pattern": "epsilon", "path": "sub"}, filepath.Join("dir", "new.txt"))
ok("Grep", map[string]any{"pattern": "hello", "path": "in.txt", "output_mode": "count"}, "in.txt: 1")
}
// A hard link in the working directory to a file outside it is a second name for
// that file: reading or writing through it is refused.
func TestConfinedFileToolsRefuseHardLinks(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("link counts are unix-only")
}
f := newConfinedFixture(t)
if err := os.Link(f.secret, filepath.Join(f.work, "hl")); err != nil {
t.Skipf("cannot create a hard link here: %v", err)
}
for _, c := range []struct {
tool string
in map[string]any
}{
{"Read", map[string]any{"file_path": "hl"}},
{"Write", map[string]any{"file_path": "hl", "content": "pwned"}},
{"Edit", map[string]any{"file_path": "hl", "old_string": confinedSecret, "new_string": "pwned"}},
} {
res := f.call(t, c.tool, c.in)
if !res.IsError || strings.Contains(res.Flatten(), confinedSecret) {
t.Errorf("%s through a hard link was not refused: %s", c.tool, res.Flatten())
}
}
if out := f.call(t, "Grep", map[string]any{"pattern": "TOPSECRET", "output_mode": "content"}).Flatten(); strings.Contains(out, confinedSecret) {
t.Errorf("Grep read the key through a hard link: %s", out)
}
f.secretUnchanged(t)
}
// The agent's shell can replace its own run directory with a symlink. The root is
// opened component by component, so the swapped directory cannot lead to / or to
// the rest of the data directory.
func TestConfinedFileToolsRunDirSymlinkSwap(t *testing.T) {
for _, target := range []string{"/", filepath.Join("..", "keys"), filepath.Join("..", "..", "keys")} {
f := newConfinedFixture(t)
if err := os.RemoveAll(f.work); err != nil {
t.Fatal(err)
}
if err := os.Symlink(target, f.work); err != nil {
t.Fatal(err)
}
for _, c := range []struct {
tool string
in map[string]any
}{
{"Read", map[string]any{"file_path": "jwt.key"}},
{"Read", map[string]any{"file_path": "keys/jwt.key"}},
{"Read", map[string]any{"file_path": "etc/passwd"}},
{"LS", map[string]any{}},
{"Write", map[string]any{"file_path": "new.txt", "content": "pwned"}},
} {
res := f.call(t, c.tool, c.in)
if !res.IsError || strings.Contains(res.Flatten(), confinedSecret) || strings.Contains(res.Flatten(), "jwt.key") {
t.Errorf("run dir -> %s: %s %v was not refused: %s", target, c.tool, c.in, res.Flatten())
}
}
f.secretUnchanged(t)
}
}
// A relative symlink that stays inside the data directory is still a symlink in the
// run directory path: os.Root would follow it, so the run directory is refused
// instead. Here it points at a root-only file's directory (the MITM CA's).
func TestConfinedFileToolsRunDirRelativeSymlinkInside(t *testing.T) {
f := newConfinedFixture(t)
ca := filepath.Join(f.base, "traffic", "_ca")
if err := os.MkdirAll(ca, 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(ca, "mitmproxy-ca.pem"), []byte(confinedSecret+"\n"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.RemoveAll(f.work); err != nil {
t.Fatal(err)
}
// The link sits in sessions/, so ../traffic/_ca is <base>/traffic/_ca: the in-base CA case.
if err := os.Symlink(filepath.Join("..", "traffic", "_ca"), f.work); err != nil {
t.Fatal(err)
}
for _, c := range []struct {
tool string
in map[string]any
}{
{"Read", map[string]any{"file_path": "mitmproxy-ca.pem"}},
{"Read", map[string]any{"file_path": "in.txt"}},
{"LS", map[string]any{}},
} {
res := f.call(t, c.tool, c.in)
if !res.IsError || strings.Contains(res.Flatten(), confinedSecret) {
t.Errorf("relative run dir symlink: %s %v was not refused: %s", c.tool, c.in, res.Flatten())
}
}
f.secretUnchanged(t)
}
// Missing or misplaced working directories fail closed.
func TestConfinedFileToolsNeedWorkDirUnderBase(t *testing.T) {
f := newConfinedFixture(t)
for _, dir := range []string{"", f.base, filepath.Dir(f.base)} {
f.tc = &actool.ToolContext{WorkingDir: dir}
if res := f.call(t, "Read", map[string]any{"file_path": f.secret}); !res.IsError {
t.Errorf("working dir %q: Read was not refused: %s", dir, res.Flatten())
}
}
}
// The model sees the SDK tools unchanged: same names, descriptions, prompts, schemas
// and read-only flags.
func TestConfinedFileToolsKeepSDKContract(t *testing.T) {
sdk := map[string]actool.CoreTool{}
for _, tl := range actool.DefaultTools() {
sdk[tl.Name()] = tl
}
got := defaultTools(t.TempDir())
if len(got) != len(sdk) {
t.Fatalf("defaultTools has %d tools, the SDK %d", len(got), len(sdk))
}
for _, tl := range got {
want := sdk[tl.Name()]
if want == nil {
t.Fatalf("unexpected tool %s", tl.Name())
}
_, confined := tl.(*confinedFileTool)
if _, replace := confinedFileToolNames[tl.Name()]; replace != confined {
t.Errorf("%s: confined=%v, want %v", tl.Name(), confined, replace)
}
if tl.Description() != want.Description() || tl.Prompt() != want.Prompt() ||
!reflect.DeepEqual(tl.InputSchema(), want.InputSchema()) ||
tl.IsReadOnly(json.RawMessage(`{}`)) != want.IsReadOnly(json.RawMessage(`{}`)) {
t.Errorf("%s: metadata differs from the SDK tool", tl.Name())
}
}
}
+14
View File
@@ -0,0 +1,14 @@
//go:build unix
package agent
import (
"io/fs"
"syscall"
)
// extraHardLinks reports a file that has a name besides the one being opened.
func extraHardLinks(fi fs.FileInfo) bool {
st, ok := fi.Sys().(*syscall.Stat_t)
return ok && st.Nlink > 1
}
+2 -2
View File
@@ -15,11 +15,11 @@ func testDB(t *testing.T) *db.DB {
t.Helper()
dsn, _, err := db.DSN()
if err != nil {
t.Skipf("no database config (%v)", err)
db.SkipPGUnavailable(t, err)
}
d, err := db.Open(dsn)
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
db.SkipPGUnavailable(t, err)
}
return d
}
+5 -4
View File
@@ -126,11 +126,11 @@ func (m *MainAgent) Chat(ctx context.Context, taskID int64, mainSeg int, as *db.
tsx.steerWork = m.steerWork // enable steer_work tool (nil = unavailable)
// domain tools + basic default toolset (Read/Write/Edit/MultiEdit/LS/Glob/Grep/Bash)
// when the asset coverage feature is off, drop add_task_scope/list_untested_assets (not in the prompt).
base := append(tsx.DropCoverageTools(tsx.MainAgentTools()), actool.DefaultTools()...)
base := append(tsx.DropCoverageTools(tsx.MainAgentTools()), defaultTools(m.workDir)...)
ctx = WithRunInfo(ctx, RunInfo{TaskID: taskID, ExplorationID: explorationID(ts)})
tools, def, cleanup := AugmentTools(ctx, "mainagent", base)
defer cleanup()
// this task's working directory <workDir>/tasks/<taskID>, created up front.
// the main run directory <workDir>/tasks/<taskID>/main, created up front.
mainDir := ensureRunDir(m.workDir, taskID, 0)
ctx = intercept.WithReviewWorkingDirectory(ctx, mainDir)
system, boundary := deferredSystem(mainAgentSystem(goal, m.workDir, mainDir), def)
@@ -155,8 +155,8 @@ func (m *MainAgent) Chat(ctx context.Context, taskID int64, mainSeg int, as *db.
DeepSeekSearchModel: m.webSearch.DeepSeekModel,
WebSearchProxy: m.webSearch.Proxy,
BashEnv: proxyEnv(m.proxyAddr, m.proxyCACert), // Bash subcommands go through the proxy + trust the CA by default
WorkingDir: mainDir, // this task's working directory <workDir>/tasks/<taskID>
ToolOutputDir: cmdOutDir(mainDir),
WorkingDir: mainDir, // the main run directory <workDir>/tasks/<taskID>/main
ToolOutputDir: TaskToolOutputDir(m.workDir, taskID, mainDir),
MaxTurns: m.maxTurns, // 0 = unlimited (configurable in agent management)
Compaction: compactionConfig(m.compactionWindow()), // long chats stay within the window
Todos: actool.NewTodoStore(), // session-level temporary todos (TodoWrite), planning only, dropped on exit
@@ -171,6 +171,7 @@ func (m *MainAgent) Chat(ctx context.Context, taskID int64, mainSeg int, as *db.
if hooks != nil { // typed-nil guard: only set when concrete (avoids harness panic)
opts.Hooks = hooks
}
opts.Hooks = SpillHooks(opts.Hooks, opts.ToolOutputDir)
// Segment 0 keeps the legacy "exp%d-main" name so existing transcripts still
// load; each new session (seg>=1) gets its own file for a clean context. The id
// is the provider-side identity (session header, usage metering) with or without
+5 -4
View File
@@ -486,7 +486,7 @@ func (p *Planner) Plan(ctx context.Context, taskID int64, as *db.AssetStore, ts
}
// Domain tools + the basic default tool set (Read/Write/Edit/MultiEdit/LS/Glob/Grep/Bash).
// When the asset-coverage feature is off, drop add_task_scope/list_untested_assets (not in the prompt).
base := append(tsx.DropCoverageTools(tsx.PlannerTools()), actool.DefaultTools()...)
base := append(tsx.DropCoverageTools(tsx.PlannerTools()), defaultTools(p.workDir)...)
ctx = WithRunInfo(ctx, RunInfo{TaskID: taskID, ExplorationID: explorationID(ts)})
tools, def, cleanup := AugmentTools(WithToolQuery(ctx, plannerToolQuery(goal, triggers)), "planner", base)
defer cleanup()
@@ -514,7 +514,7 @@ func (p *Planner) Plan(ctx context.Context, taskID int64, as *db.AssetStore, ts
// wrap-up round. Its agent-reported gap lines are framed with this message's env.
budgetRound, _ := ctx.Value(budgetRoundKey{}).(BudgetRound)
situational += tsx.coverageLedgerBlock(tc.Final || budgetRound.WrapUp, env)
// This task's work dir <workDir>/tasks/<taskID>; create it first.
// The main run directory <workDir>/tasks/<taskID>/main; create it first.
taskDir := ensureRunDir(p.workDir, taskID, 0)
ctx = intercept.WithReviewContext(ctx, taskDir, intercept.ReviewBackground{})
sysBody := plannerSystem(goal, p.workDir, taskDir) + frameLessonsBlock(p.lessonsBlock)
@@ -553,8 +553,8 @@ func (p *Planner) Plan(ctx context.Context, taskID int64, as *db.AssetStore, ts
DeepSeekSearchModel: p.webSearch.DeepSeekModel,
WebSearchProxy: p.webSearch.Proxy,
BashEnv: proxyEnv(p.proxyAddr, p.proxyCACert), // Bash subcommands go through the proxy + trust its CA by default
WorkingDir: taskDir, // this task's work dir <workDir>/tasks/<taskID>
ToolOutputDir: cmdOutDir(taskDir),
WorkingDir: taskDir, // the main run directory <workDir>/tasks/<taskID>/main
ToolOutputDir: TaskToolOutputDir(p.workDir, taskID, taskDir),
MaxTurns: p.maxTurns, // 0 = unlimited (configurable in agent management)
MaxDuration: maxDur, // 0=unlimited; with a deadline = time remaining until it
Compaction: compactionConfig(p.compactionWindow()),
@@ -575,6 +575,7 @@ func (p *Planner) Plan(ctx context.Context, taskID int64, as *db.AssetStore, ts
if hooks != nil { // typed-nil guard: only set when concrete (avoids harness panic)
opts.Hooks = hooks
}
opts.Hooks = SpillHooks(opts.Hooks, opts.ToolOutputDir)
// The session id is the provider-side identity (session header, usage metering)
// with or without persistence; Norma only puts it on ctx when it writes a transcript.
opts.SessionID = fmt.Sprintf("exp%d-planner", ts.ID())
+2
View File
@@ -27,6 +27,7 @@ import (
"github.com/Autumn-27/norma/compaction"
"github.com/Autumn-27/norma/llm"
acperm "github.com/Autumn-27/norma/permission"
actool "github.com/Autumn-27/norma/tool"
"github.com/Autumn-27/norma/transcript"
)
@@ -648,6 +649,7 @@ func TestConnection(ctx context.Context, c Config) (time.Duration, string, error
reply, err := agentcore.Run(ctx, agentcore.Options{
Provider: prov,
SystemPrompt: []string{"You are a connection test. Just output the two characters OK — no thinking, no explanation, nothing else."},
Tools: []actool.CoreTool{}, // not nil: nil means the SDK's unconfined file and shell tools
PermissionMode: acperm.ModeBypass,
MaxTurns: 1,
MaxTokens: 8192,
+1 -1
View File
@@ -19,7 +19,7 @@ import (
func TestWorkerReviewContextAcrossToolCalls(t *testing.T) {
dsn, _, err := db.DSN()
if err != nil {
t.Skip("no test database configured")
db.SkipPGUnavailable(t, err)
}
d, err := db.Open(dsn)
if err != nil {
+61
View File
@@ -0,0 +1,61 @@
package agent
import (
"os"
"path/filepath"
"strings"
"testing"
)
// The main agent and the planner work in tasks/<id>/main, a worker in
// tasks/<id>/i<N>, a chat in sessions/<id>: no run directory is an ancestor of
// another, and each is created through the server's maker when one is set.
func TestRunDirLayout(t *testing.T) {
base := t.TempDir()
mainDir := ensureRunDir(base, 7, 0)
worker := ensureRunDir(base, 7, 12)
session := EnsureSessionDir(base, "conv-3")
task := filepath.Join(base, "tasks", "7")
if mainDir != filepath.Join(task, "main") || worker != filepath.Join(task, "i12") || session != filepath.Join(base, "sessions", "conv-3") {
t.Fatalf("run directories: main %s, worker %s, session %s", mainDir, worker, session)
}
dirs := []string{mainDir, worker, session}
for _, dir := range dirs {
if fi, err := os.Stat(dir); err != nil || !fi.IsDir() {
t.Fatalf("%s was not created: %v", dir, err)
}
for _, other := range dirs {
if other != dir && strings.HasPrefix(other, dir+string(filepath.Separator)) {
t.Errorf("%s is an ancestor of %s", dir, other)
}
}
}
var made []string
SetRunDirMaker(func(b, leaf string) error {
if b != base {
t.Errorf("maker base = %s, want %s", b, base)
}
made = append(made, leaf)
return nil
})
t.Cleanup(func() { SetRunDirMaker(nil) })
want := []string{EnsureRunDir(base, 8, 0), EnsureRunDir(base, 8, 5), EnsureSessionDir(base, "conv-4")}
if strings.Join(made, "|") != strings.Join(want, "|") {
t.Errorf("maker calls = %v, want %v", made, want)
}
if _, err := os.Stat(filepath.Join(base, "tasks", "8")); !os.IsNotExist(err) {
t.Errorf("a run directory was created beside the maker: %v", err)
}
}
func TestIsRunDirName(t *testing.T) {
for name, want := range map[string]bool{
"main": true, "i1": true, "i120": true,
"": false, "i": false, "i1a": false, "i-1": false, "I1": false, "uploads": false, "cmd-output": false, "main2": false,
} {
if got := IsRunDirName(name); got != want {
t.Errorf("IsRunDirName(%q) = %v, want %v", name, got, want)
}
}
}
+5 -1
View File
@@ -59,7 +59,11 @@ func TestSideActualChatCheckpointToolResultAndTranscriptIsolation(t *testing.T)
for _, streaming := range []bool{false, true} {
t.Run(map[bool]string{true: "stream", false: "atomic"}[streaming], func(t *testing.T) {
dir := t.TempDir()
file := filepath.Join(dir, "asset.txt")
// Inside the chat's session working directory: the file tools reach nothing else.
file := filepath.Join(dir, "sessions", "conv-987654", "asset.txt")
if err := os.MkdirAll(filepath.Dir(file), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(file, []byte("controlled-homepage-result"), 0600); err != nil {
t.Fatal(err)
}
+116
View File
@@ -0,0 +1,116 @@
package agent
import (
"bytes"
"context"
"log"
"path/filepath"
"strconv"
"github.com/Autumn-27/norma/harness"
"github.com/Autumn-27/norma/llm"
)
// Command-output spills. Norma's tool.Capture writes an oversized tool output to a
// file in ToolOutputDir, from the server process. In <run dir>/cmd-output the
// executor, which can write in the run directory, could turn cmd-output into a
// symlink that root then writes through. When agent processes run as the executor
// the server therefore owns the spill directory: one per task or chat session,
// outside every run directory, that the executor can read and cannot change.
// spillDirMaker creates a spill directory below the data directory, and
// spillFileSharer lets the executor read the spill files in one (Norma writes them
// 0600). The server sets both when agent processes run as the executor
// (server/execpriv_unix.go); nil keeps the spill in <run dir>/cmd-output.
var (
spillDirMaker func(base, dir string) error
spillFileSharer func(dir string) error
)
// SetSpillDirMaker sets how spill directories are created and how the spill files in
// one are shared with the executor. Call it before any agent runs.
func SetSpillDirMaker(mkdir func(base, dir string) error, share func(dir string) error) {
spillDirMaker, spillFileSharer = mkdir, share
}
// TaskSpillDir is the server-owned spill directory of a task's agents:
// <base>/spill/tasks/<taskID>. It is removed and archived with the task directory.
func TaskSpillDir(base, taskID string) string {
return filepath.Join(base, "spill", "tasks", taskID)
}
// SessionSpillDir is the server-owned spill directory of a chat session:
// <base>/spill/sessions/<sessionID>.
func SessionSpillDir(base, sessionID string) string {
return filepath.Join(base, "spill", "sessions", sessionID)
}
// toolOutputDir returns the ToolOutputDir of a run. Without a maker it is
// <run dir>/cmd-output, which Norma creates on the first spill. With one it is
// spillDir, created here; when that fails the run gets no spill directory and
// oversized output is truncated instead.
func toolOutputDir(base, runDir, spillDir string) string {
if spillDirMaker == nil {
return filepath.Join(runDir, "cmd-output")
}
if err := spillDirMaker(base, spillDir); err != nil {
log.Printf("[agent] spill directory %s: %v", spillDir, err)
return ""
}
return spillDir
}
// TaskToolOutputDir is the ToolOutputDir of a task agent that works in runDir.
func TaskToolOutputDir(base string, taskID int64, runDir string) string {
return toolOutputDir(base, runDir, TaskSpillDir(base, strconv.FormatInt(taskID, 10)))
}
// SessionToolOutputDir is the ToolOutputDir of a chat that works in runDir.
func SessionToolOutputDir(base, sessionID, runDir string) string {
return toolOutputDir(base, runDir, SessionSpillDir(base, sessionID))
}
// spillMarker is in every result whose full output Capture wrote to a spill file.
var spillMarker = []byte("persisted-output")
// spillHooks shares the spill files of dir with the executor once the tool call that
// wrote one returns, before the model gets the path. inner may be nil.
type spillHooks struct {
inner harness.HookRunner
share func(dir string) error
dir string
}
// SpillHooks returns the hooks of a run whose ToolOutputDir is dir. With a sharer
// set they wrap hooks (which may be nil); otherwise hooks is returned as it is.
func SpillHooks(hooks harness.HookRunner, dir string) harness.HookRunner {
if spillFileSharer == nil || dir == "" {
return hooks
}
return spillHooks{inner: hooks, share: spillFileSharer, dir: dir}
}
func (h spillHooks) PreToolUse(ctx context.Context, name string, input []byte) (bool, string, []byte) {
if h.inner != nil {
return h.inner.PreToolUse(ctx, name, input)
}
return false, "", nil
}
func (h spillHooks) PostToolUse(ctx context.Context, name string, input, result []byte, isErr bool) {
if bytes.Contains(result, spillMarker) {
if err := h.share(h.dir); err != nil {
log.Printf("[agent] share the spill files in %s: %v", h.dir, err)
}
}
if h.inner != nil {
h.inner.PostToolUse(ctx, name, input, result, isErr)
}
}
func (h spillHooks) Stop(ctx context.Context, messages []llm.Message) (bool, []string, string) {
if h.inner != nil {
return h.inner.Stop(ctx, messages)
}
return false, nil, ""
}
+145
View File
@@ -0,0 +1,145 @@
package agent
import (
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"github.com/Autumn-27/norma/harness"
"github.com/Autumn-27/norma/llm"
actool "github.com/Autumn-27/norma/tool"
)
// countHooks counts the calls a spillHooks forwards.
type countHooks struct{ pre, post, stop int }
func (h *countHooks) PreToolUse(context.Context, string, []byte) (bool, string, []byte) {
h.pre++
return true, "blocked", nil
}
func (h *countHooks) PostToolUse(context.Context, string, []byte, []byte, bool) { h.post++ }
func (h *countHooks) Stop(context.Context, []llm.Message) (bool, []string, string) {
h.stop++
return true, nil, "stop"
}
// Without the privilege drop the spill stays in <run dir>/cmd-output and the hooks
// are not wrapped. With it, a task's agents share <base>/spill/tasks/<taskID> and a
// chat gets <base>/spill/sessions/<sessionID>: outside every run directory, created
// through the server's maker.
func TestToolOutputDir(t *testing.T) {
base := t.TempDir()
mainDir := filepath.Join(base, "tasks", "7", "main")
worker := filepath.Join(base, "tasks", "7", "i12")
session := filepath.Join(base, "sessions", "conv-3")
if got, want := TaskToolOutputDir(base, 7, worker), filepath.Join(worker, "cmd-output"); got != want {
t.Errorf("task ToolOutputDir without the drop = %s, want %s", got, want)
}
if got, want := SessionToolOutputDir(base, "conv-3", session), filepath.Join(session, "cmd-output"); got != want {
t.Errorf("session ToolOutputDir without the drop = %s, want %s", got, want)
}
if entries, err := os.ReadDir(base); err != nil || len(entries) != 0 {
t.Errorf("the data directory holds %v (%v), want nothing created", entries, err)
}
inner := &countHooks{}
if got := SpillHooks(inner, filepath.Join(worker, "cmd-output")); got != harness.HookRunner(inner) {
t.Errorf("hooks without the drop = %T, want them unchanged", got)
}
if got := SpillHooks(nil, filepath.Join(worker, "cmd-output")); got != nil {
t.Errorf("nil hooks without the drop = %T, want nil", got)
}
var made, shared []string
fail := false
SetSpillDirMaker(func(b, dir string) error {
if b != base {
t.Errorf("maker base = %s, want %s", b, base)
}
if fail {
return errors.New("refused")
}
made = append(made, dir)
return nil
}, func(dir string) error {
shared = append(shared, dir)
return nil
})
t.Cleanup(func() { SetSpillDirMaker(nil, nil) })
task7 := filepath.Join(base, "spill", "tasks", "7")
task8 := filepath.Join(base, "spill", "tasks", "8")
conv3 := filepath.Join(base, "spill", "sessions", "conv-3")
conv4 := filepath.Join(base, "spill", "sessions", "conv-4")
got := []string{
TaskToolOutputDir(base, 7, mainDir),
TaskToolOutputDir(base, 7, worker),
TaskToolOutputDir(base, 8, filepath.Join(base, "tasks", "8", "main")),
SessionToolOutputDir(base, "conv-3", session),
SessionToolOutputDir(base, "conv-4", filepath.Join(base, "sessions", "conv-4")),
}
want := []string{task7, task7, task8, conv3, conv4}
if strings.Join(got, "|") != strings.Join(want, "|") {
t.Errorf("ToolOutputDir with the drop = %v, want %v", got, want)
}
if strings.Join(made, "|") != strings.Join(want, "|") {
t.Errorf("maker calls = %v, want %v", made, want)
}
if TaskSpillDir(base, "7") != task7 || SessionSpillDir(base, "conv-3") != conv3 {
t.Errorf("spill directories: task %s, session %s", TaskSpillDir(base, "7"), SessionSpillDir(base, "conv-3"))
}
sep := string(filepath.Separator)
for _, dir := range got {
for _, runRoot := range []string{filepath.Join(base, "tasks"), filepath.Join(base, "sessions")} {
if strings.HasPrefix(dir, runRoot+sep) {
t.Errorf("spill directory %s is under %s", dir, runRoot)
}
}
}
// A spill directory that cannot be created is not replaced by the run directory.
fail = true
if got := TaskToolOutputDir(base, 9, filepath.Join(base, "tasks", "9", "main")); got != "" {
t.Errorf("ToolOutputDir after a maker failure = %q, want none", got)
}
if got := SpillHooks(inner, ""); got != harness.HookRunner(inner) {
t.Errorf("hooks of a run without a spill directory = %T, want them unchanged", got)
}
// The hooks share the spill files after a tool call whose result Capture spilled,
// and forward every call. The marker is the one Norma writes.
out := t.TempDir()
spilled := actool.Capture(&actool.ToolContext{OutputDir: out}, strings.Repeat("x", actool.MaxOutput(nil)+1))
entries, err := os.ReadDir(out)
if err != nil || len(entries) != 1 || !strings.Contains(spilled, filepath.Join(out, entries[0].Name())) {
t.Fatalf("Capture did not spill to %s: %v %v\n%s", out, entries, err, spilled)
}
raw, _ := json.Marshal(spilled)
small, _ := json.Marshal(actool.Capture(&actool.ToolContext{OutputDir: out}, "small output"))
ctx := context.Background()
for _, hooks := range []harness.HookRunner{SpillHooks(inner, task7), SpillHooks(nil, task7)} {
shared = nil
hooks.PostToolUse(ctx, "Bash", nil, small, false)
if len(shared) != 0 {
t.Errorf("the sharer ran for a result without a spill: %v", shared)
}
hooks.PostToolUse(ctx, "Bash", nil, raw, false)
if len(shared) != 1 || shared[0] != task7 {
t.Errorf("sharer calls after a spill = %v, want [%s]", shared, task7)
}
hooks.PreToolUse(ctx, "Bash", nil)
hooks.Stop(ctx, nil)
}
if block, msg, _ := SpillHooks(inner, task7).PreToolUse(ctx, "Bash", nil); !block || msg != "blocked" {
t.Errorf("PreToolUse = %v %q, want the inner answer", block, msg)
}
if prevent, _, msg := SpillHooks(inner, task7).Stop(ctx, nil); !prevent || msg != "stop" {
t.Errorf("Stop = %v %q, want the inner answer", prevent, msg)
}
if inner.pre != 2 || inner.post != 2 || inner.stop != 2 {
t.Errorf("forwarded calls: pre %d post %d stop %d, want 2 each", inner.pre, inner.post, inner.stop)
}
}
+24 -4
View File
@@ -2,6 +2,7 @@ package agent
import (
"database/sql"
"log"
"os"
"testing"
@@ -13,22 +14,41 @@ import (
// agent test suite so cross-package DELETE cleanup races with db/server
// packages are avoided when running `go test ./...`.
func TestMain(m *testing.M) {
// Norma keeps background-task output in os.TempDir()/norma. The tests get a temp
// directory of their own, so a session they start never writes the host's
// <temp>/norma. It is world-writable and sticky like /tmp: MkdirTemp makes it
// 0700, which an executor process could not traverse.
tmp, err := os.MkdirTemp("", "artex-test-tmp-")
if err != nil {
log.Fatalf("test temp directory: %v", err)
}
if err := os.Chmod(tmp, 0o777|os.ModeSticky); err != nil {
os.RemoveAll(tmp)
log.Fatalf("test temp directory: %v", err)
}
os.Setenv("TMPDIR", tmp)
// run removes the temp directory on every exit path: os.Exit runs no defer.
run := func() int {
code := m.Run()
os.RemoveAll(tmp)
return code
}
// Own database per package (#80): with separate databases, packages stop
// contending on shared tables and the advisory lock below stops
// serializing go test ./... across packages.
db.SuffixTestDatabase("agent")
dsn, _, err := db.DSN()
if err != nil {
os.Exit(m.Run())
os.Exit(run())
}
conn, err := sql.Open("pgx", dsn)
if err != nil || conn.Ping() != nil {
os.Exit(m.Run())
os.Exit(run())
}
defer conn.Close()
if _, err := conn.Exec(`SELECT pg_advisory_lock(7337741002)`); err != nil {
os.Exit(m.Run())
os.Exit(run())
}
defer conn.Exec(`SELECT pg_advisory_unlock(7337741002)`) //nolint:errcheck
os.Exit(m.Run())
os.Exit(run())
}
+8
View File
@@ -149,6 +149,10 @@ type ToolSet struct {
// auto-scope hook (insertAssets) is skipped, and add_task_scope/list_untested_assets
// are filtered out of the agent's tool list. The scope field stays regardless.
coverageDisabled bool
// workBase is the data directory the run directories live in, the base of the
// file tools (defaultTools). ingest_bbot opens its file through the run's working
// directory below it; empty = the tool refuses every path.
workBase string
// ownerNode is the exploration node that writes attach to: assets this run
// touches get anchored to it as lineage/provenance (NOT visibility — the asset
// graph is global and shared). Worker = its claimed intent; planner = begin root.
@@ -313,6 +317,10 @@ func (t *ToolSet) SetEnrich(e EnrichTrigger) { t.enrich = e }
// ownerNode is set are anchored to it as lineage (not visibility).
func (t *ToolSet) SetOwnerNode(id int64) { t.ownerNode = id }
// SetWorkBase sets the data directory the run directories live in (see
// ToolSet.workBase): the same base the run's file tools are confined under.
func (t *ToolSet) SetWorkBase(base string) { t.workBase = base }
// anchorOwner records a lineage edge from this run's owner node to an asset
// (no-op if unset). Provenance only — the asset graph is global and shared, so
// this no longer affects which assets a task can read.
+5
View File
@@ -5,6 +5,11 @@ import (
"strings"
)
// UntrustedBoundaryEnabled reports whether the untrusted-output boundary is on.
// Exported for the startup configuration snapshot; the agents themselves keep
// calling the private helper below.
func UntrustedBoundaryEnabled() bool { return untrustedBoundaryEnabled() }
// untrustedBoundaryEnabled reports whether Norma's untrusted-output boundary
// (agentcore.Options.UntrustedOutputBoundary: tool results framed between
// nonce-tagged markers) is on. It is on by default: unset, empty or any other
+78 -20
View File
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"path/filepath"
"strconv"
@@ -174,14 +175,14 @@ func NewWorker(prov llm.Provider, model, workDir string, tx *transcript.Store, w
return &Worker{prov: prov, model: model, workDir: workDir, tx: tx, window: window, maxTurns: maxTurns, extraTools: extra}
}
// defaultToolsExcept returns actool.DefaultTools() minus the named tools (by
// defaultToolsExcept returns defaultTools(base) minus the named tools (by
// CoreTool.Name()). Used to trim SDK default tools an agent shouldn't have.
func defaultToolsExcept(exclude ...string) []actool.CoreTool {
func defaultToolsExcept(base string, exclude ...string) []actool.CoreTool {
drop := make(map[string]bool, len(exclude))
for _, n := range exclude {
drop[n] = true
}
all := actool.DefaultTools()
all := defaultTools(base)
out := make([]actool.CoreTool, 0, len(all))
for _, t := range all {
if !drop[t.Name()] {
@@ -314,22 +315,77 @@ func workerArtifactSpec(runDir string) string {
return "\n\n**Intermediate artifact output rules**: all intermediate artifacts — scripts, payloads, captured response bodies, temporary data — **must be written to this intent's dedicated work directory " + runDir + "** (already created automatically; just write with relative paths there — no manual mkdir needed) — **do NOT write to /tmp, do NOT use other absolute paths**."
}
// ensureRunDir builds and creates an agent's working directory under base:
// <base>/tasks/<taskID> for planner/main; <base>/tasks/<taskID>/i<intentID> for a
// worker (intentID<=0 → task dir only). The "tasks/" segment groups per-task dirs
// symmetrically with the chat agent's "sessions/<sessionID>". Best-effort mkdir — on
// failure, writes fail the same way an unwritable CWD would.
func ensureRunDir(base string, taskID, intentID int64) string {
dir := filepath.Join(base, "tasks", strconv.FormatInt(taskID, 10))
if intentID > 0 {
dir = filepath.Join(dir, "i"+strconv.FormatInt(intentID, 10))
// mainRunDirName is the run directory of a task's main agent and planner.
const mainRunDirName = "main"
// runDirMaker creates a run directory below the data directory with the ownership
// the server wants. The server sets it when agent processes run as the executor
// (server/execpriv_unix.go); nil keeps a plain MkdirAll.
var runDirMaker func(base, leaf string) error
// SetRunDirMaker sets how run directories are created. Call it before any agent runs.
func SetRunDirMaker(f func(base, leaf string) error) { runDirMaker = f }
// makeRunDir creates the run directory leaf below base. Best-effort — on failure,
// writes fail the same way an unwritable CWD would.
func makeRunDir(base, leaf string) {
if runDirMaker == nil {
// 0775: the umask of the server decides the final bits.
_ = os.MkdirAll(leaf, 0o775)
return
}
_ = os.MkdirAll(dir, 0o755)
if err := runDirMaker(base, leaf); err != nil {
log.Printf("[agent] run directory %s: %v", leaf, err)
}
}
// ensureRunDir builds and creates an agent's working directory under base:
// <base>/tasks/<taskID>/main for planner/main (intentID<=0);
// <base>/tasks/<taskID>/i<intentID> for a worker. No run directory is an ancestor of
// another: the task directory holds the leaves and is no agent's working directory.
// The "tasks/" segment groups per-task dirs symmetrically with the chat agent's
// "sessions/<sessionID>".
func ensureRunDir(base string, taskID, intentID int64) string {
leaf := mainRunDirName
if intentID > 0 {
leaf = "i" + strconv.FormatInt(intentID, 10)
}
dir := filepath.Join(base, "tasks", strconv.FormatInt(taskID, 10), leaf)
makeRunDir(base, dir)
return dir
}
// cmdOutDir is the SDK large-tool-output spill dir under an agent's run dir.
func cmdOutDir(dir string) string { return filepath.Join(dir, "cmd-output") }
// EnsureRunDir is ensureRunDir for the server, which places the main agent's
// uploads in its run directory.
func EnsureRunDir(base string, taskID, intentID int64) string {
return ensureRunDir(base, taskID, intentID)
}
// EnsureSessionDir creates the chat run directory <base>/sessions/<sessionID> the
// way ensureRunDir creates a task one, and returns it.
func EnsureSessionDir(base, sessionID string) string {
dir := filepath.Join(base, "sessions", sessionID)
makeRunDir(base, dir)
return dir
}
// IsRunDirName reports whether name is a run directory inside a task directory:
// main or i<intentID>.
func IsRunDirName(name string) bool {
if name == mainRunDirName {
return true
}
digits, ok := strings.CutPrefix(name, "i")
if !ok || digits == "" {
return false
}
for _, c := range digits {
if c < '0' || c > '9' {
return false
}
}
return true
}
func workerSystem(proxyAddr, caCert, dataDir, runDir string) string {
body := renderSystem("worker", workerDefaultTmpl, WorkerVars{ProxyAddr: proxyAddr, DataDir: dataDir, Now: nowStr()})
@@ -432,6 +488,7 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
if as != nil {
tsx.SetAssetStore(as, as.Companies())
}
tsx.SetWorkBase(w.workDir) // ingest_bbot reads its file under the same base as the file tools below
tsx.SetOwnerNode(intent.ID) // assets this worker discovers anchor to its intent → visible to the task
tsx.SetEnrich(enr) // async DNS/HTTP auto-completion for assets this worker writes
tsx.SetNotifyFinding(notifyFinding) // when report_finding lands, wake the planner on the spot with "which intent + finding"
@@ -443,7 +500,7 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
// searching goes through Bash(grep/find) — narrowing the tool surface and reducing
// low-value calls. The rest of the SDK default tools (Read/Write/Edit/LS/Bash/Sleep)
// are kept as usual.
base = append(base, defaultToolsExcept("MultiEdit", "Glob", "Grep")...)
base = append(base, defaultToolsExcept(w.workDir, "MultiEdit", "Glob", "Grep")...)
ctx = WithRunInfo(ctx, RunInfo{TaskID: taskID, ExplorationID: explorationID(ts), IntentID: intent.ID})
tools, def, cleanup := AugmentTools(WithToolQuery(ctx, nodeSummary(intent)), "worker", base)
defer cleanup()
@@ -544,10 +601,10 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
// deadline → task-timeout prompt; turns = the turn budget ran out inside the
// clamped window → fall back to the per-run prompt. Non-clamped stays pure per-run.
Settlement: settle,
// large tool output spills to cmd-output/ with a head + pointer (SDK tool.Capture);
// full output preserved on disk. The truncation cap uses the SDK default (30000
// chars).
ToolOutputDir: cmdOutDir(runDir),
// large tool output spills to a file with a head + pointer (SDK tool.Capture);
// full output preserved on disk, in cmd-output/ or in the task's spill
// directory (spill.go). The truncation cap uses the SDK default (30000 chars).
ToolOutputDir: TaskToolOutputDir(w.workDir, taskID, runDir),
Compaction: compactionConfig(w.compactionWindow()), // long tool-heavy runs stay within the window
Todos: actool.NewTodoStore(), // session-level scratch todos (TodoWrite), pure planning, discarded on exit
NonStreaming: w.nonStreaming(), // when this profile selects non-streaming, use Provider.Complete
@@ -559,6 +616,7 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
if hooks != nil { // typed-nil guard: only set when concrete (avoids harness panic)
opts.Hooks = hooks
}
opts.Hooks = SpillHooks(opts.Hooks, opts.ToolOutputDir)
// The session id is the provider-side identity (session header, usage metering)
// with or without persistence; Norma only puts it on ctx when it writes a transcript.
opts.SessionID = WorkerSessionID(ts.ID(), intent.ID)
+119 -29
View File
@@ -5,6 +5,7 @@ package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"log"
@@ -18,9 +19,41 @@ import (
"github.com/Autumn-27/artex/agent"
"github.com/Autumn-27/artex/config"
"github.com/Autumn-27/artex/egress"
"github.com/Autumn-27/artex/server"
)
// configSnapshot is the single startup log line describing the effective
// configuration. It carries no credential material: the DSN is passed through
// config.Redact, and no key or token value is ever included.
type configSnapshot struct {
Version string `json:"version"`
ConfigFile string `json:"config_file"`
ConfigFileFound bool `json:"config_file_found"`
PGDSN string `json:"pg_dsn"`
PGDSNSource string `json:"pg_dsn_source"`
DataDir string `json:"data_dir"`
SkillDir string `json:"skill_dir"`
KeyDir string `json:"key_dir"`
Addr string `json:"addr"`
Proxy string `json:"proxy"`
NoiseProfile string `json:"noise_profile"`
UntrustedBoundary bool `json:"untrusted_boundary"`
TrustedProxies []string `json:"trusted_proxies"`
}
// logConfigSnapshot emits one structured line so an operator can reproduce the
// exact configuration of a run from the logs. A serialization failure is
// reported but never aborts startup.
func logConfigSnapshot(s configSnapshot) {
b, err := json.Marshal(s)
if err != nil {
log.Printf("[config] snapshot unavailable: %v", err)
return
}
log.Printf("[config] snapshot %s", b)
}
// version is the build version, injected at release time via
// -ldflags "-X main.version=<tag>". Defaults to "dev" for local builds.
var version = "dev"
@@ -44,14 +77,34 @@ func printBanner(addr string) {
// main maps run's result onto the process exit code. The body lives in a
// function that can *return* rather than os.Exit, so its own defers still run.
func main() {
// Started as the Bash shim or as an MCP drop command: drop the identity and exec
// the real program. It returns only for an ordinary start.
server.RunExecShimIfRequested(os.Args)
// Before any listener or child exists: keep the secrets out of /proc/<pid>/environ.
server.SealProcessEnv()
os.Exit(run())
}
// flagWasSet reports whether the operator passed the named flag.
func flagWasSet(name string) bool {
set := false
flag.Visit(func(f *flag.Flag) {
if f.Name == name {
set = true
}
})
return set
}
func run() int {
var (
addr = flag.String("addr", "127.0.0.1:8787", "HTTP listen address")
dataDir = flag.String("data", filepath.Join(config.BaseDir(), "data"), "runtime data directory: traffic index, task files, transcripts (default: data/ next to the executable)")
proxy = flag.String("proxy", "127.0.0.1:8788", "traffic recording proxy address (empty to disable)")
// Identity of every agent process (Bash, the shell, custom tools, MCP stdio).
// A negative uid turns the drop off; 0 (root) is refused.
execUID = flag.Int("exec-uid", server.DefaultExecUID, "uid that agent processes run as (Bash, shell, custom tools, MCP stdio); -1 turns the drop off")
execGID = flag.Int("exec-gid", server.DefaultExecGID, "gid that agent processes run as")
)
flag.Parse()
@@ -65,16 +118,74 @@ func run() int {
// page can show a live log stream. Do this first, to catch startup logs too.
server.StartLogCapture()
// surface which config file the binary reads (absolute, so `go run`'s relative
// "config.json" — resolved against the CWD — is unambiguous).
cfgPath := config.Path()
// Resolve every effective setting before starting the stores, then emit a
// single structured snapshot line. The old code logged the same facts in
// five scattered lines, which made a run hard to reproduce from logs.
cfgPath := config.Path() // absolute, so `go run`'s relative CWD is unambiguous
if abs, e := filepath.Abs(cfgPath); e == nil {
cfgPath = abs
}
if _, e := os.Stat(cfgPath); e == nil {
log.Printf("[config] config file: %s", cfgPath)
} else {
log.Printf("[config] config file: %s (not found — will only try the environment variable ARTEX_PG_DSN)", cfgPath)
_, statErr := os.Stat(cfgPath)
cfgFound := statErr == nil
skillDir, err := config.SkillDir()
if err != nil {
log.Printf("[config] %v", err)
return 1
}
if abs, e := filepath.Abs(skillDir); e == nil {
skillDir = abs
}
// JWT signing key directory (also holds secret.key, see server.NewManager).
keyDir := config.KeyDir()
// Proxies whose X-Forwarded-For the login rate limit reads; empty trusts none.
trusted, err := server.ParseTrustedProxies(os.Getenv("ARTEX_TRUSTED_PROXIES"))
if err != nil {
log.Printf("[config] ARTEX_TRUSTED_PROXIES: %v", err)
return 1
}
trustedAddrs := make([]string, 0, len(trusted))
for _, p := range trusted {
trustedAddrs = append(trustedAddrs, p.String())
}
// NewManager resolves the DSN too, but does so internally. Calling
// PostgresDSN here is idempotent (no connection, no side effects) and lets
// the snapshot show the source and a redacted view even on a failed start.
dsn, dsnSource, dsnErr := config.PostgresDSN()
if dsnErr != nil {
dsn, dsnSource = "", "unavailable"
}
logConfigSnapshot(configSnapshot{
Version: version,
ConfigFile: cfgPath,
ConfigFileFound: cfgFound,
PGDSN: config.Redact(dsn),
PGDSNSource: dsnSource,
DataDir: *dataDir,
SkillDir: skillDir,
KeyDir: keyDir,
Addr: *addr,
Proxy: *proxy,
NoiseProfile: egress.ProfileFromEnv().Name,
UntrustedBoundary: agent.UntrustedBoundaryEnabled(),
TrustedProxies: trustedAddrs,
})
// Agent processes run as the executor identity, never as the server's root (see
// server/execpriv.go). An explicit option that cannot be applied stops the start.
priv, err := server.ResolveExecPrivilege(*execUID, *execGID, flagWasSet("exec-uid") || flagWasSet("exec-gid"))
if err != nil {
log.Printf("[config] %v", err)
return 1
}
if err := server.PrepareExecPrivilege(&priv, *dataDir, config.KeyDir(), config.Path()); err != nil {
log.Printf("[config] %v", err)
return 1
}
if err := server.InstallBashShim(priv); err != nil {
log.Printf("[config] %v", err)
return 1
}
sigCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
@@ -88,28 +199,7 @@ func run() int {
}
defer mgr.Close()
skillDir, err := config.SkillDir()
if err != nil {
log.Printf("[config] %v", err)
return 1
}
if abs, err := filepath.Abs(skillDir); err == nil {
skillDir = abs
}
log.Printf("[config] skill directory: %s", skillDir)
// JWT signing key directory (also holds secret.key, see server.NewManager).
keyDir := config.KeyDir()
log.Printf("[config] JWT key directory: %s", keyDir)
// Proxies whose X-Forwarded-For the login rate limit reads; empty trusts none.
trusted, err := server.ParseTrustedProxies(os.Getenv("ARTEX_TRUSTED_PROXIES"))
if err != nil {
log.Printf("[config] ARTEX_TRUSTED_PROXIES: %v", err)
return 1
}
if len(trusted) > 0 {
log.Printf("[config] trusted proxies: %v", trusted)
}
srv := server.New(ctx, mgr, skillDir, *dataDir, keyDir, server.WithSecretScrub(), server.WithTrustedProxies(trusted))
srv := server.New(ctx, mgr, skillDir, *dataDir, keyDir, server.WithSecretScrub(), server.WithTrustedProxies(trusted), server.WithExecPrivilege(priv))
// ReadTimeout bounds how long the request headers and body may take to arrive.
// It does not limit how long a handler runs, so an event stream stays open.
// WriteTimeout is left unset: a write deadline would cut that stream.
+24
View File
@@ -2,11 +2,35 @@ package main
import (
"context"
"encoding/json"
"strings"
"testing"
"github.com/Autumn-27/artex/agent"
)
// The snapshot must be serializable and must not add fields. Password masking
// itself is config.Redact's contract, covered by config/config_test.go;
// cmd/artex only passes its result through to the log line.
func TestConfigSnapshotSerializesRedactedDSN(t *testing.T) {
redacted := "postgres://artex:***@localhost:5432/artex?sslmode=disable"
s := configSnapshot{
Version: "test",
ConfigFileFound: true,
PGDSN: redacted,
PGDSNSource: "environment variable ARTEX_PG_DSN",
TrustedProxies: []string{},
}
b, err := json.Marshal(s)
if err != nil {
t.Fatal(err)
}
out := string(b)
if !strings.Contains(out, `"pg_dsn":"`+redacted+`"`) {
t.Fatalf("snapshot did not serialize the redacted DSN: %s", out)
}
}
// The shutdown cause must survive the parent context being cancelled first;
// losing it would drop the diagnostic on every graceful stop.
func TestShutdownContextPreservesNamedCause(t *testing.T) {
+2 -2
View File
@@ -20,7 +20,7 @@ func TestActivityOutboxKeepsDetail(t *testing.T) {
func TestFlushActivityOutboxReplaysOnce(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
expID, err := d.CreateExploration("outbox", "replay")
@@ -63,7 +63,7 @@ func TestFlushActivityOutboxReplaysOnce(t *testing.T) {
func TestFlushActivityOutboxDropsStuckRow(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
const missing = int64(9_000_000_000_001)
+2 -2
View File
@@ -8,7 +8,7 @@ import "testing"
func TestActivityPageSessions(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -129,7 +129,7 @@ func TestActivityPageSessions(t *testing.T) {
func TestListByKindPage(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+1 -1
View File
@@ -14,7 +14,7 @@ func testSetup(t *testing.T) (*DB, *AssetStore, *CompanyStore) {
t.Helper()
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
return d, d.Assets(), d.Companies()
}
+1 -1
View File
@@ -168,7 +168,7 @@ func TestRewriteStockNpxBrowserMCPDoesNotAliasInput(t *testing.T) {
func TestSeedAndMigrateBrowserMCP(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
origCmd, _, origRaw := readBrowserMCP(t, d)
+12 -12
View File
@@ -17,7 +17,7 @@ func cleanupCompany(d *DB, id int64) {
func TestCompanyUpsertAndGet(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
@@ -64,7 +64,7 @@ func TestCompanyUpsertAndGet(t *testing.T) {
func TestCreateCompanyWithScopeRejectsNormalizedDuplicate(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
@@ -103,7 +103,7 @@ func TestCreateCompanyWithScopeRejectsNormalizedDuplicate(t *testing.T) {
func TestCreateCompanyWithScopeRollsBackOnScopeWriteFailure(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
@@ -128,7 +128,7 @@ func TestCreateCompanyWithScopeRollsBackOnScopeWriteFailure(t *testing.T) {
func TestCompanyScope(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
@@ -166,7 +166,7 @@ func TestCompanyScope(t *testing.T) {
func TestCompanyScopeInvalid(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
@@ -195,7 +195,7 @@ func TestCompanyScopeInvalid(t *testing.T) {
func TestResolveCompany(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
@@ -236,7 +236,7 @@ func TestResolveCompany(t *testing.T) {
func TestUpdateScope(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
@@ -281,7 +281,7 @@ func TestUpdateScope(t *testing.T) {
func TestUpdateScopeRollsBackOnInsertFailure(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
@@ -316,7 +316,7 @@ func TestUpdateScopeRollsBackOnInsertFailure(t *testing.T) {
func TestDeleteCompany(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
@@ -344,7 +344,7 @@ func TestDeleteCompany(t *testing.T) {
func TestDeleteCompanyWithAssetsDeletesBoth(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
@@ -391,7 +391,7 @@ RETURNING id`, domain, id).Scan(&assetID); err != nil {
func TestRecomputeAttribution(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
@@ -433,7 +433,7 @@ func TestRecomputeAttribution(t *testing.T) {
func TestListCompanies(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
cs := d.Companies()
+1 -1
View File
@@ -233,7 +233,7 @@ func TestCompanyScopeLimitsAndCheckedErrors(t *testing.T) {
func TestCompanyScopeCheckedReturnsSystemErrorSeparately(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
companies := d.Companies()
if err := d.Close(); err != nil {
+2 -2
View File
@@ -233,7 +233,7 @@ func TestParseStructuredCompanyScope(t *testing.T) {
func TestCompanyICPAttribution(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Closing the connection must go through t.Cleanup and be **registered before
// the data cleanup**: t.Cleanup is last-in-first-out, so registering Close
@@ -322,7 +322,7 @@ func TestCompanyICPAttribution(t *testing.T) {
func TestCompanyScopeAttribution(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
as := d.Assets()
cs := d.Companies()
+2 -2
View File
@@ -9,7 +9,7 @@ import "testing"
func TestProfileMaxTokensRoundTrip(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Close via Cleanup, registered FIRST so it runs LAST: cleanups are LIFO, and a
// plain `defer d.Close()` would fire before them — the row-deleting cleanups
@@ -77,7 +77,7 @@ func TestProfileMaxTokensRoundTrip(t *testing.T) {
func TestProfileMaxTokensDefaults(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Close via Cleanup, registered FIRST so it runs LAST: cleanups are LIFO, and a
// plain `defer d.Close()` would fire before them — the row-deleting cleanups
+2 -2
View File
@@ -17,7 +17,7 @@ import (
func TestPoolProfilesOrder(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Close via Cleanup, registered FIRST so it runs LAST: a plain defer would
// close the pool before the row-deleting cleanups below run.
@@ -83,7 +83,7 @@ func TestDeleteProfileContextHonorsCancellation(t *testing.T) {
func TestConfigStores(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+1 -1
View File
@@ -9,7 +9,7 @@ import (
func TestConversationPinOrderingAndPatch(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
+1 -1
View File
@@ -87,7 +87,7 @@ INSERT INTO exploration_anchors(node_id,asset_id) SELECT fn.id, an.id FROM fn JO
func TestBuildCoverageGraphLargeInvariants(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
f := seedLargeCoverageFixture(t, d)
+3 -3
View File
@@ -100,7 +100,7 @@ func TestCoverageJSONCarriesLedger(t *testing.T) {
func TestRecordCoverageUpsertHistory(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -157,7 +157,7 @@ func TestRecordCoverageUpsertHistory(t *testing.T) {
func TestCoverageCompletenessBudgetThenReviveDone(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -221,7 +221,7 @@ func TestCoverageCompletenessBudgetThenReviveDone(t *testing.T) {
func TestRecordCoverageRowCap(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
+1 -1
View File
@@ -10,7 +10,7 @@ import (
func TestCustomToolCRUD(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
+3
View File
@@ -350,6 +350,9 @@ ON CONFLICT (name) DO NOTHING`,
if err := d.seedDefaultInterceptRules(); err != nil {
return fmt.Errorf("seed intercept rules: %w", err)
}
if err := d.seedPersistenceApprovalRules(); err != nil {
return fmt.Errorf("seed persistence approval rules: %w", err)
}
if err := d.seedDefaultInterceptRulesV2(); err != nil {
return fmt.Errorf("seed intercept rules v2: %w", err)
}
+2 -2
View File
@@ -75,7 +75,7 @@ func testDSN(t *testing.T) string {
t.Helper()
dsn, _, err := DSN()
if err != nil {
t.Skipf("no database config (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
return dsn
}
@@ -85,7 +85,7 @@ func testDSN(t *testing.T) string {
func TestOpenSeed(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+239
View File
@@ -0,0 +1,239 @@
package db
import (
"context"
"crypto/sha256"
"database/sql"
"encoding/hex"
"encoding/json"
"io/fs"
"slices"
"strings"
"testing"
"testing/fstest"
"time"
)
func TestTrafficSnapshotIDExcludesRewriteFlag(t *testing.T) {
base := TrafficEvidenceSnapshot{
SourceTrafficID: "tr-100",
CapturedAt: 1700000000,
URL: "https://target.example/login",
Method: "POST",
Status: 200,
ContentType: "application/json",
ReqHead: "POST /login HTTP/1.1\nHost: target.example\n",
RespHead: "HTTP 200\nContent-Type: application/json\n",
ReqHash: strings.Repeat("a", 64),
RespHash: strings.Repeat("b", 64),
ReqLen: 42,
RespLen: 128,
}
idPlain := TrafficSnapshotID(base)
withFlag := base
withFlag.Rewritten = true
withFlag.RewriteRuleIDs = []int64{1, 2}
if got := TrafficSnapshotID(withFlag); got != idPlain {
t.Fatalf("TrafficSnapshotID changed with Rewritten=true/ids=[1,2]: got %s want %s", got, idPlain)
}
withOtherRules := base
withOtherRules.Rewritten = true
withOtherRules.RewriteRuleIDs = []int64{99, 42, 7}
if got := TrafficSnapshotID(withOtherRules); got != idPlain {
t.Fatalf("TrafficSnapshotID changed with different rule ids: got %s want %s", got, idPlain)
}
// Simulate an older binary whose TrafficEvidenceSnapshot struct did not have
// Rewritten or RewriteRuleIDs: it must marshal to the exact same hash.
type legacyTrafficEvidenceSnapshot struct {
ID string `json:"id"`
SourceTrafficID string `json:"source_traffic_id"`
CapturedAt int64 `json:"captured_at"`
URL string `json:"url"`
Method string `json:"method"`
Status int `json:"status"`
ContentType string `json:"content_type"`
ReqHead string `json:"req_head,omitempty"`
RespHead string `json:"resp_head,omitempty"`
ReqHash string `json:"req_hash"`
RespHash string `json:"resp_hash"`
ReqLen int64 `json:"req_len"`
RespLen int64 `json:"resp_len"`
CreatedAt time.Time `json:"created_at"`
}
norm := base.Normalize()
legacy := legacyTrafficEvidenceSnapshot{
SourceTrafficID: norm.SourceTrafficID,
CapturedAt: norm.CapturedAt,
URL: norm.URL,
Method: norm.Method,
Status: norm.Status,
ContentType: norm.ContentType,
ReqHead: norm.ReqHead,
RespHead: norm.RespHead,
ReqHash: norm.ReqHash,
RespHash: norm.RespHash,
ReqLen: norm.ReqLen,
RespLen: norm.RespLen,
}
raw, err := json.Marshal(legacy)
if err != nil {
t.Fatal(err)
}
sum := sha256.Sum256(raw)
wantLegacy := hex.EncodeToString(sum[:])
if idPlain != wantLegacy {
t.Fatalf("TrafficSnapshotID = %s, old-version hash = %s", idPlain, wantLegacy)
}
}
func TestEvidenceRewriteFlagMigrationAndRoundTrip(t *testing.T) {
dsn := scratchDSN(t, "ev_rewrite")
// Build an FS containing migrations 0001..0007 (pre-0008 database).
pre0008 := fstest.MapFS{}
entries, err := fs.ReadDir(migrationsFS(), ".")
if err != nil {
t.Fatal(err)
}
for _, e := range entries {
if e.IsDir() || strings.HasPrefix(e.Name(), "0008_") {
continue
}
data, err := fs.ReadFile(migrationsFS(), e.Name())
if err != nil {
t.Fatal(err)
}
pre0008[e.Name()] = &fstest.MapFile{Data: data}
}
d0, err := openWithMigrations(dsn, pre0008)
if err != nil {
t.Fatalf("open pre-0008: %v", err)
}
task, err := d0.CreateTask("ev-rewrite-mig", "verify 0008", nil, 0, 0)
if err != nil {
d0.Close()
t.Fatal(err)
}
legacySnap := TrafficEvidenceSnapshot{
SourceTrafficID: "tr-pre-0008",
CapturedAt: 1700000001,
URL: "https://legacy.example/a",
Method: "GET",
Status: 200,
ContentType: "text/plain",
ReqHead: "GET /a HTTP/1.1\nHost: legacy.example\n",
RespHead: "HTTP 200\n",
ReqHash: strings.Repeat("1", 64),
RespHash: strings.Repeat("2", 64),
}
legacySnap.ID = TrafficSnapshotID(legacySnap)
var legacyFindingID int64
if err := d0.WithEvidenceTx(context.Background(), func(tx *sql.Tx) error {
rec, err := RecordFindingTx(context.Background(), tx, RecordFindingInput{
TaskID: task.ID, ExplorationID: task.ExplorationID, Worker: "test",
VulnClass: "TEST", Name: "pre-0008 finding", Severity: "low", Summary: "legacy row",
}, nil)
if err != nil {
return err
}
legacyFindingID = rec.FindingID
if _, err := tx.Exec(`INSERT INTO traffic_evidence_snapshots
(id,source_traffic_id,captured_at,url,method,status,content_type,req_head,resp_head,req_hash,resp_hash,req_len,resp_len)
VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13)`,
legacySnap.ID, legacySnap.SourceTrafficID, legacySnap.CapturedAt, legacySnap.URL, legacySnap.Method,
legacySnap.Status, legacySnap.ContentType, legacySnap.ReqHead, legacySnap.RespHead,
legacySnap.ReqHash, legacySnap.RespHash, legacySnap.ReqLen, legacySnap.RespLen); err != nil {
return err
}
_, err = tx.Exec(`INSERT INTO finding_traffic_bindings(finding_id,snapshot_id,role,note,position) VALUES($1,$2,'proof','pre-0008',0)`,
legacyFindingID, legacySnap.ID)
return err
}); err != nil {
d0.Close()
t.Fatalf("seed pre-0008 snapshot: %v", err)
}
d0.Close()
// Re-open with full embedded migrations: 0008 must apply cleanly.
d, err := Open(dsn)
if err != nil {
t.Fatalf("Open with 0008: %v", err)
}
defer d.Close()
assertEmbeddedApplied(t, d)
// Existing pre-0008 row reads rewritten=false and empty rewrite_rule_ids.
ftLegacy, err := d.GetFindingTraffic(context.Background(), legacyFindingID)
if err != nil {
t.Fatal(err)
}
if len(ftLegacy.Bindings) != 1 {
t.Fatalf("legacy bindings = %d, want 1", len(ftLegacy.Bindings))
}
gotLegacy := ftLegacy.Bindings[0].Snapshot
if gotLegacy.Rewritten || len(gotLegacy.RewriteRuleIDs) != 0 {
t.Fatalf("pre-0008 snapshot = rewritten:%v ids:%v, want false/empty", gotLegacy.Rewritten, gotLegacy.RewriteRuleIDs)
}
// New snapshot with Rewritten=true and unsorted/duplicate rule IDs round-trips cleanly
// through InsertEvidenceSnapshotTx and FindingTrafficTx while preserving sealed heads.
rewrittenSnap := TrafficEvidenceSnapshot{
SourceTrafficID: "tr-rewritten-1",
CapturedAt: 1700000002,
URL: "https://rewritten.example/item",
Method: "POST",
Status: 201,
ContentType: "application/json",
ReqHead: "POST /item HTTP/1.1\nAuthorization: Bearer topsecret\n",
RespHead: "HTTP 201\nSet-Cookie: sid=topsecret\n",
ReqHash: strings.Repeat("3", 64),
RespHash: strings.Repeat("4", 64),
ReqLen: 10,
RespLen: 20,
Rewritten: true,
RewriteRuleIDs: []int64{9, 2, 9, 5},
}
rewrittenSnap.ID = TrafficSnapshotID(rewrittenSnap)
var rewrittenFindingID int64
if err := d.WithEvidenceTx(context.Background(), func(tx *sql.Tx) error {
rec, err := RecordFindingTx(context.Background(), tx, RecordFindingInput{
TaskID: task.ID, ExplorationID: task.ExplorationID, Worker: "test",
VulnClass: "TEST", Name: "rewritten finding", Severity: "medium", Summary: "post-0008",
}, []PreparedTrafficEvidence{{
Ref: TrafficRef{TrafficID: rewrittenSnap.SourceTrafficID, Role: "proof", Note: "rewritten"},
Snapshot: rewrittenSnap,
}})
if err != nil {
return err
}
rewrittenFindingID = rec.FindingID
return nil
}); err != nil {
t.Fatalf("record rewritten snapshot: %v", err)
}
ft, err := d.GetFindingTraffic(context.Background(), rewrittenFindingID)
if err != nil {
t.Fatal(err)
}
if len(ft.Bindings) != 1 {
t.Fatalf("rewritten bindings = %d, want 1", len(ft.Bindings))
}
got := ft.Bindings[0].Snapshot
if !got.Rewritten || !slices.Equal(got.RewriteRuleIDs, []int64{2, 5, 9}) {
t.Fatalf("round-tripped snapshot = rewritten:%v ids:%v, want true [2 5 9]", got.Rewritten, got.RewriteRuleIDs)
}
if got.ID != rewrittenSnap.ID || TrafficSnapshotID(got) != got.ID {
t.Fatalf("snapshot ID mismatch: got=%s want=%s recomputed=%s", got.ID, rewrittenSnap.ID, TrafficSnapshotID(got))
}
if got.ReqHead != rewrittenSnap.ReqHead || got.RespHead != rewrittenSnap.RespHead {
t.Fatalf("heads mismatch after round-trip: req=%q resp=%q", got.ReqHead, got.RespHead)
}
}
+3 -3
View File
@@ -92,7 +92,7 @@ func assertClearHeads(t *testing.T, d *DB, findingID int64, want TrafficEvidence
func TestEvidenceHeadsSealedAtInsert(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
_, fid, s := evidenceFixture(t, d, "insert", false)
@@ -106,7 +106,7 @@ func TestEvidenceHeadsSealedAtInsert(t *testing.T) {
func TestEvidenceHeadsBackfillAndKeyChecks(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
_, fid, s := evidenceFixture(t, d, "legacy", true)
@@ -171,7 +171,7 @@ func TestEvidenceHeadsBackfillAndKeyChecks(t *testing.T) {
func TestEvidenceHeadsArchiveRoundTrip(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
if err = d.EnsureLLMRecordsTable(); err != nil {
+3 -3
View File
@@ -8,7 +8,7 @@ import (
func TestInheritedActivityReadsRequireTerminalIntent(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
@@ -92,7 +92,7 @@ func TestInheritedActivityReadsRequireTerminalIntent(t *testing.T) {
func TestExplorationDirectSourceReadView(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
@@ -315,7 +315,7 @@ func TestExplorationDirectSourceReadView(t *testing.T) {
func TestTaskAssetContextUsesDirectSourceScopeAndAnchors(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
+3 -3
View File
@@ -8,7 +8,7 @@ import (
func TestExplorationFlow(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -119,7 +119,7 @@ func TestExplorationFlow(t *testing.T) {
func TestIntentPauseResumeAndCancelCleanup(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -252,7 +252,7 @@ func TestIntentPauseResumeAndCancelCleanup(t *testing.T) {
func TestNodesPageQueryMatchesID(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+1 -1
View File
@@ -8,7 +8,7 @@ import (
func TestTokenStatsBySessionUsesCompleteIntentHistory(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+2 -2
View File
@@ -53,7 +53,7 @@ func nodeByKey(tree *FindingAssetTree, key string) *FindingAssetNode {
func TestBuildFindingAssetTree(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -164,7 +164,7 @@ func TestBuildFindingAssetTree(t *testing.T) {
func TestFindingAssetScopeFilter(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+1 -1
View File
@@ -10,7 +10,7 @@ import "testing"
func TestFindingQualityByNodeID(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
tk, err := d.CreateTask("finding quality lookup", "goal", nil, 0, 0)
+1 -1
View File
@@ -10,7 +10,7 @@ import (
func TestFindingReasoningFieldsRoundTrip(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
tk, err := d.CreateTask("finding reasoning fields", "goal", nil, 0, 0)
+30 -3
View File
@@ -8,6 +8,8 @@ import (
"encoding/json"
"errors"
"fmt"
"slices"
"strconv"
"strings"
"time"
@@ -85,6 +87,8 @@ type TrafficEvidenceSnapshot struct {
RespHash string `json:"resp_hash"`
ReqLen int64 `json:"req_len"`
RespLen int64 `json:"resp_len"`
Rewritten bool `json:"rewritten,omitempty"`
RewriteRuleIDs []int64 `json:"rewrite_rule_ids,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
@@ -131,11 +135,34 @@ func TrafficSnapshotID(snapshot TrafficEvidenceSnapshot) string {
snapshot = snapshot.Normalize()
snapshot.ID = ""
snapshot.CreatedAt = time.Time{}
snapshot.Rewritten = false
snapshot.RewriteRuleIDs = nil
raw, _ := json.Marshal(snapshot)
sum := sha256.Sum256(raw)
return hex.EncodeToString(sum[:])
}
func sortedUniqueInt64(ids []int64) []int64 {
if len(ids) == 0 {
return nil
}
out := slices.Clone(ids)
slices.Sort(out)
return slices.Compact(out)
}
func formatBigintArray(ids []int64) string {
ids = sortedUniqueInt64(ids)
if len(ids) == 0 {
return "{}"
}
parts := make([]string, len(ids))
for i, id := range ids {
parts[i] = strconv.FormatInt(id, 10)
}
return "{" + strings.Join(parts, ",") + "}"
}
// LockTaskEvidenceTx serializes writes with archive queueing (which locks the
// same task row). Once queued, its snapshot must not acquire new evidence.
func LockTaskEvidenceTx(tx *sql.Tx, taskID int64) error {
@@ -205,9 +232,9 @@ func InsertEvidenceSnapshotTx(tx *sql.Tx, s TrafficEvidenceSnapshot) error {
return err
}
_, err = tx.Exec(`INSERT INTO traffic_evidence_snapshots
(id,source_traffic_id,captured_at,url,method,status,content_type,req_head,resp_head,req_hash,resp_hash,req_len,resp_len,created_at)
VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14) ON CONFLICT(id) DO NOTHING`,
s.ID, s.SourceTrafficID, s.CapturedAt, s.URL, s.Method, s.Status, s.ContentType, reqHead, respHead, s.ReqHash, s.RespHash, s.ReqLen, s.RespLen, s.CreatedAt)
(id,source_traffic_id,captured_at,url,method,status,content_type,req_head,resp_head,req_hash,resp_hash,req_len,resp_len,created_at,rewritten,rewrite_rule_ids)
VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16::bigint[]) ON CONFLICT(id) DO NOTHING`,
s.ID, s.SourceTrafficID, s.CapturedAt, s.URL, s.Method, s.Status, s.ContentType, reqHead, respHead, s.ReqHash, s.RespHash, s.ReqLen, s.RespLen, s.CreatedAt, s.Rewritten, formatBigintArray(s.RewriteRuleIDs))
return err
}
+7 -7
View File
@@ -35,7 +35,7 @@ func TestValidFindingReportState(t *testing.T) {
func TestDeleteFinding(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -81,7 +81,7 @@ func TestDeleteFinding(t *testing.T) {
func TestSetFindingStatusMirrorsNodePayload(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -144,7 +144,7 @@ func TestSetFindingStatusMirrorsNodePayload(t *testing.T) {
func TestFindingReportState(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -256,7 +256,7 @@ func TestFindingReportState(t *testing.T) {
func TestFindingsPageAndStats(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -414,7 +414,7 @@ func TestFindingsPageAndStats(t *testing.T) {
func TestListFindingsPageUsesStableIDTieBreaker(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -462,7 +462,7 @@ func TestListFindingsPageUsesStableIDTieBreaker(t *testing.T) {
func TestFindingGroupsAndUnassignedPaging(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -606,7 +606,7 @@ func TestFindingGroupsAndUnassignedPaging(t *testing.T) {
func TestAddFindingFollowUpIntent(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+2 -2
View File
@@ -16,7 +16,7 @@ func openClaimInstances(t *testing.T) (*DB, *ExplorationStore, *ExplorationStore
dsn := testDSN(t)
a, err := Open(dsn)
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { a.Close() })
b, err := Open(dsn)
@@ -341,7 +341,7 @@ func TestIntentClaimTakeoverRacesHeartbeatAtomically(t *testing.T) {
func TestIntentClaimColumnsMigrateIdempotently(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+3 -3
View File
@@ -11,7 +11,7 @@ import (
func TestCompareAndSetIntentStateAllowsSingleWinner(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
expID, err := d.CreateExploration("intent CAS", "only one controller wins")
@@ -61,7 +61,7 @@ func TestCompareAndSetIntentStateAllowsSingleWinner(t *testing.T) {
func TestCancelIntentPreservesTokenMeteringWithoutDoubleCount(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
expID, err := d.CreateExploration("cancel token rollup", "preserve consumed tokens")
@@ -229,7 +229,7 @@ func assertDailyTokenBucketsEqual(t *testing.T, got, want map[string]DailyTokenB
func TestCancelIntentPreservesOutputsYieldedByAnotherIntent(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
expID, err := d.CreateExploration("shared intent output", "preserve shared facts and findings")
+3 -3
View File
@@ -41,7 +41,7 @@ func gone(t *testing.T, es *ExplorationStore, id int64) bool {
func TestSoftDeleteIntent(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
expID, err := d.CreateExploration("soft delete", "soft delete")
@@ -89,7 +89,7 @@ func TestSoftDeleteIntent(t *testing.T) {
func TestHardDeleteCascadesExclusiveDescendants(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
expID, err := d.CreateExploration("hard cascade", "cascade delete")
@@ -127,7 +127,7 @@ func TestHardDeleteCascadesExclusiveDescendants(t *testing.T) {
func TestHardDeletePreservesSharedAndGoal(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
expID, err := d.CreateExploration("hard preserve", "keep shared/goal")
+50
View File
@@ -0,0 +1,50 @@
package db
import "fmt"
// seedPersistenceApprovalRules inserts the built-in approval rules for the platform
// tools that store something the agents load or run later (custom tools, MCP
// servers, skill files). Each call of those tools is an ask: the operator approves it
// in the console. Seeded once, behind its own settings flag, like the other built-in
// seeds, so a later disable by the operator is never undone on restart. No agent tool
// reads or edits intercept rules, so an agent cannot lower them.
func (d *DB) seedPersistenceApprovalRules() error {
if v, _, _ := d.GetSetting("intercept_persistence_rules_v1"); v == "done" {
return nil
}
rules := []struct {
name string
pattern string
message string
}{
{
name: "[built-in] Persistent change: custom tool",
pattern: `^(?:create_custom_tool|update_custom_tool)$`,
message: "Approval needed: the agent asks to create or change a custom tool. It is stored on the platform and its code runs on this server in later calls. Approve only code you have read.",
},
{
name: "[built-in] Persistent change: MCP server",
pattern: `^(?:create_mcp|update_mcp)$`,
message: "Approval needed: the agent asks to add or change an MCP server. A stdio entry starts its command on this server; an HTTP or SSE entry sends its headers to its URL.",
},
{
name: "[built-in] Persistent change: skill",
pattern: `^(?:create_skill|update_skill_file)$`,
message: "Approval needed: the agent asks to create or change a skill file. Every agent that sees the skill loads its content as instructions.",
},
}
// intercept_rules.name has no unique index, so ON CONFLICT cannot apply: the insert
// is conditional on the name instead. A start that stopped between these inserts
// and SetSetting re-runs them without duplicating a rule.
for _, r := range rules {
if _, err := d.Exec(`
INSERT INTO intercept_rules(name, enabled, priority, match_target, match_type, pattern, action, message, timeout_enabled, timeout_seconds, timeout_action)
SELECT $1, true, 95, 'tool_name', 'regex', $2, 'ask', $3, false, 60, 'deny'
WHERE NOT EXISTS (SELECT 1 FROM intercept_rules WHERE name = $1)`,
r.name, r.pattern, r.message,
); err != nil {
return fmt.Errorf("rule %q: %w", r.name, err)
}
}
return d.SetSetting("intercept_persistence_rules_v1", "done")
}
+34
View File
@@ -0,0 +1,34 @@
package db
import "testing"
// A start that stopped after the rule inserts and before the settings flag runs the
// seed again on the next start: it must succeed and must not duplicate a rule.
func TestSeedPersistenceApprovalRulesIsIdempotent(t *testing.T) {
d := openTargetDB(t) // Open already ran the seed once
names := []string{
"[built-in] Persistent change: custom tool",
"[built-in] Persistent change: MCP server",
"[built-in] Persistent change: skill",
}
for range 2 {
if _, err := d.Exec(`DELETE FROM settings WHERE key = 'intercept_persistence_rules_v1'`); err != nil {
t.Fatal(err)
}
if err := d.seedPersistenceApprovalRules(); err != nil {
t.Fatalf("seed re-run failed: %v", err)
}
}
for _, name := range names {
var n int
if err := d.QueryRow(`SELECT count(*) FROM intercept_rules WHERE name = $1`, name).Scan(&n); err != nil {
t.Fatal(err)
}
if n != 1 {
t.Errorf("%q: %d rows, want 1", name, n)
}
}
if v, _, _ := d.GetSetting("intercept_persistence_rules_v1"); v != "done" {
t.Errorf("settings flag = %q, want done", v)
}
}
+12 -12
View File
@@ -76,7 +76,7 @@ func TestRenderLessonsUsageOmitsTruncated(t *testing.T) {
func TestTaskLessonsUsesTaskScopeAndRanksCompanyFirst(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -174,7 +174,7 @@ func TestTaskLessonsUsesTaskScopeAndRanksCompanyFirst(t *testing.T) {
func TestExtractLessonsScopesSingleCompanyFromTaskScope(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -226,7 +226,7 @@ func TestExtractLessonsScopesSingleCompanyFromTaskScope(t *testing.T) {
func TestExtractLessonsSkipsMultiCompanyTask(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -273,7 +273,7 @@ func TestExtractLessonsSkipsMultiCompanyTask(t *testing.T) {
func TestExtractLessonsGenericWhenTaskHasNoCompany(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -311,7 +311,7 @@ func TestExtractLessonsGenericWhenTaskHasNoCompany(t *testing.T) {
func TestTaskLessonsRanksCategoryBeforeGeneric(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -377,7 +377,7 @@ func TestTaskLessonsRanksCategoryBeforeGeneric(t *testing.T) {
func TestLogLessonUsageRecordsOnlyRendered(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -457,7 +457,7 @@ func insertLessonFinding(t *testing.T, d *DB, taskID int64, status string) int64
func TestExtractLessonsRetractsReversedDisposition(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -569,7 +569,7 @@ func TestExtractLessonsRetractsReversedDisposition(t *testing.T) {
func TestExtractLessonsRetireOnlyWhenLastEvidence(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -660,7 +660,7 @@ func TestExtractLessonsRetireOnlyWhenLastEvidence(t *testing.T) {
func TestExtractLessonsRetractsAllRetests(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -731,7 +731,7 @@ func TestExtractLessonsRetractsAllRetests(t *testing.T) {
func TestExtractLessonsDoesNotUnretireWithEvidence(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -777,7 +777,7 @@ func TestExtractLessonsDoesNotUnretireWithEvidence(t *testing.T) {
func TestExtractLessonsLeavesAdopted(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -820,7 +820,7 @@ func TestExtractLessonsLeavesAdopted(t *testing.T) {
func TestExtractLessonsReopenRetractsNothing(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
+1 -1
View File
@@ -12,7 +12,7 @@ import (
func TestLLMRecordsMigrateAddsRawColumnsToOldTable(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
if err := d.EnsureLLMRecordsTable(); err != nil {
+1 -1
View File
@@ -12,7 +12,7 @@ import (
func TestLLMUsageTsIndex(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
+3 -3
View File
@@ -13,7 +13,7 @@ import (
func TestProfileRetryRoundTrip(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -71,7 +71,7 @@ func TestProfileRetryRoundTrip(t *testing.T) {
func TestProfileRetryClampedOnSave(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -98,7 +98,7 @@ func TestProfileRetryClampedOnSave(t *testing.T) {
func TestLLMRetryPolicyRoundTrip(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the restore below.
t.Cleanup(func() { d.Close() })
+6 -5
View File
@@ -52,11 +52,11 @@ func scratchDSN(t *testing.T, suffix string) string {
admin.Path = "/postgres"
conn, err := sql.Open("pgx", admin.String())
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
if err := conn.Ping(); err != nil {
conn.Close()
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
drop := func() error {
_, err := conn.Exec(`DROP DATABASE IF EXISTS "` + name + `" WITH (FORCE)`)
@@ -137,9 +137,10 @@ func assertEmbeddedApplied(t *testing.T, d *DB) []appliedMigration {
version int
name string
}{{1, llmRecordsMigration}, {2, llmUsageMigration}, {3, "0003_task_scope_icp_index.sql"}, {4, "0004_llm_usage_ts_index.sql"},
{5, "0005_traffic_rewrite_rules.sql"}, {6, "0006_activity_outbox.sql"}, {7, "0007_finding_report_state.sql"}}
{5, "0005_traffic_rewrite_rules.sql"}, {6, "0006_activity_outbox.sql"}, {7, "0007_finding_report_state.sql"},
{8, "0008_traffic_evidence_rewrite_flag.sql"}}
if len(applied) != len(want) {
t.Fatalf("schema_migrations = %+v, want versions 1 to 7", applied)
t.Fatalf("schema_migrations = %+v, want versions 1 to 8", applied)
}
for i, w := range want {
content, err := fs.ReadFile(migrationsFS(), w.name)
@@ -189,7 +190,7 @@ func TestOpenRecordsEmbeddedMigrations(t *testing.T) {
dsn := testDSN(t)
d, err := Open(dsn)
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
applied := assertEmbeddedApplied(t, d)
d.Close()
@@ -0,0 +1,5 @@
-- Provenance flag on evidence snapshots recording whether the capture proxy
-- applied at least one rewrite rule to the bound exchange, and which rule IDs
-- were applied. Excluded from the content-addressed snapshot ID.
ALTER TABLE traffic_evidence_snapshots ADD COLUMN IF NOT EXISTS rewritten boolean NOT NULL DEFAULT false;
ALTER TABLE traffic_evidence_snapshots ADD COLUMN IF NOT EXISTS rewrite_rule_ids bigint[] NOT NULL DEFAULT '{}'::bigint[];
+1 -1
View File
@@ -18,7 +18,7 @@ func notifyTestDB(t *testing.T) *DB {
t.Helper()
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
return d
+3 -3
View File
@@ -200,7 +200,7 @@ func assertPlainReads(t *testing.T, d *DB, r secretRows) {
func TestSecretsAtRestWriteRead(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() }) // after the fixture cleanup
r := secretFixture(t, d, false)
@@ -214,7 +214,7 @@ func TestSecretsAtRestWriteRead(t *testing.T) {
func TestSecretsBackfillLegacyAndKeyChecks(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() }) // after the fixture cleanup
r := secretFixture(t, d, true)
@@ -287,7 +287,7 @@ func encryptedCount(t *testing.T, d *DB, dir string) int {
func TestSecretSettingsEmptyAndClear(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() {
d.Exec(deleteSecretSettings)
+2 -2
View File
@@ -13,7 +13,7 @@ import (
func TestInsertSettingIfAbsentDoesNotOverwrite(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -50,7 +50,7 @@ func TestInsertSettingIfAbsentDoesNotOverwrite(t *testing.T) {
func TestRotatePasswordRollsBackHashWhenVersionCannotIncrement(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+1 -1
View File
@@ -11,7 +11,7 @@ import (
func TestSkillUsageLedger(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+1 -1
View File
@@ -19,7 +19,7 @@ func openTargetDB(t *testing.T) *DB {
t.Helper()
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
return d
+3 -3
View File
@@ -123,7 +123,7 @@ func TestTaskArchiveFormatCompatibility(t *testing.T) {
func TestTaskArchiveDatabaseRoundTrip(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
if err := d.EnsureLLMRecordsTable(); err != nil {
@@ -335,7 +335,7 @@ VALUES($1,$2,0,'quota_exhausted','balance exhausted',$3,$4,$3)`, task.ID, llmPro
func TestTaskArchiveBlockersIgnoreQueuedDependents(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
source, err := d.CreateTaskWithOptions("archive blocker source", "source", TaskCreateOptions{})
@@ -400,7 +400,7 @@ func assertPerTaskBlockers(t *testing.T, d *DB, global map[int64]int64, want map
func TestRecoverInterruptedArchiveRequiresManualRetry(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
task, err := d.CreateTaskWithOptions("interrupted archive", "must not restart automatically", TaskCreateOptions{})
+3 -3
View File
@@ -10,7 +10,7 @@ import (
func TestRegisterTaskAssetScopesCreatesAssetsAndPersistsTextScope(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
@@ -93,7 +93,7 @@ func TestRegisterTaskAssetScopesCreatesAssetsAndPersistsTextScope(t *testing.T)
func TestTaskAssetAttachDetachPreservesGlobalAssetAndAnchors(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
@@ -150,7 +150,7 @@ func TestTaskAssetAttachDetachPreservesGlobalAssetAndAnchors(t *testing.T) {
func TestIntentAssetsIncludesDirectSourceProvenance(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
+4 -4
View File
@@ -11,7 +11,7 @@ import (
func TestTaskCategoryCRUDAndTaskAssignment(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -73,7 +73,7 @@ func TestTaskCategoryCRUDAndTaskAssignment(t *testing.T) {
func TestCreateTaskRejectsMissingCategoryAtomically(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -94,7 +94,7 @@ func TestCreateTaskRejectsMissingCategoryAtomically(t *testing.T) {
func TestSetTasksCategoryBatch(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
@@ -155,7 +155,7 @@ func TestSetTasksCategoryBatch(t *testing.T) {
func TestSetTasksCategoryRejectsBadInputAtomically(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
+7 -7
View File
@@ -11,7 +11,7 @@ func TestTaskLLMProfileMutationsLockTaskBeforeProfile(t *testing.T) {
dsn := testDSN(t)
d, err := Open(dsn)
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -111,7 +111,7 @@ func TestDeleteProfileLocksNonTaskReferencesBeforeProfile(t *testing.T) {
t.Run("agent", func(t *testing.T) {
d, err := Open(dsn)
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -169,7 +169,7 @@ func TestDeleteProfileLocksNonTaskReferencesBeforeProfile(t *testing.T) {
t.Run("conversation", func(t *testing.T) {
d, err := Open(dsn)
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -226,7 +226,7 @@ func TestNonTaskProfileMutationsLockReferenceBeforeProfile(t *testing.T) {
t.Run("agent", func(t *testing.T) {
d, err := Open(dsn)
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -275,7 +275,7 @@ func TestNonTaskProfileMutationsLockReferenceBeforeProfile(t *testing.T) {
t.Run("conversation", func(t *testing.T) {
d, err := Open(dsn)
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -326,7 +326,7 @@ func TestCreateConversationAndDeleteProfileDoNotDeadlock(t *testing.T) {
dsn := testDSN(t)
d, err := Open(dsn)
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -407,7 +407,7 @@ func TestDeleteProfileRetriesReferenceCommittedAfterInitialScan(t *testing.T) {
dsn := testDSN(t)
d, err := Open(dsn)
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
+1 -1
View File
@@ -13,7 +13,7 @@ func TestDeleteTaskTrafficHostsUseOneLockedTransaction(t *testing.T) {
dsn := testDSN(t)
d, err := Open(dsn)
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+3 -3
View File
@@ -9,7 +9,7 @@ import (
func TestListTasksBulkHydratesTaskContext(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
@@ -71,7 +71,7 @@ func TestListTasksBulkHydratesTaskContext(t *testing.T) {
func TestTaskListMetricsAll(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -123,7 +123,7 @@ VALUES ($1,'goal','{}','met'), ($1,'goal','{}','open')`, task.ExplorationID); er
func TestTaskListMetricsMatchesAll(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
+1 -1
View File
@@ -11,7 +11,7 @@ import (
func TestTaskLLMSpendCountsJudgeNotChat(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+2 -2
View File
@@ -9,7 +9,7 @@ import (
func TestTaskPinOrderingAndRename(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
@@ -71,7 +71,7 @@ func TestTaskPinOrderingAndRename(t *testing.T) {
func TestDeleteConversationsReturnsExistingIDs(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+1 -1
View File
@@ -8,7 +8,7 @@ import (
func TestTaskQueuePreservesBootstrapAndFIFOPosition(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+1 -1
View File
@@ -9,7 +9,7 @@ import (
func TestTaskRotationPersistenceAndFailover(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable: %v", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
ids := []int64{}
+2 -2
View File
@@ -12,7 +12,7 @@ func TestCoverageEnabledWithoutTask(t *testing.T) {
func TestCoverageEnabledMissingRowStaysOn(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
on, err := d.Assets().CoverageEnabled(9_000_000_000_001)
@@ -24,7 +24,7 @@ func TestCoverageEnabledMissingRowStaysOn(t *testing.T) {
func TestCoverageEnabledQueryErrorStaysOn(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
SkipPGUnavailable(t, err)
}
if err := d.Close(); err != nil {
t.Fatal(err)
+2 -2
View File
@@ -12,7 +12,7 @@ import (
func TestTaskTemplateCRUDAndNormalizedUniqueness(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
@@ -87,7 +87,7 @@ func TestTaskTemplateCRUDAndNormalizedUniqueness(t *testing.T) {
func TestTaskTemplateDisjointPatchesCompose(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanup below.
t.Cleanup(func() { d.Close() })
+6 -6
View File
@@ -11,7 +11,7 @@ import (
func TestTaskLifecycleAndDeleteCascade(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() { d.Close() })
@@ -85,7 +85,7 @@ func TestTaskLifecycleAndDeleteCascade(t *testing.T) {
func TestTaskDeleteCascadeAssets(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
@@ -211,7 +211,7 @@ func TestTaskDeleteCascadeAssets(t *testing.T) {
func TestTaskRelationsAndLLMFailoverChain(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
@@ -470,7 +470,7 @@ func TestTaskRelationsAndLLMFailoverChain(t *testing.T) {
func TestTaskContextRejectsDuplicatesAndAllowsTerminalLLMEdits(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
@@ -525,7 +525,7 @@ func TestTaskContextRejectsDuplicatesAndAllowsTerminalLLMEdits(t *testing.T) {
func TestCreateTaskWithCompanyScopes(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
@@ -680,7 +680,7 @@ RETURNING id`, domainB, companyB).Scan(&companyAssetB); err != nil {
func TestListTasksOrderByIDDesc(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
// Registered FIRST so it runs LAST, after the row cleanups below.
t.Cleanup(func() { d.Close() })
+33
View File
@@ -0,0 +1,33 @@
package db
// skipper is the part of testing.TB used by the PostgreSQL test helpers below.
// A local interface keeps the production package from importing "testing":
// *testing.T, *testing.B and the test-only wrappers all satisfy it.
type skipper interface {
Helper()
Skip(args ...any)
Skipf(format string, args ...any)
}
// SkipPGUnavailable skips the calling test with one uniform reason when err
// reports that the test database could not be configured or opened.
//
// Every PostgreSQL-dependent test uses this instead of spelling the skip text
// inline (the old variants differed by package: "no database config",
// "database unavailable", "postgres unavailable — skipping", ...). Call it
// with the err returned by DSN(), Open() or NewManager(). It does not change
// when a test skips — only how the reason is reported.
func SkipPGUnavailable(t skipper, err error) {
t.Helper()
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
}
}
// SkipNoPG skips with the same uniform reason when a store is unexpectedly nil
// after a successful open (for example Manager.pg when only some backends are
// configured). It carries no error because there is none to report.
func SkipNoPG(t skipper) {
t.Helper()
t.Skip("postgres unavailable")
}
+1 -1
View File
@@ -5,7 +5,7 @@ import "testing"
func TestToolUsageLedger(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
SkipPGUnavailable(t, err)
}
defer d.Close()
+2 -2
View File
@@ -77,7 +77,7 @@ func jsonEqual(t *testing.T, a, b string) bool {
func TestToolExecSealedAtRest(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() {
d.Exec(`DELETE FROM tools WHERE key LIKE 't_secexec%'`)
@@ -129,7 +129,7 @@ func TestToolExecSealedAtRest(t *testing.T) {
func TestToolExecBackfillAndKeyChecks(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
t.Cleanup(func() {
d.Exec(`DELETE FROM tools WHERE key LIKE 't_secexec%'`)
+1 -1
View File
@@ -26,7 +26,7 @@ func openRewriteRules(t *testing.T) *DB {
t.Helper()
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
SkipPGUnavailable(t, err)
}
clean := func() { d.Exec(`DELETE FROM traffic_rewrite_rules WHERE name LIKE 't-rw-%'`) }
clean()
+4
View File
@@ -23,6 +23,10 @@ services:
args:
ARTEX_VERSION: ${ARTEX_VERSION:-dev}
image: artex:local
# Agent processes run as an unprivileged executor (-exec-uid): no setuid program
# or file capability may raise them back to root.
security_opt:
- "no-new-privileges:true"
depends_on:
postgres:
condition: service_healthy
+112
View File
@@ -0,0 +1,112 @@
# ARTEX architecture (code-level companion)
The README's [Architecture](https://github.com/BaaRai/ARTEX#architecture) section has the
component diagram and the 10,000-foot view. This document records the code-level map,
the execution flow and the trust boundaries that are easy to miss when reading the
packages in isolation.
## 1. The two graphs
ARTEX keeps two linked graphs, both in PostgreSQL (`db/schema.sql`):
| Graph | Answers | Main tables | Code |
| --- | --- | --- | --- |
| **Asset graph** | *What exists?* | `companies`, `company_scope`, `assets`, `task_scope` | `db/assets.go`, `db/companies.go`, `db/company_scope.go`, `db/task_assets.go` |
| **Exploration graph** | *What was tried, with what result and why?* | `explorations`, `exploration_nodes`, `activity`, `intent_*`, `coverage_*` | `db/exploration.go`, `db/coverage_ledger.go`, `db/finding_assets.go` |
Anchors tie a finding to the assets that justify it (`db/finding_assets.go`). Traffic
snapshots are archived separately and referenced from findings
(`db/finding_traffic.go`, `db/finding_traffic_archive.go`, `traffic/archive.go`).
`db.Open` executes the idempotent `schema.sql` on every startup, then applies pending
numbered migrations from `db/migrations/*.sql`, recording name + SHA-256 in
`schema_migrations`. A published migration is never edited; fix forward with a new one.
## 2. Package map
| Package | Responsibility | Notes |
| --- | --- | --- |
| `cmd/artex` | Process entry point | Flags (`-addr`, `-data`, `-proxy`), startup config snapshot, signal handling, graceful drain |
| `config` | Resolve config file + env | `Path`, `Load`, `PostgresDSN`, `SkillDir`, `KeyDir`, `Redact`. Env wins over file |
| `db` | All persistence | `DB`, `Manager`-independent stores; AES-256-GCM secret columns in `secrets.go` |
| `agent` | LLM roles and tools | `goals`, `planner`, `worker`, `mainagent`, `retester`; `ToolSet` in `tools.go`; tool catalog in `toolcatalog.go` |
| `server` | HTTP API + orchestration | `net/http` mux (~270 routes), `Manager`, `Engine`, JWT auth, SSE broadcaster |
| `traffic` | Recording proxy | MITM proxy built on `go-mitmproxy`, replay, rewrite rules, body archive |
| `evidence` | Evidence store | Persisted request/response snapshots used by findings |
| `intercept` | Approval gate | Judge, argv grammar, target gate, pins, risk scoring (a `PreToolUse` hook) |
| `guard` | Hook wiring | Binds `intercept` decisions into Norma's hook registry; `NewTargetOnly` for planner/mainagent |
| `egress` | Noise governor | Per-host token bucket, jitter, concurrency caps, 429/503 backoff; default profile is strict passthrough |
| `llmpool` / `llmrec` | Provider pool / recorder | Rotation, health, usage + raw wire capture |
| `mcphttp` | MCP client | HTTP/SSE transport for configured MCP servers |
| `notify` | Outbound notifications | Email, webhook, Telegram, Feishu, DingTalk with masking and SSRF guards |
| `report` | Deliverables | Markdown, CSV and JSON rendering of findings |
| `sidequestion` | Operator Q&A | Asks a human mid-run and suspends intent until answered |
| `skills/` | Skill definitions | One `SKILL.md` per subdirectory, loaded from `SkillDir()` |
The frontend lives in `web/` (Next.js 16, React 19) and is embedded into the Go
binary via `server/webui_embed.go` when built with `-tags embedui`.
## 3. End-to-end flow: goal → finding with evidence
1. **Task creation.** `server.createTask` → `Manager.CreateTaskWithOptions` writes the
task, its scope and seed goals, and creates the exploration graph (`db/tasks.go`).
2. **Loop start.** `Engine.Run` (`server/engine.go`) starts a planner loop and N worker
goroutines under a task-scoped context. Nothing needs an LLM to be configured for the
loops to exist; they idle until one is.
3. **Decomposition.** The `goals` role writes goals and operation constraints through
`set_goals` / `set_constraints` (`agent/toolcatalog.go`, `agent/tools.go`).
4. **Planning.** `Planner.Plan` (`agent/planner.go`) reads the route, queries the asset
graph, judges goals and emits intents (`addOneIntent`). Every intent is an
exploration node, so it is auditable like any other activity.
5. **Execution.** `Worker.Execute` (`agent/worker.go`) claims an intent, runs tools with
the guard hooks attached and writes observed facts back to the exploration graph.
6. **Tools and traffic.** Shell and HTTP/WebFetch calls go through the recording proxy
and the egress governor. Platform-owned HTTP clients use `intercept.PinnedDialer`
and `intercept.RedirectCheck`, so a redirect or a direct retry is re-checked at
dial time. When a proxy is in the path the gate records the pin as name-only: the
proxy makes the outbound connection, so no IP binding can be claimed.
7. **Findings.** `addFinding` (`agent/tools.go`) records severity, class, confidence and
asset references; `db/findings.go` dedups and keeps the status workflow. Evidence is
linked to captured traffic (`traffic/evidence.go`, `db/finding_traffic.go`).
8. **Completion.** `prove_goal` marks a goal met; the last met goal completes the task.
`goal_met` is deliberately unbound by default because it bypasses per-goal proof.
## 4. Execution control
- **Per-task context:** pause/cancel/kill attach a named cause via
`context.WithCancelCause` so the activity trace records who initiated a stop.
- **Per-intent control:** the UI can pause/cancel one intent; `steerBox` queues a
planner course-correction that the worker's `PreToolUse` hook drains before the next
tool call, re-planning without killing the run.
- **Delete barrier:** `Engine.BeginDelete` makes installing the barrier atomic with
registering an in-flight operation; after it returns, no new task-owned writes are
admitted and the task context is cancelled once existing writers drain.
- **Quiet mode:** an unchanged open-goal fingerprint suppresses redundant planning
rounds while work is still in flight.
## 5. Trust boundaries
| Boundary | Implementation |
| --- | --- |
| Tool output → model | Norma's untrusted-output boundary: nonce-tagged markers around tool results. On by default; `ARTEX_UNTRUSTED_BOUNDARY=0` disables it. See `agent/untrusted_boundary.go` |
| Command → host | Allow rules match one Bash simple command with the exact prefix: a pipeline, redirection, wrapper (`env`, `sudo`, `xargs`, `find -exec`, a shell `-c`), substitution or expansion does not inherit an allow. Deny and ask rules also inspect every simple command inside wrappers, lists and pipelines. `ArgvPrefixOf` derives the human-readable prefix |
| URL → network | Target gate + `PinnedDialer` + `RedirectCheck`: scope decisions are re-made against a fresh DNS lookup at dial time, and without a proxy the client connects only to the addresses it compared; with a proxy the pin is name-only |
| Secrets → disk/log | `db/secrets.go` seals API keys, proxy credentials, MCP env, task credentials and captured auth headers with AES-256-GCM, nonce per value, AAD `table.column`. JWT keys live in `KeyDir()`, never in the browsable data dir. The boot snapshot logs only `config.Redact(dsn)` |
| Approval → audit | Every decision records the approver (`jwtSubject`), source and decision; history is queryable through `/api/intercept/history` and `/api/intercept/history/{id}` (with `/execution` for the audit detail). `/api/tasks/{id}/intercept-rules` manages rules, not history |
## 6. Decisions
- **ADR-001 — Nonce-tagged untrusted output.** Tool output is framed with a per-run
nonce rather than trusting the model to distinguish instructions from data.
- **ADR-002 — Pin at dial time.** A URL decision is not enough: DNS can change between
the hook and the connection, so the dialer re-resolves and connects only to compared
addresses.
- **ADR-003 — AES-GCM with per-column AAD.** Sealed values are bound to their table and
column (and settings to their key), so a ciphertext cannot be moved to another field.
## 7. Where to look next
- `docs/OPS.md` for startup, configuration precedence, backup and key rotation.
- `docs/vulnerability-traffic-evidence.md` for how a finding keeps its receipts.
- `docs/traffic-rewrite-rules.md` for the rewrite engine.
- `scripts/verify.sh` to reproduce the CI checks locally.
+93
View File
@@ -0,0 +1,93 @@
# ARTEX operations
## 1. Startup
Docker (recommended):
```bash
cp .env.example .env # set POSTGRES_PASSWORD; optionally an LLM key
docker compose up -d --build
docker compose logs artex | grep "Setup token"
```
Open `http://localhost:8787`. The first visit to `/setup` asks for the one-time setup
token printed at startup (`[auth] no admin password is set. Setup token for the /setup
page: ...` in `server/auth.go`).
From source (no embedded UI):
```bash
go run ./cmd/artex # backend only, 127.0.0.1:8787
cd web && npm run dev # UI on :5173, /api proxied to the backend
# or: ./dev.sh # both, plus the traffic proxy on :8788
```
The process prints one `[config] snapshot {...}` line at startup. It is the authoritative
record of the effective configuration: config path, DSN source and **redacted** DSN,
data/skill/key directories, listen address, proxy, noise profile, untrusted-output
boundary and trusted proxies. No password or token value is ever logged there.
## 2. Configuration precedence
| Setting | Resolution order | Code |
| --- | --- | --- |
| Config file | `ARTEX_CONFIG` → `./config.json` → `config.json` next to the binary | `config.Path` |
| PostgreSQL DSN | `ARTEX_PG_DSN` → `database.dsn` → assembled `host/port/user/password/dbname` | `config.PostgresDSN` |
| Skill directory | `ARTEX_SKILL_DIR` → `skill_dir` → `<binary dir>/skills` | `config.SkillDir` |
| JWT / secret key directory | `ARTEX_JWT_KEY_DIR` → `<binary dir>` (never the browsable data dir) | `config.KeyDir` |
| Secret encryption key | `ARTEX_SECRET_KEY` → `<key dir>/secret.key` → newly generated key file | `db.InitSecrets` |
| Noise profile | `ARTEX_NOISE_PROFILE` (`off` default; `quiet`, `balanced`, `aggressive`), also settable in UI | `egress.ProfileFromEnv` |
| Untrusted-output boundary | on by default; `ARTEX_UNTRUSTED_BOUNDARY=0\|false\|off\|no` disables it | `agent/untrusted_boundary.go` |
| Trusted proxies for login rate limit | `ARTEX_TRUSTED_PROXIES` (comma-separated IPs/CIDRs; empty trusts none) | `server.ParseTrustedProxies` |
| Target user agent | `ARTEX_TARGET_USER_AGENT` (applied by non-`off` noise profiles) | `egress.ProfileByName` |
| Budgets | `ARTEX_TASK_BUDGET_TOKENS` / `_TURNS`, `ARTEX_INTENT_BUDGET_TOKENS` / `_TURNS` | `agent/budget.go` |
| Live model judge test | `ARTEX_REVIEW_LIVE_CONFIG` (opt-in integration test only) | `server/intercept_live_test.go` |
Agent profiles, providers, notifications and other runtime settings live in the database
and are edited in **System → LLM / Settings / Notify**, not in `config.json`.
## 3. Backup and restore
```bash
scripts/backup.sh # database dump only, 14 rotations
scripts/backup.sh --with-data --keep 30 # also archive the data/ bind mount
```
The dump runs inside the `postgres` compose service, so no host `psql` is needed. Restore:
```bash
gunzip -c backups/artex-db-<ts>.sql.gz | docker compose exec -T postgres \
psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"
```
Back up the key directory as well: without it, JWTs cannot be verified and sealed
secrets cannot be decrypted. Keep it outside the database backup but under the same
access controls.
## 4. Key rotation
- **`jwt.key`** — signing key for the 7-day HS256 session tokens. Replacing it
invalidates every existing session; all users must log in again. Existing key files
are never overwritten by startup (`loadOrCreateJWTKey`).
- **`secret.key`** (or `ARTEX_SECRET_KEY`) — AES-256-GCM key for API keys, MCP env,
task credentials and captured auth headers. Startup never overwrites an existing key.
If it changes, values sealed under the old key cannot be decrypted (the code returns
an error rather than leaking ciphertext as plaintext). Re-enter those secrets in the
UI, or keep the previous key available until the migration is complete.
Plan a short maintenance window for either rotation.
## 5. Diagnostics and local verification
- `GET /api/health` — liveness, build version and provider health.
- `GET /api/logs`, `GET /api/logs/stream` — live backend log sink; the UI shows it under
**System → Logs**. The startup snapshot line is here too.
- `scripts/verify.sh` — reproduces the CI checks locally (gofmt, build, vet, tests,
web type-check and unit tests, shellcheck, python checks). `--full` adds `-race` and
the web static build; `--go-only` / `--web-only` narrow the run.
- `./scripts/backup.sh --help` — backup options.
When Postgres is not configured, database-dependent tests skip themselves; that is
expected. Set `ARTEX_PG_DSN` to run them. CI uses `go test -p 1 ./...` because test
packages share a Postgres service and use suffixed per-package databases
(`SuffixTestDatabase`).
+25
View File
@@ -0,0 +1,25 @@
[Unit]
Description=ARTEX application
After=network.target
StartLimitIntervalSec=60
StartLimitBurst=5
[Service]
Type=simple
User=artex
Group=artex
WorkingDirectory=/opt/artex
Environment=ARTEX_JWT_KEY_DIR=/opt/artex/keys
ExecStart=/opt/artex/artex -addr 127.0.0.1:8787 -proxy 127.0.0.1:8788
Restart=on-failure
RestartSec=5s
TimeoutStopSec=30s
KillSignal=SIGTERM
KillMode=control-group
UMask=0002
StandardOutput=journal
StandardError=journal
SyslogIdentifier=artex
[Install]
WantedBy=multi-user.target
+178
View File
@@ -0,0 +1,178 @@
# Local ARTEX systemd deployment
This guide describes how to deploy ARTEX locally under `systemd` using the example unit at `docs/examples/artex.service`.
## Placeholders and defaults
The example unit uses these defaults; replace them if your host uses different paths or names:
- Install directory (`<install-dir>`): `/opt/artex` (executable at `/opt/artex/artex`).
- System user and group (`<artex-user>`): `artex` (non-root).
- PostgreSQL service unit (`<pg-unit>`): distribution-specific — for example `postgresql.service` on RHEL/Fedora/Arch or `postgresql@<pg-version>-<pg-cluster>.service` on Debian/Ubuntu (where `<pg-version>/<pg-cluster>` comes from `pg_lsclusters`, such as `18/main`).
- Configuration: `<install-dir>/config.json`, ignored by Git, owner-only mode `0600`.
- Web UI: `http://127.0.0.1:8787`; recording proxy: `127.0.0.1:8788`.
- Example unit file: `docs/examples/artex.service`; installed at `/etc/systemd/system/artex.service`.
## System user and directory permissions
Create a dedicated non-login system user and group before installing the binary:
```bash
sudo useradd --system --user-group --home-dir /opt/artex --shell /usr/sbin/nologin artex
```
Create the install, runtime data, and key directories with explicit ownership and modes:
```bash
sudo install -d -o root -g artex -m 0755 /opt/artex
sudo install -d -o artex -g artex -m 0770 /opt/artex/data
sudo install -d -o artex -g artex -m 0700 /opt/artex/keys
```
Directory ownership and modes:
- `<install-dir>` (`/opt/artex`): `root:<artex-user>`, mode `0755` (executable `/opt/artex/artex` mode `0755`).
- `<install-dir>/data` (`/opt/artex/data`): `<artex-user>:<artex-user>`, mode `0770` so the service user (and, in a root-run deployment using an executor group, that group) can write task run directories while blocking world access.
- `<install-dir>/keys` (`/opt/artex/keys`): `<artex-user>:<artex-user>`, mode `0700`. Holds `jwt.key` and `secret.key` (mode `0600`) outside the browsable `data/` directory. `docs/examples/artex.service` sets `Environment=ARTEX_JWT_KEY_DIR=/opt/artex/keys` so the non-root service writes key files here rather than into `root`-owned `/opt/artex` (if `ARTEX_JWT_KEY_DIR` were left unset, ARTEX would fall back to `<install-dir>` and fail to create `jwt.key` and `secret.key` when `<install-dir>` is `root:<artex-user>` `0755`).
- `<install-dir>/config.json` (`/opt/artex/config.json`): `<artex-user>:<artex-user>`, mode `0600`.
## Database and credentials
Install PostgreSQL on the host (or use an existing reachable PostgreSQL instance) and create a dedicated non-superuser login role and database named `artex` with no role-creation or database-creation privileges.
On Debian/Ubuntu, confirm the cluster name (`<pg-version>/<pg-cluster>`) with `pg_lsclusters`, start `<pg-unit>`, and create the role and database if absent:
```bash
sudo systemctl start <pg-unit>
sudo -u postgres createuser --pwprompt artex
sudo -u postgres createdb --owner=artex artex
```
Enter a unique password at the prompt. In `<install-dir>/config.json`, set `database.host` to `127.0.0.1`, `port` to `5432`, `user` and `dbname` to `artex`, `password` to that secret, and `sslmode` to `disable` (disabling TLS is appropriate only for loopback traffic). Preserve other settings and restrict access:
```bash
sudo chown artex:artex /opt/artex/config.json
sudo chmod 0600 /opt/artex/config.json
```
`ARTEX_PG_DSN`, if set in a systemd drop-in override, takes precedence over `config.json`. Never commit configuration, credentials, encryption/signing keys, or startup logs.
## Build and install
The binary embeds the UI when built with `-tags embedui`. To rebuild the UI from a source checkout, run `npm ci` and `npm run build:static` in `web/`, create the parent directory with `mkdir -p server/webui`, and replace `server/webui/dist` with `web/out`. Build the binary with:
```bash
CGO_ENABLED=0 go build -tags embedui -trimpath -o artex ./cmd/artex
```
Install the binary into `<install-dir>`:
```bash
sudo install -o root -g artex -m 0755 artex /opt/artex/artex
```
For a running deployment, build into a separate staging path and stop the service before replacing `/opt/artex/artex` or generated assets. Avoid rerunning `install.sh` without reviewing its prompts: it regenerates `config.json` and starts a separate process.
Review `User`, `Group`, `WorkingDirectory`, `Environment=ARTEX_JWT_KEY_DIR=...`, and `ExecStart` in `docs/examples/artex.service` before installing it on your host:
```bash
systemd-analyze verify docs/examples/artex.service
sudo install -o root -g root -m 0644 docs/examples/artex.service /etc/systemd/system/artex.service
sudo systemctl daemon-reload
```
### Database dependency and environment overrides
`docs/examples/artex.service` intentionally declares only `After=network.target` and does not hardcode a distribution-specific PostgreSQL unit (such as Debian's `postgresql@<pg-version>-<pg-cluster>.service`), so it works across Linux distributions and with remote PostgreSQL servers.
When PostgreSQL runs locally on the same host, add a systemd drop-in (`sudo systemctl edit artex.service`) so ARTEX starts after and depends on your local PostgreSQL unit:
```ini
[Unit]
Requires=<pg-unit>
After=<pg-unit>
```
For example, set `<pg-unit>` to `postgresql.service` on RHEL/Fedora/Arch, or to `postgresql@<pg-version>-<pg-cluster>.service` on Debian/Ubuntu (and verify `/etc/postgresql/<pg-version>/<pg-cluster>/start.conf` is `auto`, not `disabled`). `ARTEX_JWT_KEY_DIR=/opt/artex/keys` is already set in `docs/examples/artex.service`; override it in the drop-in only if you install ARTEX under a different `<install-dir>`. Run `sudo systemctl daemon-reload` after editing the drop-in.
### Enable and start
If ARTEX is running outside `systemd`, identify its PID/executable and send that verified process `SIGTERM`. Wait for shutdown and release of ports `8787` and `8788` before enabling the service:
```bash
ss -ltnp '( sport = :8787 or sport = :8788 )'
sudo systemctl enable --now artex.service
systemctl status artex.service --no-pager
curl --fail --silent --show-error http://127.0.0.1:8787/api/health
```
The unit runs `/opt/artex/artex` directly without an extra supervisor script. It starts after `network.target`, restarts on failure after five seconds, allows 30 seconds for graceful shutdown, logs to the journal, and uses `UMask=0002` (rather than `0077`, which would block group writes in a root-run deployment where executor privilege drop sets umask `002` for the executor group; sensitive files such as `config.json`, `keys/jwt.key`, and `keys/secret.key` are protected explicitly with `0600`/`0700` permissions). Five starts within 60 seconds trigger the start limit.
### Agent isolation
Because `docs/examples/artex.service` runs the server as the non-root `<artex-user>` (`User=artex`), the `-exec-uid`/`-exec-gid` privilege drop cannot be applied: `ResolveExecPrivilege` disables the drop when the server is not root (and errors if those flags are passed explicitly). The example unit deliberately omits `-exec-uid`/`-exec-gid`, so agents and spawned tools run under the `<artex-user>` service identity rather than a separate executor account. The root-owned run-directory layout and executor-group containment exist only when the server runs as root with explicit `-exec-uid`/`-exec-gid`; if you require unprivileged-executor containment, use the Docker deployment, which is the supported path for that isolation model.
## Web setup
On a new database, visit `http://127.0.0.1:8787/setup`, provide the one-time setup token from the private startup log or journal (`journalctl -u artex.service`), and choose a unique password of at least 12 characters. The username is `ARTEX`. Store the password in your password manager rather than in a plaintext file on disk.
## Reboot persistence verification (example)
Pre-reboot checks confirm persistent ARTEX enablement, active PostgreSQL and ARTEX services, HTTP 200 health, and initialized authentication, but do not by themselves prove survival across a reboot.
To verify reboot persistence on a host, record the current boot ID and timestamp in an ignored runtime file such as `<install-dir>/data/reboot-verification-baseline.json` before rebooting, and do not overwrite that baseline until the test is complete:
```bash
sudo -u artex python3 -c 'import json, time; from pathlib import Path; Path("/opt/artex/data/reboot-verification-baseline.json").write_text(json.dumps({"boot_id": Path("/proc/sys/kernel/random/boot_id").read_text().strip(), "ts": time.time()}))'
```
Save other work before rebooting; this disconnects local and remote sessions:
```bash
sudo systemctl reboot
```
After reconnecting, do not manually start ARTEX before checking automatic startup. Compare the current boot ID with the saved baseline and check the services:
```bash
cd /opt/artex
python3 -c 'import json; from pathlib import Path; old=json.loads(Path("data/reboot-verification-baseline.json").read_text())["boot_id"]; new=Path("/proc/sys/kernel/random/boot_id").read_text().strip(); print("PASS: a reboot occurred" if old != new else "FAIL: no reboot occurred"); raise SystemExit(0 if old != new else 1)'
systemctl is-enabled artex.service
systemctl is-active <pg-unit> artex.service
curl --fail --silent --show-error http://127.0.0.1:8787/api/health
python3 -c 'import json, urllib.request; r=json.load(urllib.request.urlopen("http://127.0.0.1:8787/api/auth/status", timeout=5)); print("Authentication initialized:", r.get("initialized")); raise SystemExit(0 if r.get("initialized") else 1)'
systemctl show artex.service --no-pager -p ActiveState -p SubState -p MainPID -p ActiveEnterTimestamp
```
Success requires a changed boot ID, enabled ARTEX, both services active without manual startup, HTTP 200, and authentication still initialized. A new PID alone is not proof of reboot. If the system has just booted, allow startup time and repeat health checks. Refresh the baseline before running a future independent reboot test.
### Example reboot verification record
Example of a completed post-reboot verification (recorded from the original Debian/PostgreSQL 18 reference deployment on 2026-11-17, without manually starting or restarting either service):
- Current boot ID differed from the saved pre-reboot baseline, confirming an actual reboot.
- `artex.service` remained enabled and was `active (running)` under its non-root service user with `NRestarts=0`.
- The local PostgreSQL cluster service was `active`.
- `/api/health` returned HTTP 200.
- `/api/auth/status` confirmed initialized authentication was preserved.
Observed timestamps, PIDs, and restart counts are host-specific examples, not fixed configuration values.
## Operations and troubleshooting
```bash
sudo systemctl restart artex.service
sudo systemctl stop artex.service
sudo systemctl start artex.service
journalctl -b -u artex.service --no-pager
journalctl -u artex.service -f
```
Treat journal output as potentially sensitive; never share unredacted setup tokens or credentials. If startup fails, check PostgreSQL status, database settings and permissions on `<install-dir>/config.json`, `<install-dir>/data`, and `<install-dir>/keys`, occupied ports (`8787`/`8788`), and the journal. After a PostgreSQL dependency failure, restart ARTEX explicitly once the database is healthy. To recover from the start limit:
```bash
sudo systemctl reset-failed artex.service
sudo systemctl start artex.service
```
Back up PostgreSQL, `<install-dir>/config.json`, `<install-dir>/data`, and `<install-dir>/keys` (`jwt.key` and `secret.key`) securely before upgrades. The application stays loopback-only; remote access requires a separately reviewed tunnel or reverse proxy.
+18 -4
View File
@@ -4,8 +4,9 @@ A rewrite rule matches a request or a response that goes through the capture pro
how to change it: the method, the path and query, headers, the body, the status code.
**Status: the capture proxy applies enabled rules to HTTP traffic (request and response phases),
and records what it did.** Not yet in this change: archive format, a per-replay rule toggle, the web UI,
and the proof flag. See [Application in the capture proxy](#application-in-the-capture-proxy) for
records what it did, tags bound evidence snapshots, supports per-replay rule opt-in, and exposes
rewrite counters on `GET /api/traffic`.** Not yet in this change: archive format and the traffic-list /
rule-management web UI. See [Application in the capture proxy](#application-in-the-capture-proxy) for
what is covered and what is not.
Rules are a convenience for the operator, not a security control: a rule that fails leaves the
@@ -207,5 +208,18 @@ the slot may be converted into a tombstone if a new reservation needs the room.
### Replay
Replay goes through the recording proxy like any other traffic, so the enabled rules apply to a
replay and the resulting exchange is traced like a captured one. There is no per-replay toggle yet.
`POST /api/traffic/replay` sends the edited request through the recording proxy so the exchange is
captured in Traffic, but **bypasses capture rewrite rules by default** (`apply_rules: false`). To
run enabled capture rules against a replay and trace them like captured traffic, opt in with
`"apply_rules": true` (or check **Apply capture rewrite rules** in the Replay panel).
When `apply_rules` is false, `traffic.Replay` registers a single-use random 32-byte hex nonce in an
in-memory registry, attaches the internal header `X-ARTEX-Replay-Nonce: <nonce>`, and removes the
nonce when the call returns. The capture proxy strips `X-ARTEX-Replay-Nonce` unconditionally in the
request hook so the target never sees it; only a value matching a registered nonce bypasses both
request and response rule phases, while any forged header is stripped and ignored.
### Stats
`GET /api/traffic` includes a `rewrite` object exposing the proxy's `RewriteStats` counters since
start: `large_responses`, `capacity_refusals`, `tombstone_drops`, `conversions`, and `sweeps`.
+21 -1
View File
@@ -13,6 +13,7 @@ import (
"log"
"os"
"path/filepath"
"slices"
"strings"
"time"
@@ -176,7 +177,26 @@ func (s *Store) prepare(ctx context.Context, refs []db.TrafficRef) ([]db.Prepare
out = append(out, db.PreparedTrafficEvidence{Ref: byID[e.ID], Snapshot: v})
return nil
})
return out, err
if err != nil {
return nil, err
}
// Enrich after ReadEvidence returns so RewriteInfo is never called while
// ReadEvidence holds the traffic writer lock.
for i := range out {
rewritten, ruleIDs, err := s.Traffic.RewriteInfo(out[i].Snapshot.SourceTrafficID)
if err != nil {
return nil, fmt.Errorf("read rewrite provenance for %s: %w", out[i].Snapshot.SourceTrafficID, err)
}
if rewritten {
out[i].Snapshot.Rewritten = true
if len(ruleIDs) > 0 {
ids := slices.Clone(ruleIDs)
slices.Sort(ids)
out[i].Snapshot.RewriteRuleIDs = slices.Compact(ids)
}
}
}
return out, nil
}
func (s *Store) Record(ctx context.Context, in db.RecordFindingInput, refs []db.TrafficRef) (out *db.RecordedFinding, err error) {
+166
View File
@@ -8,14 +8,21 @@ import (
"errors"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/Autumn-27/artex/db"
"github.com/Autumn-27/artex/traffic"
"github.com/Autumn-27/artex/traffic/rewrite"
)
func evidenceFixture(t *testing.T) (*Store, db.RecordFindingInput, string) {
@@ -331,3 +338,162 @@ func TestEvidenceGCGraceAndActiveRestore(t *testing.T) {
t.Fatal("accepted tampered snapshot")
}
}
func startRewriteProxy(t *testing.T, dir string) (*traffic.Traffic, string) {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
addr := ln.Addr().String()
ln.Close()
tr, err := traffic.Open(dir, addr)
if err != nil {
t.Fatal(err)
}
go func() { _ = tr.Start() }()
t.Cleanup(func() { _ = tr.Close() })
deadline := time.Now().Add(5 * time.Second)
for {
c, err := net.Dial("tcp", addr)
if err == nil {
c.Close()
break
}
if time.Now().After(deadline) {
t.Fatalf("proxy did not listen on %s: %v", addr, err)
}
time.Sleep(10 * time.Millisecond)
}
return tr, "http://" + addr
}
func TestEvidenceBindingRewrittenExchange(t *testing.T) {
dsn := os.Getenv("ARTEX_PG_DSN")
if dsn == "" {
t.Skip("ARTEX_PG_DSN is required for evidence integration tests")
}
pg, err := db.Open(dsn)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { pg.Close() })
task, err := pg.CreateTask("evidence rewrite "+t.Name(), "rewrite provenance", nil, 0, 0)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { pg.DeleteFindingsByTask(task.ID); pg.DeleteTask(task.ID) })
target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/plain")
_, _ = io.WriteString(w, "upstream response")
}))
t.Cleanup(target.Close)
dir := t.TempDir()
tr, proxyURL := startRewriteProxy(t, filepath.Join(dir, "traffic"))
// Rules 5 (request) and 2 (response) apply to /rewritten; rule 9 only matches
// /never-matches so it is skipped. Neither applies to /plain.
set := rewrite.Compile([]rewrite.Rule{
{
ID: 5, Name: "req-header", Enabled: true, Phase: rewrite.PhaseRequest,
Matchers: rewrite.Matchers{URLRegex: "^/rewritten$"},
Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-Injected", Value: "yes"}},
},
{
ID: 2, Name: "resp-header", Enabled: true, Phase: rewrite.PhaseResponse,
Matchers: rewrite.Matchers{URLRegex: "^/rewritten$"},
Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-Resp-Tag", Value: "1"}},
},
{
ID: 9, Name: "skipped-rule", Enabled: true, Phase: rewrite.PhaseRequest,
Matchers: rewrite.Matchers{URLRegex: "^/never-matches$"},
Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-Skipped", Value: "1"}},
},
}, rewrite.Options{})
if len(set.LoadErrors()) != 0 {
t.Fatalf("rewrite rule load errors: %v", set.LoadErrors())
}
tr.SetRewriteRules(set)
pu, err := url.Parse(proxyURL)
if err != nil {
t.Fatal(err)
}
client := &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(pu)}}
for _, path := range []string{"/rewritten", "/plain"} {
req, err := http.NewRequest(http.MethodGet, target.URL+path, nil)
if err != nil {
t.Fatal(err)
}
resp, err := client.Do(req)
if err != nil {
t.Fatal(err)
}
_, _ = io.ReadAll(resp.Body)
resp.Body.Close()
}
exchangeID := func(suffix string) string {
var id string
deadline := time.Now().Add(3 * time.Second)
for {
err := tr.DB().QueryRow(`SELECT id FROM exchanges WHERE url LIKE ?`, "%"+suffix).Scan(&id)
if err == nil {
return id
}
if time.Now().After(deadline) {
t.Fatalf("exchange %s not recorded: %v", suffix, err)
}
time.Sleep(20 * time.Millisecond)
}
}
rewrittenID := exchangeID("/rewritten")
plainID := exchangeID("/plain")
store := New(pg, tr, filepath.Join(dir, "evidence"))
in := db.RecordFindingInput{
TaskID: task.ID, ExplorationID: task.ExplorationID, Worker: "test",
VulnClass: "TEST", Name: "Rewrite provenance", Severity: "low", Summary: "proxy rewrite test",
}
ctx := context.Background()
rec, err := store.Record(ctx, in, []db.TrafficRef{
{TrafficID: rewrittenID, Role: "proof", Note: "rewritten"},
{TrafficID: plainID, Role: "baseline", Note: "plain"},
})
if err != nil {
t.Fatalf("store.Record: %v", err)
}
if len(rec.Traffic.Bindings) != 2 {
t.Fatalf("bindings = %d, want 2", len(rec.Traffic.Bindings))
}
rewSnap := rec.Traffic.Bindings[0].Snapshot
if !rewSnap.Rewritten {
t.Fatalf("rewritten snapshot has Rewritten=false, want true")
}
if !slices.Equal(rewSnap.RewriteRuleIDs, []int64{2, 5}) {
t.Fatalf("rewritten snapshot rule IDs = %v, want [2 5]", rewSnap.RewriteRuleIDs)
}
// Content-addressed ID must exclude the rewrite flag and rule IDs.
stripped := rewSnap
stripped.Rewritten = false
stripped.RewriteRuleIDs = nil
if gotID := db.TrafficSnapshotID(stripped); gotID != rewSnap.ID {
t.Fatalf("snapshot ID depended on rewrite flag: stripped=%s stored=%s", gotID, rewSnap.ID)
}
plainSnap := rec.Traffic.Bindings[1].Snapshot
if plainSnap.Rewritten || len(plainSnap.RewriteRuleIDs) != 0 {
t.Fatalf("plain snapshot = rewritten:%v ids:%v, want false/empty", plainSnap.Rewritten, plainSnap.RewriteRuleIDs)
}
// Verify RewriteInfo errors fail preparation instead of recording a silent false.
if _, err := tr.DB().Exec(`DROP TABLE exchange_rewrites`); err != nil {
t.Fatal(err)
}
if _, err := store.prepare(ctx, []db.TrafficRef{{TrafficID: plainID, Role: "proof"}}); err == nil || !strings.Contains(err.Error(), "rewrite provenance") {
t.Fatalf("prepare with broken exchange_rewrites = %v, want rewrite provenance error", err)
}
}
+9 -3
View File
@@ -160,13 +160,19 @@ func (g *Guard) applyIntercept(ctx context.Context, ev hook.Event, cmd string) h
}
scope, gated := g.gateScope(ctx)
rules := !g.targetOnly && g.interceptor != nil && g.interceptor.IsToolEnabled(ev.ToolName)
if !rules && !gated {
// Persistence tools (custom tools, MCP servers, skills) always meet their built-in
// approval, in the target-only guard too; see intercept.MatchPersistence.
persist := g.interceptor != nil && intercept.IsPersistenceTool(ev.ToolName)
if !rules && !gated && !persist {
return hook.Result{}
}
ctx = intercept.WithCall(ctx, ev.ToolName, ev.Input)
var dec intercept.Decision
if rules {
var matched bool
matched := false
if persist {
dec, matched = g.interceptor.MatchPersistence(ev.ToolName, ev.Input)
}
if !matched && rules {
dec, matched = g.interceptor.Match(ev.ToolName, ev.Input)
if !matched {
// No rule matched. Ask the LLM fallback judge (if enabled); when it is off
+1 -1
View File
@@ -325,7 +325,7 @@ func newGateDB(t *testing.T) *gateDB {
t.Helper()
dsn, _, err := db.DSN()
if err != nil {
t.Skipf("no database config (%v) - skipping", err)
db.SkipPGUnavailable(t, err)
}
d, err := db.Open(dsn)
if err != nil {
+77
View File
@@ -0,0 +1,77 @@
package intercept
import (
"encoding/json"
"strings"
)
// PersistenceTools are the platform tools that store something the agents load or
// run later: a custom tool (its code runs on the host in later calls), an MCP server
// (a stdio entry starts its command), a skill file (agents load it as instructions).
// A prompt-injected page can make an agent call them, so each call needs operator
// approval. The approval comes from the built-in ask rules seeded in db
// ("[built-in] Persistent change: ..."). Nothing an agent can call edits those rules;
// only the console can disable them.
var PersistenceTools = []string{
"create_custom_tool", "update_custom_tool",
"create_mcp", "update_mcp",
"create_skill", "update_skill_file",
}
// IsPersistenceTool reports whether name is one of PersistenceTools.
func IsPersistenceTool(name string) bool {
for _, t := range PersistenceTools {
if t == name {
return true
}
}
return false
}
// MatchPersistence decides a call of a persistence tool from the loaded rules. Any
// enabled deny rule that matches wins. Otherwise the first matching built-in ask rule
// answers, so the approval needs no other rule. User allow rules are not consulted:
// an allow rule cannot skip the approval.
func (i *Interceptor) MatchPersistence(toolName string, input []byte) (Decision, bool) {
if !IsPersistenceTool(toolName) {
return Decision{}, false
}
rules, err := i.rules()
if err != nil {
return Decision{}, false
}
var ask *compiledRule
for idx := range rules {
r := rules[idx]
if !ruleMatches(r, toolName, input) {
continue
}
if r.Action == "deny" {
return persistenceDecision(r), true
}
if ask == nil && r.Action == "ask" && strings.HasPrefix(r.Name, builtinRulePrefix) {
ask = &rules[idx]
}
}
if ask == nil {
return Decision{}, false
}
return persistenceDecision(*ask), true
}
func persistenceDecision(r compiledRule) Decision {
msg := r.Message
if msg == "" {
msg = defaultMessage(r.Action, r.Name)
}
configJSON, _ := json.Marshal(r.InterceptRule)
return Decision{
RuleName: r.Name, ConfigDigest: digestInput(configJSON),
Action: r.Action,
Message: msg,
RuleID: r.ID,
TimeoutEnabled: r.TimeoutEnabled,
TimeoutSeconds: r.TimeoutSeconds,
TimeoutAction: r.TimeoutAction,
}
}
+88
View File
@@ -0,0 +1,88 @@
package intercept
import (
"testing"
"github.com/Autumn-27/artex/db"
)
const customToolApproval = `^(?:create_custom_tool|update_custom_tool)$`
func builtinAsk(enabled bool) db.InterceptRule {
return db.InterceptRule{
ID: 1, Name: "[built-in] Persistent change: custom tool", Enabled: enabled, Priority: 95,
MatchTarget: "tool_name", MatchType: "regex", Pattern: customToolApproval,
Action: "ask", Message: "approve the custom tool",
}
}
func TestPersistenceToolsAreSixNamedTools(t *testing.T) {
if len(PersistenceTools) != 6 {
t.Fatalf("got %d persistence tools, want 6", len(PersistenceTools))
}
for _, name := range PersistenceTools {
if !IsPersistenceTool(name) {
t.Fatalf("%s not recognised", name)
}
}
if IsPersistenceTool("Bash") {
t.Fatal("Bash is not a persistence tool")
}
}
func TestBuiltinAskAnswersPersistenceCall(t *testing.T) {
ic := New(nil)
ic.SetRulesForTest([]db.InterceptRule{builtinAsk(true)})
dec, ok := ic.MatchPersistence("create_custom_tool", []byte(`{"key":"x"}`))
if !ok || dec.Action != "ask" {
t.Fatalf("got %+v ok=%v, want an ask", dec, ok)
}
if dec.RuleName != "[built-in] Persistent change: custom tool" {
t.Fatalf("rule name = %q", dec.RuleName)
}
}
func TestPersistenceIgnoresOtherTools(t *testing.T) {
ic := New(nil)
ic.SetRulesForTest([]db.InterceptRule{builtinAsk(true)})
if _, ok := ic.MatchPersistence("Bash", []byte(`{"command":"create_custom_tool"}`)); ok {
t.Fatal("a tool outside PersistenceTools must not be answered here")
}
}
// An operator-disabled built-in rule is not loaded, so nothing answers the call.
func TestDisabledBuiltinAskDoesNotApply(t *testing.T) {
ic := New(nil)
ic.SetRulesForTest([]db.InterceptRule{builtinAsk(false)})
if _, ok := ic.MatchPersistence("create_custom_tool", nil); ok {
t.Fatal("a disabled rule must not answer")
}
}
// An allow rule at a higher priority cannot skip the built-in approval.
func TestUserAllowCannotBypassApproval(t *testing.T) {
ic := New(nil)
ic.SetRulesForTest([]db.InterceptRule{
{ID: 2, Name: "user allow all", Enabled: true, Priority: 500,
MatchTarget: "tool_name", MatchType: "regex", Pattern: `.*`, Action: "allow"},
builtinAsk(true),
})
dec, ok := ic.MatchPersistence("update_custom_tool", nil)
if !ok || dec.Action != "ask" {
t.Fatalf("got %+v ok=%v, want the approval to stand", dec, ok)
}
}
// A user deny that matches a persistence call wins over the built-in approval.
func TestUserDenyWinsOverApproval(t *testing.T) {
ic := New(nil)
ic.SetRulesForTest([]db.InterceptRule{
{ID: 3, Name: "user deny custom tools", Enabled: true, Priority: 10,
MatchTarget: "tool_name", MatchType: "regex", Pattern: `custom_tool`, Action: "deny"},
builtinAsk(true),
})
dec, ok := ic.MatchPersistence("create_custom_tool", nil)
if !ok || dec.Action != "deny" {
t.Fatalf("got %+v ok=%v, want deny", dec, ok)
}
}

Some files were not shown because too many files have changed in this diff Show More