Compare commits

...
Author SHA1 Message Date
BaaRai 944c131e90 docs(context-telemetry): align Phase C expected-gain bullet with minimal split 2026-11-20 20:41:48 +01:00
BaaRai 0dbb6de8d5 feat(agent): opt-in worker system cache-prefix split (P3.5 phase C) 2026-11-20 20:29:17 +01:00
BaaRai 3559be6482 Merge pull request #426 from BaaRai/feat/context-telemetry
feat(llmrec): opt-in LLM context size telemetry and aggregation (P3.5 phase A)
2026-11-20 19:37:48 +01:00
BaaRai e10c2b76aa Merge pull request #419 from ldevbr/contrib/next-telemetry-off
build(web): disable Next.js telemetry in the web build stage
2026-11-20 18:16:00 +01:00
BaaRai 8793f7b56c Merge pull request #408 from Lusitaniae/pr/tool-checksums
build: pin tool and node digests instead of fetching their checksums files
2026-11-20 18:15:21 +01:00
Luis ef25384063 build(web): disable Next.js telemetry in the web build stage
`next build` contacts telemetry.nextjs.org during the image build. Set
NEXT_TELEMETRY_DISABLED=1 in the web stage so the build makes no telemetry
request. No runtime behavior changes.
2026-11-20 10:26:01 -05:00
Pedro Gomes b7bc9f9517 build: pin tool and node digests instead of fetching their checksums files
Each tool archive was verified against the *_checksums.txt of its own GitHub
release, fetched in the same build step as the archive. That catches a truncated
download, a stale CDN entry and a moved tag. It does not catch the publisher: the
digest and the artifact it describes ship from the same release, so a replaced
asset plus a replaced checksums file agree with each other and the build goes
green. The build was trusting the channel it downloads from to vouch for itself.

Pin the digests in the Dockerfile instead — one per asset per architecture, next
to the version ARG — and have fetch() take the expected digest as an argument.
The assets and their URLs are unchanged; a swapped asset now fails the build
unless the diff that swaps it also changes a digest a reviewer reads. Node's
tarball in the runtime stage gets the same treatment for the same reason; it was
the last fetched-SHASUMS256 check left.

The digests were taken from the artifacts themselves and cross-checked against
the published sums files, so this change is build-identical: same 14 bytes of
input, same output, different trust anchor. Verified by building the tools stage
for both arches with the pinned values.

Cost, and it is a real one: a version bump is three edits instead of one, and a
publisher re-uploading an asset under an existing tag breaks the build until the
digest is bumped deliberately. Two hashes per tool bump is the chore that buys
not trusting the release channel.

sqlmap already worked this way (pinned tag, asserted commit); this makes the
binary tools consistent with it.
2026-11-20 09:29:34 +00:00
5 changed files with 452 additions and 36 deletions
+1
View File
@@ -23,6 +23,7 @@ All notable changes to this project are documented in this file. The format is b
- **`ask_operator` target warning**: a request whose question names a host or URL outside the goal targets is tagged as containing target-derived text (#197).
- **Opt-in update banner**: `ARTEX_UPDATE_CHECK` makes the backend check the latest GitHub release at most once per 24 hours and the UI show a banner; off by default (#202).
- **Release provenance**: release archives carry GitHub build provenance attestations and a CycloneDX SBOM (`gh attestation verify <zip> --repo BaaRai/ARTEX`); release notes are the matching `CHANGELOG.md` section (#219, #201).
- **Opt-in worker cache-prefix split (P3.5 Phase C)**: setting `ARTEX_CACHE_PREFIX=1` (off by default) splits the `worker` system prompt at `DynamicBoundary = 1` into a sibling-shareable static prefix (`body + untrustedContentTail + workerTrafficBlock`) and a per-intent dynamic suffix (`workerArtifactSpec(runDir)` through `<available-deferred-tools>`) without reordering sections or altering the visible joined prompt text. Planner, mainagent, chat, and goals are unchanged; custom worker prompts referencing `{{.Now}}` remain volatile in the static segment. See `docs/context-telemetry.md`.
- **Opt-in LLM context size telemetry (P3.5 Phase A)**: setting `ARTEX_CONTEXT_TELEMETRY=1` (off by default) records per-call request section byte sizes (`system_static`/`system_dynamic` split at `DynamicBoundary`, serialized `tools`, `messages` by role, `tool_result` subset, and `last_message_bytes`), tool/message counts, and token/cache usage in the new `llm_context` table (migration `0009_llm_context.sql`), with zero prompt/behavior change and zero content stored, independent of `llm_record`. `GET /api/llm/context/summary` aggregates input percentiles (`p50`/`p90`/`p99`), pro-rata section shares over real `input_tokens`, `cache_read/input` ratios by `(worker, model)`, and top tasks. See `docs/context-telemetry.md`.
- **HAR 1.2 traffic import and export**: `POST /api/traffic/import/har` imports raw HAR 1.2 JSON captures (up to 32 MiB / 2,000 entries / 32 MiB decoded bodies) with deterministic `har-<sha256>` IDs (and `_artexExchangeId` preservation for ARTEX exports so distinct same-second exchanges round-trip without collapsing), base64 decoding, `postData.params` and `cookies` array fallbacks, URL userinfo stripping, blob spilling above 256 KiB, and full-text indexing, without applying capture rewrite rules (`exchange_rewrites` stays empty); per-entry errors are reported alongside `imported`/`skipped`/`failed` counts. `GET /api/traffic/export/har` exports filtered traffic (reusing the `GET /api/traffic` filters) as a compliant HAR 1.2 document (with explicit unknown placeholders for unrecorded timings and header sizes) in stable `(ts, id)` order, encoding UTF-8 bodies as text and binary bodies as base64, capped at 1,000 entries / 20 MiB raw headers+bodies (31 MiB JSON backstop, guaranteed under the 32 MiB import limit) with explicit truncation reporting via `X-ARTEX-HAR-*` headers and `_truncated`/`_truncatedReason`/`_totalMatched` fields. The Traffic page adds **Import HAR** and **Export HAR** controls. See `docs/traffic-har.md`.
- **Capture-proxy rewrite rules, storage and API**: `GET`, `POST`, `PUT` and `DELETE /api/traffic/rewrite-rules` and `POST /api/traffic/rewrite-rules/{id}/toggle` manage rules that match a request or a response (method, host, URL, headers, body, status) and describe a rewrite (method, path and query, headers, body, status). Rules are validated on save, limited to 100 of which 50 enabled, created disabled, and stored in the new table `traffic_rewrite_rules` (migration `0005_traffic_rewrite_rules.sql`) with `matchers` and `actions` encrypted at rest. The value of an action marked `sensitive` is never returned, and an update sent without it keeps the stored one. The server compiles the enabled rules at startup and after every change and hands the snapshot to the proxy. The proxy applies them from the next entry. See `docs/traffic-rewrite-rules.md`.
+57 -26
View File
@@ -14,6 +14,7 @@
# Pinned to the Node 22 line the CI/release workflows use. On the build platform
# only; the frontend static export is architecture-independent.
FROM --platform=$BUILDPLATFORM node:22-bookworm-slim AS web
ENV NEXT_TELEMETRY_DISABLED=1
WORKDIR /src/web
COPY web/package.json web/package-lock.json ./
RUN npm ci --ignore-scripts
@@ -47,19 +48,44 @@ RUN ARTEX_SKIP_FRONTEND=1 \
# Web-recon toolbox (drost-community take 1, #100): pinned release binaries, staged
# on the build platform and copied into /usr/local/bin of the runtime image so hunt
# workers can run them directly through their Bash tool. Versions pinned per tool;
# upgrade deliberately. Each archive is verified against the checksums file of its
# own GitHub release (exact asset name, sha256). That file comes from the same
# release channel as the archive, so it catches corrupted or swapped downloads and
# moved tags, not a compromised publisher. sqlmap has no release binaries: it is a
# shallow clone of a pinned tag, checked against the expected commit.
# upgrade deliberately.
#
# Every archive is verified against a SHA-256 written down here, per asset and per
# architecture, not against the *_checksums.txt of its own GitHub release. Checking
# against that file catches a corrupted or swapped download and a moved tag, but the
# digest and the artifact it describes come from the same publisher and the same
# release: a replacement asset and a replacement checksums file agree with each
# other and the build passes. Pinning the digest here moves that decision out of the
# build, so a swapped asset fails unless the diff that swaps it also changes the
# digest a reviewer reads.
#
# Cost: a version bump is now three edits (the version ARG and the two digests), and
# a publisher re-uploading an asset under the same tag breaks the build until the
# digest is bumped on purpose. That is the trade: the build stops trusting the
# channel it downloads from.
#
# sqlmap has no release binaries: it is a shallow clone of a pinned tag, checked
# against the expected commit, which is the same idea.
FROM --platform=$BUILDPLATFORM debian:bookworm-slim AS tools
ARG TARGETARCH
ARG NUCLEI_VERSION=v3.11.1
ARG NUCLEI_SHA256_AMD64=ea63d4ae232808cd7c6bc00d0142428e231fab59dae01042246097d195835ab6
ARG NUCLEI_SHA256_ARM64=8044e3d9768ba0a744b2872c1a87e813006f013da97ca9f50f7661a4203bec07
ARG HTTPX_VERSION=v1.12.0
ARG HTTPX_SHA256_AMD64=9d8439e8b6c9aa7d1e2314817a392e00d5178da3af5652f7475f88868f418f76
ARG HTTPX_SHA256_ARM64=fd7b123c1dfbc3d69f19f524e4eebcd6ec06b9a6cbd56813c76f11645197331e
ARG SUBFINDER_VERSION=v2.17.0
ARG SUBFINDER_SHA256_AMD64=a39c3e5c7a26ef3c430915c4d6cbf7716a00f26769a75e9405aba796142eba37
ARG SUBFINDER_SHA256_ARM64=24913d49dddc1d3a5e99a4a7580da5be980959b9415b96f52c4f7f4124392190
ARG KATANA_VERSION=v1.8.0
ARG KATANA_SHA256_AMD64=cb027ae99d0757061b70c07a15c7a80f62c5a535ab52c4a329f80c4b0a7309d2
ARG KATANA_SHA256_ARM64=80e269210727a855511ffac8dfb01297b2344742acec71a35e8ba4a67a90f0f2
ARG FFUF_VERSION=v2.3.0
ARG FFUF_SHA256_AMD64=b2a3c725fcb9da175159682f54d6e9149f2905b00d84d155e7efc5d599975ceb
ARG FFUF_SHA256_ARM64=e2d712615d53da797ee929006e41f2df5a7960a5353133db9c150ddaa22ae4e8
ARG DALFOX_VERSION=v3.2.4
ARG DALFOX_SHA256_AMD64=c11cfd2f8261973f0a2a2a1eba5a147ce77712a29acfa6343f9847fb8bb7eca2
ARG DALFOX_SHA256_ARM64=fd56b0b7da2d06868cd73b504f41d21314629feffcd5bdc97718db1c9dd2fa70
ARG SQLMAP_VERSION=1.9.9
ARG SQLMAP_COMMIT=cceda39208b5aad478de2c3f731df4239d9881f6
RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -67,17 +93,21 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
RUN set -eux; \
case "${TARGETARCH:-amd64}" in \
amd64) PD_ARCH=amd64; FFUF_ARCH=amd64; DALFOX_ARCH=x86_64 ;; \
arm64) PD_ARCH=arm64; FFUF_ARCH=arm64; DALFOX_ARCH=aarch64 ;; \
amd64) PD_ARCH=amd64; FFUF_ARCH=amd64; DALFOX_ARCH=x86_64; \
NUCLEI_SUM="$NUCLEI_SHA256_AMD64"; HTTPX_SUM="$HTTPX_SHA256_AMD64"; \
SUBFINDER_SUM="$SUBFINDER_SHA256_AMD64"; KATANA_SUM="$KATANA_SHA256_AMD64"; \
FFUF_SUM="$FFUF_SHA256_AMD64"; DALFOX_SUM="$DALFOX_SHA256_AMD64" ;; \
arm64) PD_ARCH=arm64; FFUF_ARCH=arm64; DALFOX_ARCH=aarch64; \
NUCLEI_SUM="$NUCLEI_SHA256_ARM64"; HTTPX_SUM="$HTTPX_SHA256_ARM64"; \
SUBFINDER_SUM="$SUBFINDER_SHA256_ARM64"; KATANA_SUM="$KATANA_SHA256_ARM64"; \
FFUF_SUM="$FFUF_SHA256_ARM64"; DALFOX_SUM="$DALFOX_SHA256_ARM64" ;; \
*) echo "unsupported architecture: ${TARGETARCH}" >&2; exit 1 ;; \
esac; \
mkdir -p /out; \
fetch() { base="$1"; asset="$2"; sums="$3"; name="$4"; \
fetch() { base="$1"; asset="$2"; want="$3"; name="$4"; \
rm -rf /tmp/t; mkdir -p /tmp/t/x; \
curl -fsSL "$base/$asset" -o "/tmp/t/$asset"; \
curl -fsSL "$base/$sums" -o /tmp/t/sums.txt; \
want="$(awk -v a="$asset" '$2 == a || $2 == "*" a { print $1 }' /tmp/t/sums.txt)"; \
if [ -z "$want" ]; then echo "no checksum entry for $asset" >&2; exit 1; fi; \
if [ -z "$want" ]; then echo "no pinned digest for $asset" >&2; exit 1; fi; \
got="$(sha256sum "/tmp/t/$asset" | cut -d' ' -f1)"; \
if [ "$want" != "$got" ]; then echo "checksum mismatch for $asset: want $want got $got" >&2; exit 1; fi; \
case "$asset" in *.zip) unzip -q "/tmp/t/$asset" -d /tmp/t/x ;; *) tar -xzf "/tmp/t/$asset" -C /tmp/t/x ;; esac; \
@@ -86,17 +116,17 @@ RUN set -eux; \
rm -rf /tmp/t; }; \
gh="https://github.com"; \
fetch "$gh/projectdiscovery/nuclei/releases/download/$NUCLEI_VERSION" \
"nuclei_${NUCLEI_VERSION#v}_linux_${PD_ARCH}.zip" "nuclei_${NUCLEI_VERSION#v}_checksums.txt" nuclei; \
"nuclei_${NUCLEI_VERSION#v}_linux_${PD_ARCH}.zip" "$NUCLEI_SUM" nuclei; \
fetch "$gh/projectdiscovery/httpx/releases/download/$HTTPX_VERSION" \
"httpx_${HTTPX_VERSION#v}_linux_${PD_ARCH}.zip" "httpx_${HTTPX_VERSION#v}_checksums.txt" httpx; \
"httpx_${HTTPX_VERSION#v}_linux_${PD_ARCH}.zip" "$HTTPX_SUM" httpx; \
fetch "$gh/projectdiscovery/subfinder/releases/download/$SUBFINDER_VERSION" \
"subfinder_${SUBFINDER_VERSION#v}_linux_${PD_ARCH}.zip" "subfinder_${SUBFINDER_VERSION#v}_checksums.txt" subfinder; \
"subfinder_${SUBFINDER_VERSION#v}_linux_${PD_ARCH}.zip" "$SUBFINDER_SUM" subfinder; \
fetch "$gh/projectdiscovery/katana/releases/download/$KATANA_VERSION" \
"katana_${KATANA_VERSION#v}_linux_${PD_ARCH}.zip" "katana-${KATANA_VERSION#v}-checksums.txt" katana; \
"katana_${KATANA_VERSION#v}_linux_${PD_ARCH}.zip" "$KATANA_SUM" katana; \
fetch "$gh/ffuf/ffuf/releases/download/$FFUF_VERSION" \
"ffuf_${FFUF_VERSION#v}_linux_${FFUF_ARCH}.tar.gz" "ffuf_${FFUF_VERSION#v}_checksums.txt" ffuf; \
"ffuf_${FFUF_VERSION#v}_linux_${FFUF_ARCH}.tar.gz" "$FFUF_SUM" ffuf; \
fetch "$gh/hahwul/dalfox/releases/download/$DALFOX_VERSION" \
"dalfox-${DALFOX_VERSION}-linux-${DALFOX_ARCH}-musl.tar.gz" "dalfox-${DALFOX_VERSION}-linux-${DALFOX_ARCH}-musl.tar.gz.sha256" dalfox; \
"dalfox-${DALFOX_VERSION}-linux-${DALFOX_ARCH}-musl.tar.gz" "$DALFOX_SUM" dalfox; \
git clone --quiet --depth 1 --branch "$SQLMAP_VERSION" https://github.com/sqlmapproject/sqlmap /opt/sqlmap; \
commit="$(git -C /opt/sqlmap rev-parse HEAD)"; \
if [ "$commit" != "$SQLMAP_COMMIT" ]; then echo "sqlmap commit mismatch: want $SQLMAP_COMMIT got $commit" >&2; exit 1; fi; \
@@ -110,26 +140,27 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates ripgrep curl wget vim git jq unzip xz-utils \
dnsutils iputils-ping netcat-openbsd inetutils-telnet whois nmap \
&& rm -rf /var/lib/apt/lists/*
# Node 22 from the official nodejs.org tarball, verified against its SHASUMS256.txt:
# bookworm's apt nodejs is 18, Playwright requires >=20, and 20 is end-of-life.
# 22 matches the CI and release workflows.
# Node 22 from the official nodejs.org tarball: bookworm's apt nodejs is 18,
# Playwright requires >=20, and 20 is end-of-life. 22 matches the CI and release
# workflows. Verified against a digest pinned here rather than against the fetched
# SHASUMS256.txt, for the reason given at the tools stage.
ARG NODE_VERSION=22.23.4
ARG NODE_SHA256_AMD64=1805bcd35e6815559bd2eb97db438d3fdb0eb3a5e6a977b7c1badca76337e9ce
ARG NODE_SHA256_ARM64=6287aa46b6c375d816e63ae66d024fbbfd70404825aabdfdf898a970871d2f4b
RUN set -eux; \
case "${TARGETARCH:-amd64}" in \
amd64) NODE_ARCH=x64 ;; \
arm64) NODE_ARCH=arm64 ;; \
amd64) NODE_ARCH=x64; want="$NODE_SHA256_AMD64" ;; \
arm64) NODE_ARCH=arm64; want="$NODE_SHA256_ARM64" ;; \
*) echo "unsupported architecture: ${TARGETARCH}" >&2; exit 1 ;; \
esac; \
base="https://nodejs.org/dist/v${NODE_VERSION}"; \
asset="node-v${NODE_VERSION}-linux-${NODE_ARCH}.tar.xz"; \
curl -fsSL "$base/$asset" -o "/tmp/$asset"; \
curl -fsSL "$base/SHASUMS256.txt" -o /tmp/SHASUMS256.txt; \
want="$(awk -v a="$asset" '$2 == a { print $1 }' /tmp/SHASUMS256.txt)"; \
if [ -z "$want" ]; then echo "no checksum entry for $asset" >&2; exit 1; fi; \
[ -n "$want" ] || { echo "no pinned digest for $asset" >&2; exit 1; }; \
got="$(sha256sum "/tmp/$asset" | cut -d' ' -f1)"; \
if [ "$want" != "$got" ]; then echo "checksum mismatch for $asset: want $want got $got" >&2; exit 1; fi; \
tar -xJf "/tmp/$asset" -C /usr/local --strip-components=1 --no-same-owner; \
rm -f "/tmp/$asset" /tmp/SHASUMS256.txt; \
rm -f "/tmp/$asset"; \
node --version; npm --version
# Pinned, checksum-verified recon binaries and the sqlmap checkout from the tools stage.
COPY --from=tools /out/ /usr/local/bin/
+54 -8
View File
@@ -387,12 +387,57 @@ func IsRunDirName(name string) bool {
return true
}
func workerSystem(proxyAddr, caCert, dataDir, runDir string) string {
// CachePrefixEnv is the opt-in environment flag enabling the worker system
// prompt cache-prefix split (P3.5 phase C). Off by default: when unset/false,
// worker system segments and DynamicBoundary are byte-identical to the legacy
// layout. Planner, mainagent, chat, and goals are unaffected.
const CachePrefixEnv = "ARTEX_CACHE_PREFIX"
func cachePrefixEnabled() bool {
switch strings.ToLower(strings.TrimSpace(os.Getenv(CachePrefixEnv))) {
case "1", "true", "on", "yes":
return true
}
return false
}
// workerSystemParts splits workerSystem at the first per-intent volatile
// segment (workerArtifactSpec(runDir)) without reordering any text:
// - staticPart = body + untrustedContentTail + workerTrafficBlock
// - dynamicHead = workerArtifactSpec(runDir) + noiseProfileBlock
//
// Custom DB prompts that reference {{.Now}} remain volatile inside staticPart.
func workerSystemParts(proxyAddr, caCert, dataDir, runDir string) (staticPart, dynamicHead string) {
body := renderSystem("worker", workerDefaultTmpl, WorkerVars{ProxyAddr: proxyAddr, DataDir: dataDir, Now: nowStr()})
// caCert is present only when the recording MITM is on, which is exactly when
// the traffic_* tools are registered — so it gates the traffic-tool note.
// Optional finding guidance is added for every role after tool resolution.
return body + untrustedContentTail(body) + workerTrafficBlock(caCert != "") + workerArtifactSpec(runDir) + noiseProfileBlock()
staticPart = body + untrustedContentTail(body) + workerTrafficBlock(caCert != "")
dynamicHead = workerArtifactSpec(runDir) + noiseProfileBlock()
return staticPart, dynamicHead
}
func workerSystem(proxyAddr, caCert, dataDir, runDir string) string {
staticPart, dynamicHead := workerSystemParts(proxyAddr, caCert, dataDir, runDir)
return staticPart + dynamicHead
}
// assembleWorkerSystem builds the worker's SystemPrompt segments and
// DynamicBoundary. When split is false (default), it delegates directly to
// deferredSystem(staticPart+dynamicTail, def). When split is true
// (ARTEX_CACHE_PREFIX=1), it emits two segments [staticPart, dynamic] with
// DynamicBoundary=1 so sibling workers in the same task share the cached static
// prefix, absorbing the leading "\n\n" separator from workerArtifactSpec so
// strings.Join(system, "\n\n") is byte-identical to the unsplit prompt.
func assembleWorkerSystem(staticPart, dynamicTail string, def DeferredInfo, split bool) (system []string, boundary int) {
if !split {
return deferredSystem(staticPart+dynamicTail, def)
}
dyn := strings.TrimPrefix(dynamicTail+def.FindingGuidance, "\n\n")
if block := actool.RenderDeferredToolsBlock(def.GlobalNames); block != "" {
dyn += "\n\n" + block
}
return []string{staticPart, dyn}, 1
}
// renderIntentTask formats the claimed intent for the worker's launch USER message:
@@ -524,18 +569,19 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
// inherit a parent run's background into the action reviewer.
ctx = intercept.WithReviewContext(ctx, runDir, intercept.ReviewBackground{})
overview := renderWorkerGraphOverview(tsx.graphOverviewData(), newDataEnvelope())
sysBody := workerSystem(w.proxyAddr, w.proxyCACert, w.workDir, runDir) + frameLessonsBlock(w.lessonsBlock)
staticPart, dynamicTail := workerSystemParts(w.proxyAddr, w.proxyCACert, w.workDir, runDir)
dynamicTail += frameLessonsBlock(w.lessonsBlock)
if w.wantConstraints() {
sysBody += constraintBlock(ts) // inject the operation constraints (if any) into the system prompt; the worker strictly obeys them during execution
dynamicTail += constraintBlock(ts) // inject the operation constraints (if any) into the system prompt; the worker strictly obeys them during execution
}
// intent block → intent-anchored assets block → launch instruction, appended to
// the system tail in that order (same append pattern as constraintBlock).
sysBody += renderIntentTask(intent)
dynamicTail += renderIntentTask(intent)
if as != nil {
if ids := intentAssetIDs(intent); len(ids) > 0 {
if assets, err := as.GetByIDs(ids); err == nil && len(assets) > 0 {
if b, err := json.Marshal(assets); err == nil {
sysBody += "\n\nThe target assets corresponding to this intent's asset_ids:\n" + string(b)
dynamicTail += "\n\nThe target assets corresponding to this intent's asset_ids:\n" + string(b)
}
// The assets this intent explicitly targets → automatically join the task's
// test scope (same conservative granularity as insertAssets). upsertTaskScope's
@@ -551,8 +597,8 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
}
}
}
sysBody += "\n\nStart executing the intent above: do only it, produce only facts, assets, findings, stop when done."
system, boundary := deferredSystem(sysBody, def)
dynamicTail += "\n\nStart executing the intent above: do only it, produce only facts, assets, findings, stop when done."
system, boundary := assembleWorkerSystem(staticPart, dynamicTail, def, cachePrefixEnabled())
// The task-level deadline (injected via ctx) clamps this run's wall-clock budget +
// decides the wrap-up prompt (see taskclock.go).
tc := taskClockFrom(ctx)
+326
View File
@@ -0,0 +1,326 @@
package agent
import (
"context"
"encoding/json"
"fmt"
"iter"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/Autumn-27/artex/db"
"github.com/Autumn-27/artex/llmrec"
"github.com/Autumn-27/norma/llm"
actool "github.com/Autumn-27/norma/tool"
)
type captureReqProvider struct {
mu sync.Mutex
reqs []llm.CompletionRequest
}
func (p *captureReqProvider) Stream(_ context.Context, req llm.CompletionRequest) iter.Seq2[llm.StreamEvent, error] {
p.mu.Lock()
p.reqs = append(p.reqs, req)
p.mu.Unlock()
return func(yield func(llm.StreamEvent, error) bool) {
if !yield(llm.StreamEvent{Type: llm.SEMessageStart, Usage: llm.Usage{InputTokens: 100}}, nil) {
return
}
if !yield(llm.StreamEvent{Type: llm.SETextDelta, Text: "done"}, nil) {
return
}
yield(llm.StreamEvent{Type: llm.SEMessageDelta, StopReason: "end_turn", Usage: llm.Usage{OutputTokens: 4}}, nil)
}
}
func (p *captureReqProvider) Complete(_ context.Context, req llm.CompletionRequest) (llm.Message, string, llm.Usage, error) {
p.mu.Lock()
p.reqs = append(p.reqs, req)
p.mu.Unlock()
return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{llm.TextBlock("done")}}, "end_turn", llm.Usage{InputTokens: 100, OutputTokens: 4}, nil
}
func (p *captureReqProvider) last() llm.CompletionRequest {
p.mu.Lock()
defer p.mu.Unlock()
if len(p.reqs) == 0 {
return llm.CompletionRequest{}
}
return p.reqs[len(p.reqs)-1]
}
func TestAssembleWorkerSystemFlagOffAndOnByteExact(t *testing.T) {
for _, tc := range []struct {
name string
caCert string
globalNames []string
guidance string
}{
{name: "no_deferred_no_traffic", caCert: "", globalNames: nil, guidance: ""},
{name: "with_deferred_and_traffic", caCert: "/ca.pem", globalNames: []string{"mcp__zap__scan", "mcp__burp__send"}, guidance: "\n\n[Finding guidance]"},
} {
t.Run(tc.name, func(t *testing.T) {
def := DeferredInfo{
GlobalNames: tc.globalNames,
FindingGuidance: tc.guidance,
}
buildTail := func(runDir, intentSummary, assetDomain string) (string, string) {
staticPart, dynamicTail := workerSystemParts("127.0.0.1:8788", tc.caCert, "/data", runDir)
dynamicTail += "\n\n## Prior evidence\n- lesson A"
dynamicTail += "\n\n[Constraints]\n- do not touch prod"
dynamicTail += renderIntentTask(&db.Node{ID: 7, Payload: json.RawMessage(`{"summary":"` + intentSummary + `"}`)})
dynamicTail += "\n\nThe target assets corresponding to this intent's asset_ids:\n[{\"domain\":\"" + assetDomain + "\"}]"
dynamicTail += "\n\nStart executing the intent above: do only it, produce only facts, assets, findings, stop when done."
return staticPart, dynamicTail
}
s1, d1 := buildTail("/data/tasks/1/i10", "probe auth", "alpha.example")
s2, d2 := buildTail("/data/tasks/1/i20", "probe billing", "beta.example")
// Legacy baseline (equivalent to flag off).
legacySys1, legacyBound1 := deferredSystem(workerSystem("127.0.0.1:8788", tc.caCert, "/data", "/data/tasks/1/i10")+strings.TrimPrefix(d1, workerArtifactSpec("/data/tasks/1/i10")+noiseProfileBlock()), def)
offSys1, offBound1 := assembleWorkerSystem(s1, d1, def, false)
if offBound1 != legacyBound1 || !slices.Equal(offSys1, legacySys1) {
t.Fatalf("flag off diverged from legacy: got (%d, len=%d) want (%d, len=%d)", offBound1, len(offSys1), legacyBound1, len(legacySys1))
}
// Flag ON: split into [static, dynamic] with DynamicBoundary=1.
onSys1, onBound1 := assembleWorkerSystem(s1, d1, def, true)
onSys2, onBound2 := assembleWorkerSystem(s2, d2, def, true)
if onBound1 != 1 || onBound2 != 1 || len(onSys1) != 2 || len(onSys2) != 2 {
t.Fatalf("flag on shape = (%d, %d) / (%d, %d), want boundary=1 len=2", onBound1, len(onSys1), onBound2, len(onSys2))
}
// Static segment identical across sibling workers.
if onSys1[0] != onSys2[0] {
t.Fatalf("static prefix differs between sibling workers")
}
// Dynamic segment differs and carries per-intent data + deferred block.
if onSys1[1] == onSys2[1] {
t.Fatalf("dynamic segment unexpectedly identical")
}
for _, want := range []string{"/data/tasks/1/i10", "probe auth", "alpha.example", "Start executing the intent above:"} {
if !strings.Contains(onSys1[1], want) {
t.Errorf("dynamic[1] missing %q", want)
}
if strings.Contains(onSys1[0], want) {
t.Errorf("static[0] unexpectedly contains volatile %q", want)
}
}
if len(tc.globalNames) > 0 {
if !strings.Contains(onSys1[1], "<available-deferred-tools>") || !strings.Contains(onSys1[1], "mcp__zap__scan") {
t.Errorf("deferred tools block missing from dynamic segment")
}
if strings.Contains(onSys1[0], "<available-deferred-tools>") {
t.Errorf("deferred tools block unexpectedly in static segment")
}
}
// Byte-exact visible text: Join(onSys, "\n\n") == Join(offSys, "\n\n").
wantVisible1 := strings.Join(offSys1, "\n\n")
gotVisible1 := strings.Join(onSys1, "\n\n")
if gotVisible1 != wantVisible1 {
t.Fatalf("visible text mismatch (len got=%d want=%d)", len(gotVisible1), len(wantVisible1))
}
offSys2, _ := assembleWorkerSystem(s2, d2, def, false)
if strings.Join(onSys2, "\n\n") != strings.Join(offSys2, "\n\n") {
t.Fatalf("worker 2 visible text mismatch")
}
})
}
}
func TestWorkerExecuteCachePrefixEndToEnd(t *testing.T) {
d := testDB(t)
t.Cleanup(func() { _ = d.Close() })
stamp := time.Now().UnixNano()
task, err := d.CreateTask(fmt.Sprintf("cache-prefix-%d", stamp), "verify worker cache prefix", nil, 0, 0)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = d.DeleteTask(task.ID) })
ts := d.Exploration(task.ExplorationID)
as := d.Assets()
var asset1ID, asset2ID int64
dom1 := fmt.Sprintf("alpha-%d.cache.test", stamp)
dom2 := fmt.Sprintf("beta-%d.cache.test", stamp)
if err := d.QueryRow(`INSERT INTO assets(type,domain,root_domain,task_ids) VALUES('subdomain',$1,$1,ARRAY[$2]::bigint[]) RETURNING id`, dom1, task.ID).Scan(&asset1ID); err != nil {
t.Fatal(err)
}
if err := d.QueryRow(`INSERT INTO assets(type,domain,root_domain,task_ids) VALUES('subdomain',$1,$1,ARRAY[$2]::bigint[]) RETURNING id`, dom2, task.ID).Scan(&asset2ID); err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_, _ = d.Exec(`DELETE FROM assets WHERE id IN ($1,$2)`, asset1ID, asset2ID)
})
if _, err := ts.AddConstraint("deny", "stay within test scope", "human"); err != nil {
t.Fatal(err)
}
p1Map := map[string]any{"summary": "inspect alpha auth", "asset_ids": []int64{asset1ID}}
p2Map := map[string]any{"summary": "inspect beta checkout", "asset_ids": []int64{asset2ID}}
i1ID, err := ts.AddNode("intent", p1Map, 0, "running", "planner", nil)
if err != nil {
t.Fatal(err)
}
i2ID, err := ts.AddNode("intent", p2Map, 0, "running", "planner", nil)
if err != nil {
t.Fatal(err)
}
raw1, _ := json.Marshal(p1Map)
raw2, _ := json.Marshal(p2Map)
intent1 := &db.Node{ID: i1ID, Payload: raw1}
intent2 := &db.Node{ID: i2ID, Payload: raw2}
// Inject a global deferred MCP name so the deferred block is exercised end-to-end.
prevAugment := ToolAugment()
SetToolAugment(func(_ context.Context, _ string) ([]actool.CoreTool, DeferredInfo, func()) {
return nil, DeferredInfo{
GlobalNames: []string{"mcp__demo__scanner"},
}, func() {}
})
t.Cleanup(func() { SetToolAugment(prevAugment) })
workDir := t.TempDir()
runWorker := func(intent *db.Node, lessons string) llm.CompletionRequest {
t.Helper()
prov := &captureReqProvider{}
w := NewWorker(prov, "test-model", workDir, nil, 0, 2)
w.SetProxy("127.0.0.1:8788", "/ca.pem")
w.SetLessonsBlock(lessons)
if _, _, err := w.Execute(t.Context(), "work#1", task.ID, as, ts, intent, passHooks{}, nil, nil, nil); err != nil {
t.Fatalf("Worker.Execute(intent=%d): %v", intent.ID, err)
}
return prov.last()
}
// 1. Flag OFF (default): legacy segments and boundary. With a global deferred
// tool block, deferredSystem emits [sysText, deferredBlock] with boundary=2,
// and norma's UntrustedOutputBoundary (on by default) appends its fixed note as
// System[2], giving len=3 and boundary=2.
t.Setenv(CachePrefixEnv, "")
req1Off := runWorker(intent1, "")
req2Off := runWorker(intent2, "")
if req1Off.DynamicBoundary != 2 || len(req1Off.System) != 3 {
t.Fatalf("flag off worker 1: boundary=%d len=%d, want 2/3", req1Off.DynamicBoundary, len(req1Off.System))
}
if req1Off.System[0] == req2Off.System[0] {
t.Fatalf("expected flag-off worker system[0] to differ because runDir/intent are inside it")
}
// 2. Flag ON: static prefix identical across sibling workers, dynamic differs,
// visible text byte-identical, tools unchanged, deferred block inside dynamic,
// norma's appended untrusted-output note preserved as System[2].
t.Setenv(CachePrefixEnv, "1")
req1On := runWorker(intent1, "")
req2On := runWorker(intent2, "")
if req1On.DynamicBoundary != 1 || req2On.DynamicBoundary != 1 || len(req1On.System) != 3 || len(req2On.System) != 3 {
t.Fatalf("flag on shape: w1=(%d,%d) w2=(%d,%d), want (1,3)",
req1On.DynamicBoundary, len(req1On.System), req2On.DynamicBoundary, len(req2On.System))
}
if req1On.System[2] != req1Off.System[2] || req2On.System[2] != req2Off.System[2] {
t.Fatalf("norma untrusted-output note segment changed under ARTEX_CACHE_PREFIX=1")
}
if req1On.System[0] != req2On.System[0] {
t.Fatalf("flag on static prefix differs between sibling workers")
}
if req1On.System[1] == req2On.System[1] {
t.Fatalf("flag on dynamic suffix should differ between sibling workers")
}
runDir1 := filepath.Join(workDir, "tasks", fmt.Sprint(task.ID), fmt.Sprintf("i%d", intent1.ID))
runDir2 := filepath.Join(workDir, "tasks", fmt.Sprint(task.ID), fmt.Sprintf("i%d", intent2.ID))
for _, pair := range []struct {
name string
req llm.CompletionRequest
runDir string
intent string
domain string
}{
{"worker1", req1On, runDir1, "inspect alpha auth", dom1},
{"worker2", req2On, runDir2, "inspect beta checkout", dom2},
} {
for _, needle := range []string{pair.runDir, pair.intent, pair.domain, "stay within test scope", "Start executing the intent above:", "<available-deferred-tools>", "mcp__demo__scanner"} {
if !strings.Contains(pair.req.System[1], needle) {
t.Errorf("%s dynamic segment missing %q", pair.name, needle)
}
if strings.Contains(pair.req.System[0], needle) {
t.Errorf("%s static segment unexpectedly contains %q", pair.name, needle)
}
}
}
if strings.Join(req1On.System, "\n\n") != strings.Join(req1Off.System, "\n\n") {
t.Fatalf("worker 1 visible prompt changed under ARTEX_CACHE_PREFIX=1")
}
if strings.Join(req2On.System, "\n\n") != strings.Join(req2Off.System, "\n\n") {
t.Fatalf("worker 2 visible prompt changed under ARTEX_CACHE_PREFIX=1")
}
// With RSI lessons enabled (framed with a per-run random nonce), lessons stay in
// System[1] so System[0] remains byte-identical across sibling workers.
const sampleLessons = "- [positive, 1 evidence] sqli: verified"
req1Lessons := runWorker(intent1, sampleLessons)
req2Lessons := runWorker(intent2, sampleLessons)
if req1Lessons.System[0] != req1On.System[0] || req2Lessons.System[0] != req1On.System[0] {
t.Fatalf("lessons block leaked into static prefix")
}
if !strings.Contains(req1Lessons.System[1], sampleLessons) || !strings.Contains(req2Lessons.System[1], sampleLessons) {
t.Fatalf("lessons block missing from dynamic segment")
}
toolsOffJSON, _ := json.Marshal(req1Off.Tools)
toolsOnJSON, _ := json.Marshal(req1On.Tools)
if string(toolsOnJSON) != string(toolsOffJSON) {
t.Fatalf("tools changed under ARTEX_CACHE_PREFIX=1")
}
// Telemetry observes non-zero static & dynamic split with identical static bytes.
m1 := llmrec.MeasureContext(req1On)
m2 := llmrec.MeasureContext(req2On)
if m1.SystemStaticBytes <= 0 || m1.SystemDynamicBytes <= 0 || m1.SystemStaticBytes != m2.SystemStaticBytes {
t.Fatalf("MeasureContext mismatch: m1=(%d,%d) m2=(%d,%d)",
m1.SystemStaticBytes, m1.SystemDynamicBytes, m2.SystemStaticBytes, m2.SystemDynamicBytes)
}
// 3. Planner and mainagent are completely unchanged by ARTEX_CACHE_PREFIX=1.
runPlanner := func() llm.CompletionRequest {
prov := &captureReqProvider{}
if _, _, err := NewPlanner(prov, "test-model", workDir, nil, 0, 2).Plan(t.Context(), task.ID, as, ts, "goal", nil, passHooks{}, nil); err != nil {
t.Fatal(err)
}
return prov.last()
}
runMain := func() llm.CompletionRequest {
prov := &captureReqProvider{}
if _, err := NewMainAgent(prov, "test-model", workDir, nil, 0, 2).Chat(t.Context(), task.ID, 0, as, ts, "goal", "status?", passHooks{}, nil, nil, nil, nil, nil); err != nil {
t.Fatal(err)
}
return prov.last()
}
t.Setenv(CachePrefixEnv, "")
planOff := runPlanner()
mainOff := runMain()
t.Setenv(CachePrefixEnv, "1")
planOn := runPlanner()
mainOn := runMain()
if planOn.DynamicBoundary != planOff.DynamicBoundary || !slices.Equal(planOn.System, planOff.System) {
t.Fatalf("planner changed under ARTEX_CACHE_PREFIX=1: off=(%d,len=%d) on=(%d,len=%d)",
planOff.DynamicBoundary, len(planOff.System), planOn.DynamicBoundary, len(planOn.System))
}
if mainOn.DynamicBoundary != mainOff.DynamicBoundary || !slices.Equal(mainOn.System, mainOff.System) {
t.Fatalf("mainagent changed under ARTEX_CACHE_PREFIX=1: off=(%d,len=%d) on=(%d,len=%d)",
mainOff.DynamicBoundary, len(mainOff.System), mainOn.DynamicBoundary, len(mainOn.System))
}
}
+14 -2
View File
@@ -131,6 +131,18 @@ ORDER BY input_tokens DESC;
| Custom prompts only | — | `{{.Now}}` (`agent/prompt.go:46`, 1-second resolution). Built-in templates do **not** reference `{{.Now}}` (Go's `text/template` ignores the unused `Now` field); it only affects operator-customized prompts that explicitly use `{{.Now}}`. |
- **Expected gain**:
- **`worker` turn-1 cache reuse (real, code-backed)**: Today `workerArtifactSpec(runDir)` is concatenated into the single `sysBody` before `noiseProfileBlock`, lessons, constraints, `renderIntentTask(intent)`, and the target-assets block, so every sibling worker spawn busts the prompt cache from `runDir` onward. Splitting `worker` system segments at `DynamicBoundary` — placing the task-stable prefix (`workerDefaultTmpl` + `untrustedContentTail` + `workerTrafficBlock` + `noiseProfileBlock` + `frameLessonsBlock` + `constraintBlock` + `def.FindingGuidance` + `<available-deferred-tools>`) **before** `DynamicBoundary` (`system_static`) and the per-intent suffix (`workerArtifactSpec(runDir)` + `renderIntentTask(intent)` + target assets JSON + closing launch line) **after** `DynamicBoundary` (`system_dynamic`) — keeps the intent protected from compaction inside `System` while allowing sibling workers in the same task to share the cached `system_static + tools` prefix on turn 1 (in the fixture, `system + tools` is 36.0% of aggregate worker input and ~76.8% of turn-1 input, 21.5 KiB / 28.0 KiB, though the fixture does not isolate the per-intent suffix; measure the exact shareable prefix live).
- **`worker` turn-1 cache reuse (real, code-backed)**: Today `workerArtifactSpec(runDir)` is concatenated into the single `sysBody` before `noiseProfileBlock`, lessons, constraints, `renderIntentTask(intent)`, and the target-assets block, so every sibling worker spawn busts the prompt cache from `runDir` onward. The minimal `ARTEX_CACHE_PREFIX=1` split shipped in Phase C places the prefix before `workerArtifactSpec(runDir)` (`body` + `untrustedContentTail` + `workerTrafficBlock`) **before** `DynamicBoundary = 1` (`system_static`) and leaves everything from `workerArtifactSpec(runDir)` onward (`workerArtifactSpec(runDir)` + `noiseProfileBlock` + lessons + constraints + `renderIntentTask(intent)` + target assets JSON + closing launch line + `def.FindingGuidance` + `<available-deferred-tools>`) **after** `DynamicBoundary = 1` (`system_dynamic`) in its exact original order. This keeps the visible joined prompt byte-identical and keeps the intent protected from compaction inside `System` while allowing sibling workers in the same task to share the cached `system_static + tools` prefix on turn 1 (in the fixture, `system + tools` is 36.0% of aggregate worker input and ~76.8% of turn-1 input, 21.5 KiB / 28.0 KiB, though the fixture does not isolate the per-intent suffix; measure the exact shareable prefix live). Moving the remaining task-stable blocks (`noiseProfileBlock`, lessons, constraints, `def.FindingGuidance`, `<available-deferred-tools>`) before the boundary would require reordering sections relative to `workerArtifactSpec(runDir)` and is left as a potential future optimization outside this PR.
- **`planner` cross-round cache ratio (unverified hypothesis)**: The fixture above assumed a turn-1 cache miss on every planner round (yielding 48.5%), which is **not** demonstrated by the built-in code because `plannerSystem` is stable per task. Real cross-round misses could still happen from provider cache TTL expiry between rounds, constraint edits, or tool-RAG tool-list churn — treat any planner cache gain as a hypothesis to measure with `ARTEX_CONTEXT_TELEMETRY=1` on real tasks rather than a proven savings.
- **Risk**: Low for the `worker` `ARTEX_CACHE_PREFIX` split (pure segment reordering using `norma`'s existing `DynamicBoundary` contract; custom prompts using `{{.Now}}` stay an explicit edge case). Medium for `ARTEX_COMPACTION_MEMO` (injecting compaction memos alters model reasoning continuity and only pays off on tasks that repeatedly trigger compaction — verify with live `ARTEX_CONTEXT_TELEMETRY=1` data first).
- **Risk**: Low for the `worker` `ARTEX_CACHE_PREFIX` split (pure boundary split using `norma`'s existing `DynamicBoundary` contract, with zero text reordering; custom prompts using `{{.Now}}` stay an explicit edge case). Medium for `ARTEX_COMPACTION_MEMO` (injecting compaction memos alters model reasoning continuity and only pays off on tasks that repeatedly trigger compaction — verify with live `ARTEX_CONTEXT_TELEMETRY=1` data first).
---
## Phase C: Worker Cache Prefix Split (`ARTEX_CACHE_PREFIX=1`)
`ARTEX_CACHE_PREFIX=1` (accepts `1`, `true`, `on`, `yes`, case-insensitive; **off by default**) enables a minimal, zero-reordering system-prompt split for the **`worker`** lane only so sibling workers in the same task share a cacheable static prefix without changing the text visible to the model:
- **Static segment (`System[0]`, before `DynamicBoundary = 1`)**: `body + untrustedContentTail + workerTrafficBlock`.
- **Dynamic segment (`System[1]`, after `DynamicBoundary = 1`)**: `workerArtifactSpec(runDir) + noiseProfileBlock + lessons + constraints + renderIntentTask(intent) + target assets + launch instruction + FindingGuidance + <available-deferred-tools>` (in exact original order). The leading `"\n\n"` from `workerArtifactSpec` is absorbed because providers join segments with `"\n\n"`, keeping `strings.Join(System, "\n\n")` byte-identical to the legacy unsplit prompt.
- **Unchanged**: When `ARTEX_CACHE_PREFIX` is unset/false, worker segments and `DynamicBoundary` are byte-identical to the legacy layout. `planner`, `mainagent`, `chat`, and `goals` are completely unchanged regardless of the flag.
- **Custom prompts caveat**: The split covers only the built-in suffix starting at `workerArtifactSpec(runDir)`. If an operator customizes the `worker` prompt in the database to include `{{.Now}}` (1-second resolution), that timestamp remains inside `System[0]` and will keep the static prefix volatile across spawns.
- **Measuring live impact**: Enable both `ARTEX_CONTEXT_TELEMETRY=1` and `ARTEX_CACHE_PREFIX=1` on real tasks and compare `system_static` / `system_dynamic` shares and `cache_read_ratio` via `GET /api/llm/context/summary`.