mirror of
https://github.com/BaaRai/ARTEX.git
synced 2026-11-21 15:40:54 +08:00
Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
944c131e90 | ||
|
|
0dbb6de8d5 | ||
|
|
3559be6482 | ||
|
|
3bbeb2c54e | ||
|
|
eecf8895aa | ||
|
|
dd5c371ead | ||
|
|
e10c2b76aa | ||
|
|
4841016892 | ||
|
|
8793f7b56c | ||
|
|
ef25384063 | ||
|
|
9465dfa67a | ||
|
|
b7bc9f9517 |
@@ -23,6 +23,8 @@ All notable changes to this project are documented in this file. The format is b
|
||||
- **`ask_operator` target warning**: a request whose question names a host or URL outside the goal targets is tagged as containing target-derived text (#197).
|
||||
- **Opt-in update banner**: `ARTEX_UPDATE_CHECK` makes the backend check the latest GitHub release at most once per 24 hours and the UI show a banner; off by default (#202).
|
||||
- **Release provenance**: release archives carry GitHub build provenance attestations and a CycloneDX SBOM (`gh attestation verify <zip> --repo BaaRai/ARTEX`); release notes are the matching `CHANGELOG.md` section (#219, #201).
|
||||
- **Opt-in worker cache-prefix split (P3.5 Phase C)**: setting `ARTEX_CACHE_PREFIX=1` (off by default) splits the `worker` system prompt at `DynamicBoundary = 1` into a sibling-shareable static prefix (`body + untrustedContentTail + workerTrafficBlock`) and a per-intent dynamic suffix (`workerArtifactSpec(runDir)` through `<available-deferred-tools>`) without reordering sections or altering the visible joined prompt text. Planner, mainagent, chat, and goals are unchanged; custom worker prompts referencing `{{.Now}}` remain volatile in the static segment. See `docs/context-telemetry.md`.
|
||||
- **Opt-in LLM context size telemetry (P3.5 Phase A)**: setting `ARTEX_CONTEXT_TELEMETRY=1` (off by default) records per-call request section byte sizes (`system_static`/`system_dynamic` split at `DynamicBoundary`, serialized `tools`, `messages` by role, `tool_result` subset, and `last_message_bytes`), tool/message counts, and token/cache usage in the new `llm_context` table (migration `0009_llm_context.sql`), with zero prompt/behavior change and zero content stored, independent of `llm_record`. `GET /api/llm/context/summary` aggregates input percentiles (`p50`/`p90`/`p99`), pro-rata section shares over real `input_tokens`, `cache_read/input` ratios by `(worker, model)`, and top tasks. See `docs/context-telemetry.md`.
|
||||
- **HAR 1.2 traffic import and export**: `POST /api/traffic/import/har` imports raw HAR 1.2 JSON captures (up to 32 MiB / 2,000 entries / 32 MiB decoded bodies) with deterministic `har-<sha256>` IDs (and `_artexExchangeId` preservation for ARTEX exports so distinct same-second exchanges round-trip without collapsing), base64 decoding, `postData.params` and `cookies` array fallbacks, URL userinfo stripping, blob spilling above 256 KiB, and full-text indexing, without applying capture rewrite rules (`exchange_rewrites` stays empty); per-entry errors are reported alongside `imported`/`skipped`/`failed` counts. `GET /api/traffic/export/har` exports filtered traffic (reusing the `GET /api/traffic` filters) as a compliant HAR 1.2 document (with explicit unknown placeholders for unrecorded timings and header sizes) in stable `(ts, id)` order, encoding UTF-8 bodies as text and binary bodies as base64, capped at 1,000 entries / 20 MiB raw headers+bodies (31 MiB JSON backstop, guaranteed under the 32 MiB import limit) with explicit truncation reporting via `X-ARTEX-HAR-*` headers and `_truncated`/`_truncatedReason`/`_totalMatched` fields. The Traffic page adds **Import HAR** and **Export HAR** controls. See `docs/traffic-har.md`.
|
||||
- **Capture-proxy rewrite rules, storage and API**: `GET`, `POST`, `PUT` and `DELETE /api/traffic/rewrite-rules` and `POST /api/traffic/rewrite-rules/{id}/toggle` manage rules that match a request or a response (method, host, URL, headers, body, status) and describe a rewrite (method, path and query, headers, body, status). Rules are validated on save, limited to 100 of which 50 enabled, created disabled, and stored in the new table `traffic_rewrite_rules` (migration `0005_traffic_rewrite_rules.sql`) with `matchers` and `actions` encrypted at rest. The value of an action marked `sensitive` is never returned, and an update sent without it keeps the stored one. The server compiles the enabled rules at startup and after every change and hands the snapshot to the proxy. The proxy applies them from the next entry. See `docs/traffic-rewrite-rules.md`.
|
||||
- **Capture-proxy rewrite rules applied**: the capture proxy applies enabled rules to each HTTP request before it is sent to the target and to each response before it reaches the client. Each action that ran is recorded in the new table `exchange_rewrites` in the same transaction as the exchange, `traffic_get` shows a `REWRITTEN` line, and `traffic_search` never finds a value a rule wrote. SSE status and headers are rewritten at the start of the stream. **WebSocket handshakes are not rewritten**: the capture library runs its hook after the upgrade, so the status and headers cannot be changed, and frames pass unchanged. **Responses of 5 MiB or more** are forwarded unchanged and not recorded in full, because the library does not call the response hook for them. Snapshots of original requests are capped at 1024 entries and 64 MiB. Replays (`POST /api/traffic/replay`) bypass capture rewrite rules by default via an internal `X-ARTEX-Replay-Nonce` header stripped unconditionally by the proxy, and apply the rules when opted in with `apply_rules: true` (the **Apply capture rewrite rules** checkbox in the Replay panel). `GET /api/traffic` exposes the proxy's `rewrite` counters (`large_responses`, `capacity_refusals`, `tombstone_drops`, `conversions`, `sweeps`), flags each exchange row with `rewritten`, and filters with `rewritten=all|rewritten|original` (also accepting `true`/`false`), surfaced with a filter control and row badge on the Traffic page. Traffic archives export version 2 with per-exchange `rewritten` and sorted distinct applied `rewrite_rule_ids` while remaining deterministic and backward-compatible with version 1 archives. The rule-management web UI comes later.
|
||||
|
||||
+57
-26
@@ -14,6 +14,7 @@
|
||||
# Pinned to the Node 22 line the CI/release workflows use. On the build platform
|
||||
# only; the frontend static export is architecture-independent.
|
||||
FROM --platform=$BUILDPLATFORM node:22-bookworm-slim AS web
|
||||
ENV NEXT_TELEMETRY_DISABLED=1
|
||||
WORKDIR /src/web
|
||||
COPY web/package.json web/package-lock.json ./
|
||||
RUN npm ci --ignore-scripts
|
||||
@@ -47,19 +48,44 @@ RUN ARTEX_SKIP_FRONTEND=1 \
|
||||
# Web-recon toolbox (drost-community take 1, #100): pinned release binaries, staged
|
||||
# on the build platform and copied into /usr/local/bin of the runtime image so hunt
|
||||
# workers can run them directly through their Bash tool. Versions pinned per tool;
|
||||
# upgrade deliberately. Each archive is verified against the checksums file of its
|
||||
# own GitHub release (exact asset name, sha256). That file comes from the same
|
||||
# release channel as the archive, so it catches corrupted or swapped downloads and
|
||||
# moved tags, not a compromised publisher. sqlmap has no release binaries: it is a
|
||||
# shallow clone of a pinned tag, checked against the expected commit.
|
||||
# upgrade deliberately.
|
||||
#
|
||||
# Every archive is verified against a SHA-256 written down here, per asset and per
|
||||
# architecture, not against the *_checksums.txt of its own GitHub release. Checking
|
||||
# against that file catches a corrupted or swapped download and a moved tag, but the
|
||||
# digest and the artifact it describes come from the same publisher and the same
|
||||
# release: a replacement asset and a replacement checksums file agree with each
|
||||
# other and the build passes. Pinning the digest here moves that decision out of the
|
||||
# build, so a swapped asset fails unless the diff that swaps it also changes the
|
||||
# digest a reviewer reads.
|
||||
#
|
||||
# Cost: a version bump is now three edits (the version ARG and the two digests), and
|
||||
# a publisher re-uploading an asset under the same tag breaks the build until the
|
||||
# digest is bumped on purpose. That is the trade: the build stops trusting the
|
||||
# channel it downloads from.
|
||||
#
|
||||
# sqlmap has no release binaries: it is a shallow clone of a pinned tag, checked
|
||||
# against the expected commit, which is the same idea.
|
||||
FROM --platform=$BUILDPLATFORM debian:bookworm-slim AS tools
|
||||
ARG TARGETARCH
|
||||
ARG NUCLEI_VERSION=v3.11.1
|
||||
ARG NUCLEI_SHA256_AMD64=ea63d4ae232808cd7c6bc00d0142428e231fab59dae01042246097d195835ab6
|
||||
ARG NUCLEI_SHA256_ARM64=8044e3d9768ba0a744b2872c1a87e813006f013da97ca9f50f7661a4203bec07
|
||||
ARG HTTPX_VERSION=v1.12.0
|
||||
ARG HTTPX_SHA256_AMD64=9d8439e8b6c9aa7d1e2314817a392e00d5178da3af5652f7475f88868f418f76
|
||||
ARG HTTPX_SHA256_ARM64=fd7b123c1dfbc3d69f19f524e4eebcd6ec06b9a6cbd56813c76f11645197331e
|
||||
ARG SUBFINDER_VERSION=v2.17.0
|
||||
ARG SUBFINDER_SHA256_AMD64=a39c3e5c7a26ef3c430915c4d6cbf7716a00f26769a75e9405aba796142eba37
|
||||
ARG SUBFINDER_SHA256_ARM64=24913d49dddc1d3a5e99a4a7580da5be980959b9415b96f52c4f7f4124392190
|
||||
ARG KATANA_VERSION=v1.8.0
|
||||
ARG KATANA_SHA256_AMD64=cb027ae99d0757061b70c07a15c7a80f62c5a535ab52c4a329f80c4b0a7309d2
|
||||
ARG KATANA_SHA256_ARM64=80e269210727a855511ffac8dfb01297b2344742acec71a35e8ba4a67a90f0f2
|
||||
ARG FFUF_VERSION=v2.3.0
|
||||
ARG FFUF_SHA256_AMD64=b2a3c725fcb9da175159682f54d6e9149f2905b00d84d155e7efc5d599975ceb
|
||||
ARG FFUF_SHA256_ARM64=e2d712615d53da797ee929006e41f2df5a7960a5353133db9c150ddaa22ae4e8
|
||||
ARG DALFOX_VERSION=v3.2.4
|
||||
ARG DALFOX_SHA256_AMD64=c11cfd2f8261973f0a2a2a1eba5a147ce77712a29acfa6343f9847fb8bb7eca2
|
||||
ARG DALFOX_SHA256_ARM64=fd56b0b7da2d06868cd73b504f41d21314629feffcd5bdc97718db1c9dd2fa70
|
||||
ARG SQLMAP_VERSION=1.9.9
|
||||
ARG SQLMAP_COMMIT=cceda39208b5aad478de2c3f731df4239d9881f6
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
@@ -67,17 +93,21 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
RUN set -eux; \
|
||||
case "${TARGETARCH:-amd64}" in \
|
||||
amd64) PD_ARCH=amd64; FFUF_ARCH=amd64; DALFOX_ARCH=x86_64 ;; \
|
||||
arm64) PD_ARCH=arm64; FFUF_ARCH=arm64; DALFOX_ARCH=aarch64 ;; \
|
||||
amd64) PD_ARCH=amd64; FFUF_ARCH=amd64; DALFOX_ARCH=x86_64; \
|
||||
NUCLEI_SUM="$NUCLEI_SHA256_AMD64"; HTTPX_SUM="$HTTPX_SHA256_AMD64"; \
|
||||
SUBFINDER_SUM="$SUBFINDER_SHA256_AMD64"; KATANA_SUM="$KATANA_SHA256_AMD64"; \
|
||||
FFUF_SUM="$FFUF_SHA256_AMD64"; DALFOX_SUM="$DALFOX_SHA256_AMD64" ;; \
|
||||
arm64) PD_ARCH=arm64; FFUF_ARCH=arm64; DALFOX_ARCH=aarch64; \
|
||||
NUCLEI_SUM="$NUCLEI_SHA256_ARM64"; HTTPX_SUM="$HTTPX_SHA256_ARM64"; \
|
||||
SUBFINDER_SUM="$SUBFINDER_SHA256_ARM64"; KATANA_SUM="$KATANA_SHA256_ARM64"; \
|
||||
FFUF_SUM="$FFUF_SHA256_ARM64"; DALFOX_SUM="$DALFOX_SHA256_ARM64" ;; \
|
||||
*) echo "unsupported architecture: ${TARGETARCH}" >&2; exit 1 ;; \
|
||||
esac; \
|
||||
mkdir -p /out; \
|
||||
fetch() { base="$1"; asset="$2"; sums="$3"; name="$4"; \
|
||||
fetch() { base="$1"; asset="$2"; want="$3"; name="$4"; \
|
||||
rm -rf /tmp/t; mkdir -p /tmp/t/x; \
|
||||
curl -fsSL "$base/$asset" -o "/tmp/t/$asset"; \
|
||||
curl -fsSL "$base/$sums" -o /tmp/t/sums.txt; \
|
||||
want="$(awk -v a="$asset" '$2 == a || $2 == "*" a { print $1 }' /tmp/t/sums.txt)"; \
|
||||
if [ -z "$want" ]; then echo "no checksum entry for $asset" >&2; exit 1; fi; \
|
||||
if [ -z "$want" ]; then echo "no pinned digest for $asset" >&2; exit 1; fi; \
|
||||
got="$(sha256sum "/tmp/t/$asset" | cut -d' ' -f1)"; \
|
||||
if [ "$want" != "$got" ]; then echo "checksum mismatch for $asset: want $want got $got" >&2; exit 1; fi; \
|
||||
case "$asset" in *.zip) unzip -q "/tmp/t/$asset" -d /tmp/t/x ;; *) tar -xzf "/tmp/t/$asset" -C /tmp/t/x ;; esac; \
|
||||
@@ -86,17 +116,17 @@ RUN set -eux; \
|
||||
rm -rf /tmp/t; }; \
|
||||
gh="https://github.com"; \
|
||||
fetch "$gh/projectdiscovery/nuclei/releases/download/$NUCLEI_VERSION" \
|
||||
"nuclei_${NUCLEI_VERSION#v}_linux_${PD_ARCH}.zip" "nuclei_${NUCLEI_VERSION#v}_checksums.txt" nuclei; \
|
||||
"nuclei_${NUCLEI_VERSION#v}_linux_${PD_ARCH}.zip" "$NUCLEI_SUM" nuclei; \
|
||||
fetch "$gh/projectdiscovery/httpx/releases/download/$HTTPX_VERSION" \
|
||||
"httpx_${HTTPX_VERSION#v}_linux_${PD_ARCH}.zip" "httpx_${HTTPX_VERSION#v}_checksums.txt" httpx; \
|
||||
"httpx_${HTTPX_VERSION#v}_linux_${PD_ARCH}.zip" "$HTTPX_SUM" httpx; \
|
||||
fetch "$gh/projectdiscovery/subfinder/releases/download/$SUBFINDER_VERSION" \
|
||||
"subfinder_${SUBFINDER_VERSION#v}_linux_${PD_ARCH}.zip" "subfinder_${SUBFINDER_VERSION#v}_checksums.txt" subfinder; \
|
||||
"subfinder_${SUBFINDER_VERSION#v}_linux_${PD_ARCH}.zip" "$SUBFINDER_SUM" subfinder; \
|
||||
fetch "$gh/projectdiscovery/katana/releases/download/$KATANA_VERSION" \
|
||||
"katana_${KATANA_VERSION#v}_linux_${PD_ARCH}.zip" "katana-${KATANA_VERSION#v}-checksums.txt" katana; \
|
||||
"katana_${KATANA_VERSION#v}_linux_${PD_ARCH}.zip" "$KATANA_SUM" katana; \
|
||||
fetch "$gh/ffuf/ffuf/releases/download/$FFUF_VERSION" \
|
||||
"ffuf_${FFUF_VERSION#v}_linux_${FFUF_ARCH}.tar.gz" "ffuf_${FFUF_VERSION#v}_checksums.txt" ffuf; \
|
||||
"ffuf_${FFUF_VERSION#v}_linux_${FFUF_ARCH}.tar.gz" "$FFUF_SUM" ffuf; \
|
||||
fetch "$gh/hahwul/dalfox/releases/download/$DALFOX_VERSION" \
|
||||
"dalfox-${DALFOX_VERSION}-linux-${DALFOX_ARCH}-musl.tar.gz" "dalfox-${DALFOX_VERSION}-linux-${DALFOX_ARCH}-musl.tar.gz.sha256" dalfox; \
|
||||
"dalfox-${DALFOX_VERSION}-linux-${DALFOX_ARCH}-musl.tar.gz" "$DALFOX_SUM" dalfox; \
|
||||
git clone --quiet --depth 1 --branch "$SQLMAP_VERSION" https://github.com/sqlmapproject/sqlmap /opt/sqlmap; \
|
||||
commit="$(git -C /opt/sqlmap rev-parse HEAD)"; \
|
||||
if [ "$commit" != "$SQLMAP_COMMIT" ]; then echo "sqlmap commit mismatch: want $SQLMAP_COMMIT got $commit" >&2; exit 1; fi; \
|
||||
@@ -110,26 +140,27 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ca-certificates ripgrep curl wget vim git jq unzip xz-utils \
|
||||
dnsutils iputils-ping netcat-openbsd inetutils-telnet whois nmap \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
# Node 22 from the official nodejs.org tarball, verified against its SHASUMS256.txt:
|
||||
# bookworm's apt nodejs is 18, Playwright requires >=20, and 20 is end-of-life.
|
||||
# 22 matches the CI and release workflows.
|
||||
# Node 22 from the official nodejs.org tarball: bookworm's apt nodejs is 18,
|
||||
# Playwright requires >=20, and 20 is end-of-life. 22 matches the CI and release
|
||||
# workflows. Verified against a digest pinned here rather than against the fetched
|
||||
# SHASUMS256.txt, for the reason given at the tools stage.
|
||||
ARG NODE_VERSION=22.23.4
|
||||
ARG NODE_SHA256_AMD64=1805bcd35e6815559bd2eb97db438d3fdb0eb3a5e6a977b7c1badca76337e9ce
|
||||
ARG NODE_SHA256_ARM64=6287aa46b6c375d816e63ae66d024fbbfd70404825aabdfdf898a970871d2f4b
|
||||
RUN set -eux; \
|
||||
case "${TARGETARCH:-amd64}" in \
|
||||
amd64) NODE_ARCH=x64 ;; \
|
||||
arm64) NODE_ARCH=arm64 ;; \
|
||||
amd64) NODE_ARCH=x64; want="$NODE_SHA256_AMD64" ;; \
|
||||
arm64) NODE_ARCH=arm64; want="$NODE_SHA256_ARM64" ;; \
|
||||
*) echo "unsupported architecture: ${TARGETARCH}" >&2; exit 1 ;; \
|
||||
esac; \
|
||||
base="https://nodejs.org/dist/v${NODE_VERSION}"; \
|
||||
asset="node-v${NODE_VERSION}-linux-${NODE_ARCH}.tar.xz"; \
|
||||
curl -fsSL "$base/$asset" -o "/tmp/$asset"; \
|
||||
curl -fsSL "$base/SHASUMS256.txt" -o /tmp/SHASUMS256.txt; \
|
||||
want="$(awk -v a="$asset" '$2 == a { print $1 }' /tmp/SHASUMS256.txt)"; \
|
||||
if [ -z "$want" ]; then echo "no checksum entry for $asset" >&2; exit 1; fi; \
|
||||
[ -n "$want" ] || { echo "no pinned digest for $asset" >&2; exit 1; }; \
|
||||
got="$(sha256sum "/tmp/$asset" | cut -d' ' -f1)"; \
|
||||
if [ "$want" != "$got" ]; then echo "checksum mismatch for $asset: want $want got $got" >&2; exit 1; fi; \
|
||||
tar -xJf "/tmp/$asset" -C /usr/local --strip-components=1 --no-same-owner; \
|
||||
rm -f "/tmp/$asset" /tmp/SHASUMS256.txt; \
|
||||
rm -f "/tmp/$asset"; \
|
||||
node --version; npm --version
|
||||
# Pinned, checksum-verified recon binaries and the sqlmap checkout from the tools stage.
|
||||
COPY --from=tools /out/ /usr/local/bin/
|
||||
|
||||
+54
-8
@@ -387,12 +387,57 @@ func IsRunDirName(name string) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
func workerSystem(proxyAddr, caCert, dataDir, runDir string) string {
|
||||
// CachePrefixEnv is the opt-in environment flag enabling the worker system
|
||||
// prompt cache-prefix split (P3.5 phase C). Off by default: when unset/false,
|
||||
// worker system segments and DynamicBoundary are byte-identical to the legacy
|
||||
// layout. Planner, mainagent, chat, and goals are unaffected.
|
||||
const CachePrefixEnv = "ARTEX_CACHE_PREFIX"
|
||||
|
||||
func cachePrefixEnabled() bool {
|
||||
switch strings.ToLower(strings.TrimSpace(os.Getenv(CachePrefixEnv))) {
|
||||
case "1", "true", "on", "yes":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// workerSystemParts splits workerSystem at the first per-intent volatile
|
||||
// segment (workerArtifactSpec(runDir)) without reordering any text:
|
||||
// - staticPart = body + untrustedContentTail + workerTrafficBlock
|
||||
// - dynamicHead = workerArtifactSpec(runDir) + noiseProfileBlock
|
||||
//
|
||||
// Custom DB prompts that reference {{.Now}} remain volatile inside staticPart.
|
||||
func workerSystemParts(proxyAddr, caCert, dataDir, runDir string) (staticPart, dynamicHead string) {
|
||||
body := renderSystem("worker", workerDefaultTmpl, WorkerVars{ProxyAddr: proxyAddr, DataDir: dataDir, Now: nowStr()})
|
||||
// caCert is present only when the recording MITM is on, which is exactly when
|
||||
// the traffic_* tools are registered — so it gates the traffic-tool note.
|
||||
// Optional finding guidance is added for every role after tool resolution.
|
||||
return body + untrustedContentTail(body) + workerTrafficBlock(caCert != "") + workerArtifactSpec(runDir) + noiseProfileBlock()
|
||||
staticPart = body + untrustedContentTail(body) + workerTrafficBlock(caCert != "")
|
||||
dynamicHead = workerArtifactSpec(runDir) + noiseProfileBlock()
|
||||
return staticPart, dynamicHead
|
||||
}
|
||||
|
||||
func workerSystem(proxyAddr, caCert, dataDir, runDir string) string {
|
||||
staticPart, dynamicHead := workerSystemParts(proxyAddr, caCert, dataDir, runDir)
|
||||
return staticPart + dynamicHead
|
||||
}
|
||||
|
||||
// assembleWorkerSystem builds the worker's SystemPrompt segments and
|
||||
// DynamicBoundary. When split is false (default), it delegates directly to
|
||||
// deferredSystem(staticPart+dynamicTail, def). When split is true
|
||||
// (ARTEX_CACHE_PREFIX=1), it emits two segments [staticPart, dynamic] with
|
||||
// DynamicBoundary=1 so sibling workers in the same task share the cached static
|
||||
// prefix, absorbing the leading "\n\n" separator from workerArtifactSpec so
|
||||
// strings.Join(system, "\n\n") is byte-identical to the unsplit prompt.
|
||||
func assembleWorkerSystem(staticPart, dynamicTail string, def DeferredInfo, split bool) (system []string, boundary int) {
|
||||
if !split {
|
||||
return deferredSystem(staticPart+dynamicTail, def)
|
||||
}
|
||||
dyn := strings.TrimPrefix(dynamicTail+def.FindingGuidance, "\n\n")
|
||||
if block := actool.RenderDeferredToolsBlock(def.GlobalNames); block != "" {
|
||||
dyn += "\n\n" + block
|
||||
}
|
||||
return []string{staticPart, dyn}, 1
|
||||
}
|
||||
|
||||
// renderIntentTask formats the claimed intent for the worker's launch USER message:
|
||||
@@ -524,18 +569,19 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
|
||||
// inherit a parent run's background into the action reviewer.
|
||||
ctx = intercept.WithReviewContext(ctx, runDir, intercept.ReviewBackground{})
|
||||
overview := renderWorkerGraphOverview(tsx.graphOverviewData(), newDataEnvelope())
|
||||
sysBody := workerSystem(w.proxyAddr, w.proxyCACert, w.workDir, runDir) + frameLessonsBlock(w.lessonsBlock)
|
||||
staticPart, dynamicTail := workerSystemParts(w.proxyAddr, w.proxyCACert, w.workDir, runDir)
|
||||
dynamicTail += frameLessonsBlock(w.lessonsBlock)
|
||||
if w.wantConstraints() {
|
||||
sysBody += constraintBlock(ts) // inject the operation constraints (if any) into the system prompt; the worker strictly obeys them during execution
|
||||
dynamicTail += constraintBlock(ts) // inject the operation constraints (if any) into the system prompt; the worker strictly obeys them during execution
|
||||
}
|
||||
// intent block → intent-anchored assets block → launch instruction, appended to
|
||||
// the system tail in that order (same append pattern as constraintBlock).
|
||||
sysBody += renderIntentTask(intent)
|
||||
dynamicTail += renderIntentTask(intent)
|
||||
if as != nil {
|
||||
if ids := intentAssetIDs(intent); len(ids) > 0 {
|
||||
if assets, err := as.GetByIDs(ids); err == nil && len(assets) > 0 {
|
||||
if b, err := json.Marshal(assets); err == nil {
|
||||
sysBody += "\n\nThe target assets corresponding to this intent's asset_ids:\n" + string(b)
|
||||
dynamicTail += "\n\nThe target assets corresponding to this intent's asset_ids:\n" + string(b)
|
||||
}
|
||||
// The assets this intent explicitly targets → automatically join the task's
|
||||
// test scope (same conservative granularity as insertAssets). upsertTaskScope's
|
||||
@@ -551,8 +597,8 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
|
||||
}
|
||||
}
|
||||
}
|
||||
sysBody += "\n\nStart executing the intent above: do only it, produce only facts, assets, findings, stop when done."
|
||||
system, boundary := deferredSystem(sysBody, def)
|
||||
dynamicTail += "\n\nStart executing the intent above: do only it, produce only facts, assets, findings, stop when done."
|
||||
system, boundary := assembleWorkerSystem(staticPart, dynamicTail, def, cachePrefixEnabled())
|
||||
// The task-level deadline (injected via ctx) clamps this run's wall-clock budget +
|
||||
// decides the wrap-up prompt (see taskclock.go).
|
||||
tc := taskClockFrom(ctx)
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/llmrec"
|
||||
"github.com/Autumn-27/norma/llm"
|
||||
actool "github.com/Autumn-27/norma/tool"
|
||||
)
|
||||
|
||||
type captureReqProvider struct {
|
||||
mu sync.Mutex
|
||||
reqs []llm.CompletionRequest
|
||||
}
|
||||
|
||||
func (p *captureReqProvider) Stream(_ context.Context, req llm.CompletionRequest) iter.Seq2[llm.StreamEvent, error] {
|
||||
p.mu.Lock()
|
||||
p.reqs = append(p.reqs, req)
|
||||
p.mu.Unlock()
|
||||
return func(yield func(llm.StreamEvent, error) bool) {
|
||||
if !yield(llm.StreamEvent{Type: llm.SEMessageStart, Usage: llm.Usage{InputTokens: 100}}, nil) {
|
||||
return
|
||||
}
|
||||
if !yield(llm.StreamEvent{Type: llm.SETextDelta, Text: "done"}, nil) {
|
||||
return
|
||||
}
|
||||
yield(llm.StreamEvent{Type: llm.SEMessageDelta, StopReason: "end_turn", Usage: llm.Usage{OutputTokens: 4}}, nil)
|
||||
}
|
||||
}
|
||||
|
||||
func (p *captureReqProvider) Complete(_ context.Context, req llm.CompletionRequest) (llm.Message, string, llm.Usage, error) {
|
||||
p.mu.Lock()
|
||||
p.reqs = append(p.reqs, req)
|
||||
p.mu.Unlock()
|
||||
return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{llm.TextBlock("done")}}, "end_turn", llm.Usage{InputTokens: 100, OutputTokens: 4}, nil
|
||||
}
|
||||
|
||||
func (p *captureReqProvider) last() llm.CompletionRequest {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if len(p.reqs) == 0 {
|
||||
return llm.CompletionRequest{}
|
||||
}
|
||||
return p.reqs[len(p.reqs)-1]
|
||||
}
|
||||
|
||||
func TestAssembleWorkerSystemFlagOffAndOnByteExact(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
caCert string
|
||||
globalNames []string
|
||||
guidance string
|
||||
}{
|
||||
{name: "no_deferred_no_traffic", caCert: "", globalNames: nil, guidance: ""},
|
||||
{name: "with_deferred_and_traffic", caCert: "/ca.pem", globalNames: []string{"mcp__zap__scan", "mcp__burp__send"}, guidance: "\n\n[Finding guidance]"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
def := DeferredInfo{
|
||||
GlobalNames: tc.globalNames,
|
||||
FindingGuidance: tc.guidance,
|
||||
}
|
||||
buildTail := func(runDir, intentSummary, assetDomain string) (string, string) {
|
||||
staticPart, dynamicTail := workerSystemParts("127.0.0.1:8788", tc.caCert, "/data", runDir)
|
||||
dynamicTail += "\n\n## Prior evidence\n- lesson A"
|
||||
dynamicTail += "\n\n[Constraints]\n- do not touch prod"
|
||||
dynamicTail += renderIntentTask(&db.Node{ID: 7, Payload: json.RawMessage(`{"summary":"` + intentSummary + `"}`)})
|
||||
dynamicTail += "\n\nThe target assets corresponding to this intent's asset_ids:\n[{\"domain\":\"" + assetDomain + "\"}]"
|
||||
dynamicTail += "\n\nStart executing the intent above: do only it, produce only facts, assets, findings, stop when done."
|
||||
return staticPart, dynamicTail
|
||||
}
|
||||
|
||||
s1, d1 := buildTail("/data/tasks/1/i10", "probe auth", "alpha.example")
|
||||
s2, d2 := buildTail("/data/tasks/1/i20", "probe billing", "beta.example")
|
||||
|
||||
// Legacy baseline (equivalent to flag off).
|
||||
legacySys1, legacyBound1 := deferredSystem(workerSystem("127.0.0.1:8788", tc.caCert, "/data", "/data/tasks/1/i10")+strings.TrimPrefix(d1, workerArtifactSpec("/data/tasks/1/i10")+noiseProfileBlock()), def)
|
||||
offSys1, offBound1 := assembleWorkerSystem(s1, d1, def, false)
|
||||
if offBound1 != legacyBound1 || !slices.Equal(offSys1, legacySys1) {
|
||||
t.Fatalf("flag off diverged from legacy: got (%d, len=%d) want (%d, len=%d)", offBound1, len(offSys1), legacyBound1, len(legacySys1))
|
||||
}
|
||||
|
||||
// Flag ON: split into [static, dynamic] with DynamicBoundary=1.
|
||||
onSys1, onBound1 := assembleWorkerSystem(s1, d1, def, true)
|
||||
onSys2, onBound2 := assembleWorkerSystem(s2, d2, def, true)
|
||||
|
||||
if onBound1 != 1 || onBound2 != 1 || len(onSys1) != 2 || len(onSys2) != 2 {
|
||||
t.Fatalf("flag on shape = (%d, %d) / (%d, %d), want boundary=1 len=2", onBound1, len(onSys1), onBound2, len(onSys2))
|
||||
}
|
||||
// Static segment identical across sibling workers.
|
||||
if onSys1[0] != onSys2[0] {
|
||||
t.Fatalf("static prefix differs between sibling workers")
|
||||
}
|
||||
// Dynamic segment differs and carries per-intent data + deferred block.
|
||||
if onSys1[1] == onSys2[1] {
|
||||
t.Fatalf("dynamic segment unexpectedly identical")
|
||||
}
|
||||
for _, want := range []string{"/data/tasks/1/i10", "probe auth", "alpha.example", "Start executing the intent above:"} {
|
||||
if !strings.Contains(onSys1[1], want) {
|
||||
t.Errorf("dynamic[1] missing %q", want)
|
||||
}
|
||||
if strings.Contains(onSys1[0], want) {
|
||||
t.Errorf("static[0] unexpectedly contains volatile %q", want)
|
||||
}
|
||||
}
|
||||
if len(tc.globalNames) > 0 {
|
||||
if !strings.Contains(onSys1[1], "<available-deferred-tools>") || !strings.Contains(onSys1[1], "mcp__zap__scan") {
|
||||
t.Errorf("deferred tools block missing from dynamic segment")
|
||||
}
|
||||
if strings.Contains(onSys1[0], "<available-deferred-tools>") {
|
||||
t.Errorf("deferred tools block unexpectedly in static segment")
|
||||
}
|
||||
}
|
||||
|
||||
// Byte-exact visible text: Join(onSys, "\n\n") == Join(offSys, "\n\n").
|
||||
wantVisible1 := strings.Join(offSys1, "\n\n")
|
||||
gotVisible1 := strings.Join(onSys1, "\n\n")
|
||||
if gotVisible1 != wantVisible1 {
|
||||
t.Fatalf("visible text mismatch (len got=%d want=%d)", len(gotVisible1), len(wantVisible1))
|
||||
}
|
||||
offSys2, _ := assembleWorkerSystem(s2, d2, def, false)
|
||||
if strings.Join(onSys2, "\n\n") != strings.Join(offSys2, "\n\n") {
|
||||
t.Fatalf("worker 2 visible text mismatch")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkerExecuteCachePrefixEndToEnd(t *testing.T) {
|
||||
d := testDB(t)
|
||||
t.Cleanup(func() { _ = d.Close() })
|
||||
|
||||
stamp := time.Now().UnixNano()
|
||||
task, err := d.CreateTask(fmt.Sprintf("cache-prefix-%d", stamp), "verify worker cache prefix", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = d.DeleteTask(task.ID) })
|
||||
|
||||
ts := d.Exploration(task.ExplorationID)
|
||||
as := d.Assets()
|
||||
var asset1ID, asset2ID int64
|
||||
dom1 := fmt.Sprintf("alpha-%d.cache.test", stamp)
|
||||
dom2 := fmt.Sprintf("beta-%d.cache.test", stamp)
|
||||
if err := d.QueryRow(`INSERT INTO assets(type,domain,root_domain,task_ids) VALUES('subdomain',$1,$1,ARRAY[$2]::bigint[]) RETURNING id`, dom1, task.ID).Scan(&asset1ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := d.QueryRow(`INSERT INTO assets(type,domain,root_domain,task_ids) VALUES('subdomain',$1,$1,ARRAY[$2]::bigint[]) RETURNING id`, dom2, task.ID).Scan(&asset2ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
_, _ = d.Exec(`DELETE FROM assets WHERE id IN ($1,$2)`, asset1ID, asset2ID)
|
||||
})
|
||||
|
||||
if _, err := ts.AddConstraint("deny", "stay within test scope", "human"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
p1Map := map[string]any{"summary": "inspect alpha auth", "asset_ids": []int64{asset1ID}}
|
||||
p2Map := map[string]any{"summary": "inspect beta checkout", "asset_ids": []int64{asset2ID}}
|
||||
i1ID, err := ts.AddNode("intent", p1Map, 0, "running", "planner", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
i2ID, err := ts.AddNode("intent", p2Map, 0, "running", "planner", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw1, _ := json.Marshal(p1Map)
|
||||
raw2, _ := json.Marshal(p2Map)
|
||||
intent1 := &db.Node{ID: i1ID, Payload: raw1}
|
||||
intent2 := &db.Node{ID: i2ID, Payload: raw2}
|
||||
|
||||
// Inject a global deferred MCP name so the deferred block is exercised end-to-end.
|
||||
prevAugment := ToolAugment()
|
||||
SetToolAugment(func(_ context.Context, _ string) ([]actool.CoreTool, DeferredInfo, func()) {
|
||||
return nil, DeferredInfo{
|
||||
GlobalNames: []string{"mcp__demo__scanner"},
|
||||
}, func() {}
|
||||
})
|
||||
t.Cleanup(func() { SetToolAugment(prevAugment) })
|
||||
|
||||
workDir := t.TempDir()
|
||||
runWorker := func(intent *db.Node, lessons string) llm.CompletionRequest {
|
||||
t.Helper()
|
||||
prov := &captureReqProvider{}
|
||||
w := NewWorker(prov, "test-model", workDir, nil, 0, 2)
|
||||
w.SetProxy("127.0.0.1:8788", "/ca.pem")
|
||||
w.SetLessonsBlock(lessons)
|
||||
if _, _, err := w.Execute(t.Context(), "work#1", task.ID, as, ts, intent, passHooks{}, nil, nil, nil); err != nil {
|
||||
t.Fatalf("Worker.Execute(intent=%d): %v", intent.ID, err)
|
||||
}
|
||||
return prov.last()
|
||||
}
|
||||
|
||||
// 1. Flag OFF (default): legacy segments and boundary. With a global deferred
|
||||
// tool block, deferredSystem emits [sysText, deferredBlock] with boundary=2,
|
||||
// and norma's UntrustedOutputBoundary (on by default) appends its fixed note as
|
||||
// System[2], giving len=3 and boundary=2.
|
||||
t.Setenv(CachePrefixEnv, "")
|
||||
req1Off := runWorker(intent1, "")
|
||||
req2Off := runWorker(intent2, "")
|
||||
if req1Off.DynamicBoundary != 2 || len(req1Off.System) != 3 {
|
||||
t.Fatalf("flag off worker 1: boundary=%d len=%d, want 2/3", req1Off.DynamicBoundary, len(req1Off.System))
|
||||
}
|
||||
if req1Off.System[0] == req2Off.System[0] {
|
||||
t.Fatalf("expected flag-off worker system[0] to differ because runDir/intent are inside it")
|
||||
}
|
||||
|
||||
// 2. Flag ON: static prefix identical across sibling workers, dynamic differs,
|
||||
// visible text byte-identical, tools unchanged, deferred block inside dynamic,
|
||||
// norma's appended untrusted-output note preserved as System[2].
|
||||
t.Setenv(CachePrefixEnv, "1")
|
||||
req1On := runWorker(intent1, "")
|
||||
req2On := runWorker(intent2, "")
|
||||
|
||||
if req1On.DynamicBoundary != 1 || req2On.DynamicBoundary != 1 || len(req1On.System) != 3 || len(req2On.System) != 3 {
|
||||
t.Fatalf("flag on shape: w1=(%d,%d) w2=(%d,%d), want (1,3)",
|
||||
req1On.DynamicBoundary, len(req1On.System), req2On.DynamicBoundary, len(req2On.System))
|
||||
}
|
||||
if req1On.System[2] != req1Off.System[2] || req2On.System[2] != req2Off.System[2] {
|
||||
t.Fatalf("norma untrusted-output note segment changed under ARTEX_CACHE_PREFIX=1")
|
||||
}
|
||||
if req1On.System[0] != req2On.System[0] {
|
||||
t.Fatalf("flag on static prefix differs between sibling workers")
|
||||
}
|
||||
if req1On.System[1] == req2On.System[1] {
|
||||
t.Fatalf("flag on dynamic suffix should differ between sibling workers")
|
||||
}
|
||||
|
||||
runDir1 := filepath.Join(workDir, "tasks", fmt.Sprint(task.ID), fmt.Sprintf("i%d", intent1.ID))
|
||||
runDir2 := filepath.Join(workDir, "tasks", fmt.Sprint(task.ID), fmt.Sprintf("i%d", intent2.ID))
|
||||
for _, pair := range []struct {
|
||||
name string
|
||||
req llm.CompletionRequest
|
||||
runDir string
|
||||
intent string
|
||||
domain string
|
||||
}{
|
||||
{"worker1", req1On, runDir1, "inspect alpha auth", dom1},
|
||||
{"worker2", req2On, runDir2, "inspect beta checkout", dom2},
|
||||
} {
|
||||
for _, needle := range []string{pair.runDir, pair.intent, pair.domain, "stay within test scope", "Start executing the intent above:", "<available-deferred-tools>", "mcp__demo__scanner"} {
|
||||
if !strings.Contains(pair.req.System[1], needle) {
|
||||
t.Errorf("%s dynamic segment missing %q", pair.name, needle)
|
||||
}
|
||||
if strings.Contains(pair.req.System[0], needle) {
|
||||
t.Errorf("%s static segment unexpectedly contains %q", pair.name, needle)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if strings.Join(req1On.System, "\n\n") != strings.Join(req1Off.System, "\n\n") {
|
||||
t.Fatalf("worker 1 visible prompt changed under ARTEX_CACHE_PREFIX=1")
|
||||
}
|
||||
if strings.Join(req2On.System, "\n\n") != strings.Join(req2Off.System, "\n\n") {
|
||||
t.Fatalf("worker 2 visible prompt changed under ARTEX_CACHE_PREFIX=1")
|
||||
}
|
||||
|
||||
// With RSI lessons enabled (framed with a per-run random nonce), lessons stay in
|
||||
// System[1] so System[0] remains byte-identical across sibling workers.
|
||||
const sampleLessons = "- [positive, 1 evidence] sqli: verified"
|
||||
req1Lessons := runWorker(intent1, sampleLessons)
|
||||
req2Lessons := runWorker(intent2, sampleLessons)
|
||||
if req1Lessons.System[0] != req1On.System[0] || req2Lessons.System[0] != req1On.System[0] {
|
||||
t.Fatalf("lessons block leaked into static prefix")
|
||||
}
|
||||
if !strings.Contains(req1Lessons.System[1], sampleLessons) || !strings.Contains(req2Lessons.System[1], sampleLessons) {
|
||||
t.Fatalf("lessons block missing from dynamic segment")
|
||||
}
|
||||
|
||||
toolsOffJSON, _ := json.Marshal(req1Off.Tools)
|
||||
toolsOnJSON, _ := json.Marshal(req1On.Tools)
|
||||
if string(toolsOnJSON) != string(toolsOffJSON) {
|
||||
t.Fatalf("tools changed under ARTEX_CACHE_PREFIX=1")
|
||||
}
|
||||
|
||||
// Telemetry observes non-zero static & dynamic split with identical static bytes.
|
||||
m1 := llmrec.MeasureContext(req1On)
|
||||
m2 := llmrec.MeasureContext(req2On)
|
||||
if m1.SystemStaticBytes <= 0 || m1.SystemDynamicBytes <= 0 || m1.SystemStaticBytes != m2.SystemStaticBytes {
|
||||
t.Fatalf("MeasureContext mismatch: m1=(%d,%d) m2=(%d,%d)",
|
||||
m1.SystemStaticBytes, m1.SystemDynamicBytes, m2.SystemStaticBytes, m2.SystemDynamicBytes)
|
||||
}
|
||||
|
||||
// 3. Planner and mainagent are completely unchanged by ARTEX_CACHE_PREFIX=1.
|
||||
runPlanner := func() llm.CompletionRequest {
|
||||
prov := &captureReqProvider{}
|
||||
if _, _, err := NewPlanner(prov, "test-model", workDir, nil, 0, 2).Plan(t.Context(), task.ID, as, ts, "goal", nil, passHooks{}, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return prov.last()
|
||||
}
|
||||
runMain := func() llm.CompletionRequest {
|
||||
prov := &captureReqProvider{}
|
||||
if _, err := NewMainAgent(prov, "test-model", workDir, nil, 0, 2).Chat(t.Context(), task.ID, 0, as, ts, "goal", "status?", passHooks{}, nil, nil, nil, nil, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return prov.last()
|
||||
}
|
||||
|
||||
t.Setenv(CachePrefixEnv, "")
|
||||
planOff := runPlanner()
|
||||
mainOff := runMain()
|
||||
t.Setenv(CachePrefixEnv, "1")
|
||||
planOn := runPlanner()
|
||||
mainOn := runMain()
|
||||
|
||||
if planOn.DynamicBoundary != planOff.DynamicBoundary || !slices.Equal(planOn.System, planOff.System) {
|
||||
t.Fatalf("planner changed under ARTEX_CACHE_PREFIX=1: off=(%d,len=%d) on=(%d,len=%d)",
|
||||
planOff.DynamicBoundary, len(planOff.System), planOn.DynamicBoundary, len(planOn.System))
|
||||
}
|
||||
if mainOn.DynamicBoundary != mainOff.DynamicBoundary || !slices.Equal(mainOn.System, mainOff.System) {
|
||||
t.Fatalf("mainagent changed under ARTEX_CACHE_PREFIX=1: off=(%d,len=%d) on=(%d,len=%d)",
|
||||
mainOff.DynamicBoundary, len(mainOff.System), mainOn.DynamicBoundary, len(mainOn.System))
|
||||
}
|
||||
}
|
||||
@@ -100,7 +100,7 @@ func TestEvidenceRewriteFlagMigrationAndRoundTrip(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, e := range entries {
|
||||
if e.IsDir() || strings.HasPrefix(e.Name(), "0008_") {
|
||||
if e.IsDir() || e.Name() >= "0008_" {
|
||||
continue
|
||||
}
|
||||
data, err := fs.ReadFile(migrationsFS(), e.Name())
|
||||
|
||||
@@ -0,0 +1,316 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// LLMContext is one opt-in context-size telemetry row written when
|
||||
// ARTEX_CONTEXT_TELEMETRY=1. It records only byte sizes, counts, token usage,
|
||||
// and call metadata — never any prompt, tool, or message content.
|
||||
type LLMContext struct {
|
||||
TaskID string `json:"task_id"`
|
||||
Worker string `json:"worker"`
|
||||
Model string `json:"model"`
|
||||
Profile string `json:"profile"`
|
||||
ToolCount int `json:"tool_count"`
|
||||
MessageCount int `json:"message_count"`
|
||||
SystemStaticBytes int64 `json:"system_static_bytes"`
|
||||
SystemDynamicBytes int64 `json:"system_dynamic_bytes"`
|
||||
ToolsBytes int64 `json:"tools_bytes"`
|
||||
MessagesUserBytes int64 `json:"messages_user_bytes"`
|
||||
MessagesAssistantBytes int64 `json:"messages_assistant_bytes"`
|
||||
MessagesOtherBytes int64 `json:"messages_other_bytes"`
|
||||
ToolResultBytes int64 `json:"tool_result_bytes"`
|
||||
LastMessageBytes int64 `json:"last_message_bytes"`
|
||||
InputTokens int `json:"input_tokens"`
|
||||
OutputTokens int `json:"output_tokens"`
|
||||
CacheRead int `json:"cache_read"`
|
||||
CacheWrite int `json:"cache_write"`
|
||||
Status string `json:"status"`
|
||||
}
|
||||
|
||||
// TotalMeasuredBytes returns the partition sum of request section bytes
|
||||
// (excluding ToolResultBytes, which is a subset of MessagesUserBytes).
|
||||
func (c LLMContext) TotalMeasuredBytes() int64 {
|
||||
return c.SystemStaticBytes + c.SystemDynamicBytes + c.ToolsBytes +
|
||||
c.MessagesUserBytes + c.MessagesAssistantBytes + c.MessagesOtherBytes
|
||||
}
|
||||
|
||||
// LLMContextSectionShares holds the pro-rata share (0..1) of real input_tokens
|
||||
// attributed to each request section by byte proportion. The six partition
|
||||
// fields (SystemStatic through MessagesOther) sum to 1.0 whenever at least one
|
||||
// call has both measured bytes > 0 and input_tokens > 0. ToolResultSubset is an
|
||||
// informational sub-share of MessagesUser (not added to the sum).
|
||||
type LLMContextSectionShares struct {
|
||||
SystemStatic float64 `json:"system_static"`
|
||||
SystemDynamic float64 `json:"system_dynamic"`
|
||||
Tools float64 `json:"tools"`
|
||||
MessagesUser float64 `json:"messages_user"`
|
||||
MessagesAsst float64 `json:"messages_assistant"`
|
||||
MessagesOther float64 `json:"messages_other"`
|
||||
ToolResultSubset float64 `json:"tool_result_subset"`
|
||||
}
|
||||
|
||||
// ComputeLLMContextShares deterministically allocates each call's real
|
||||
// input_tokens across sections proportionally to measured bytes, then divides
|
||||
// each section's attributed tokens by the total attributed input tokens.
|
||||
// Rows with zero measured bytes or zero input tokens are ignored so they never
|
||||
// produce NaN or distort the proportions.
|
||||
func ComputeLLMContextShares(rows []LLMContext) LLMContextSectionShares {
|
||||
var (
|
||||
totInput float64
|
||||
sysStat float64
|
||||
sysDyn float64
|
||||
tools float64
|
||||
msgUser float64
|
||||
msgAsst float64
|
||||
msgOther float64
|
||||
toolRes float64
|
||||
)
|
||||
for _, r := range rows {
|
||||
totalBytes := r.TotalMeasuredBytes()
|
||||
if totalBytes <= 0 || r.InputTokens <= 0 {
|
||||
continue
|
||||
}
|
||||
in := float64(r.InputTokens)
|
||||
tb := float64(totalBytes)
|
||||
totInput += in
|
||||
sysStat += in * (float64(r.SystemStaticBytes) / tb)
|
||||
sysDyn += in * (float64(r.SystemDynamicBytes) / tb)
|
||||
tools += in * (float64(r.ToolsBytes) / tb)
|
||||
msgUser += in * (float64(r.MessagesUserBytes) / tb)
|
||||
msgAsst += in * (float64(r.MessagesAssistantBytes) / tb)
|
||||
msgOther += in * (float64(r.MessagesOtherBytes) / tb)
|
||||
toolRes += in * (float64(r.ToolResultBytes) / tb)
|
||||
}
|
||||
if totInput <= 0 {
|
||||
return LLMContextSectionShares{}
|
||||
}
|
||||
return LLMContextSectionShares{
|
||||
SystemStatic: sysStat / totInput,
|
||||
SystemDynamic: sysDyn / totInput,
|
||||
Tools: tools / totInput,
|
||||
MessagesUser: msgUser / totInput,
|
||||
MessagesAsst: msgAsst / totInput,
|
||||
MessagesOther: msgOther / totInput,
|
||||
ToolResultSubset: toolRes / totInput,
|
||||
}
|
||||
}
|
||||
|
||||
// LLMContextGroupStat aggregates llm_context rows for one (worker, model) pair.
|
||||
type LLMContextGroupStat struct {
|
||||
Worker string `json:"worker"`
|
||||
Model string `json:"model"`
|
||||
Calls int `json:"calls"`
|
||||
InputTokensTotal int64 `json:"input_tokens_total"`
|
||||
OutputTokensTotal int64 `json:"output_tokens_total"`
|
||||
CacheReadTotal int64 `json:"cache_read_total"`
|
||||
CacheWriteTotal int64 `json:"cache_write_total"`
|
||||
CacheReadRatio float64 `json:"cache_read_ratio"`
|
||||
InputP50 float64 `json:"input_p50"`
|
||||
InputP90 float64 `json:"input_p90"`
|
||||
InputP99 float64 `json:"input_p99"`
|
||||
InputMax int `json:"input_max"`
|
||||
AvgToolCount float64 `json:"avg_tool_count"`
|
||||
AvgMessageCount float64 `json:"avg_message_count"`
|
||||
AvgLastMessageBytes float64 `json:"avg_last_message_bytes"`
|
||||
Shares LLMContextSectionShares `json:"shares"`
|
||||
}
|
||||
|
||||
// LLMContextTaskStat summarizes context consumption for one task_id.
|
||||
type LLMContextTaskStat struct {
|
||||
TaskID string `json:"task_id"`
|
||||
Calls int `json:"calls"`
|
||||
InputTokensTotal int64 `json:"input_tokens_total"`
|
||||
CacheReadTotal int64 `json:"cache_read_total"`
|
||||
CacheReadRatio float64 `json:"cache_read_ratio"`
|
||||
InputP90 float64 `json:"input_p90"`
|
||||
InputMax int `json:"input_max"`
|
||||
Shares LLMContextSectionShares `json:"shares"`
|
||||
}
|
||||
|
||||
// LLMContextSummary is the aggregation output for Phase A context measurement.
|
||||
type LLMContextSummary struct {
|
||||
ByWorkerModel []LLMContextGroupStat `json:"by_worker_model"`
|
||||
TopTasks []LLMContextTaskStat `json:"top_tasks"`
|
||||
}
|
||||
|
||||
// LLMContextQuery scopes LLMContextSummary.
|
||||
type LLMContextQuery struct {
|
||||
Days int // > 0 restricts to ts >= now() - Days; <= 0 = all time
|
||||
TaskID string // optional exact task_id filter
|
||||
TopTasks int // max rows in TopTasks (default 20, clamped 1..100)
|
||||
}
|
||||
|
||||
// InsertLLMContext appends one context-telemetry row. Best-effort: callers log
|
||||
// and continue on error so telemetry never breaks an LLM call.
|
||||
func (d *DB) InsertLLMContext(c *LLMContext) error {
|
||||
status := strings.TrimSpace(c.Status)
|
||||
if status == "" {
|
||||
status = "ok"
|
||||
}
|
||||
_, err := d.Exec(`
|
||||
INSERT INTO llm_context(
|
||||
task_id, worker, model, profile,
|
||||
tool_count, message_count,
|
||||
system_static_bytes, system_dynamic_bytes, tools_bytes,
|
||||
messages_user_bytes, messages_assistant_bytes, messages_other_bytes,
|
||||
tool_result_bytes, last_message_bytes,
|
||||
input_tokens, output_tokens, cache_read, cache_write, status
|
||||
) VALUES (
|
||||
NULLIF($1,''), NULLIF($2,''), NULLIF($3,''), NULLIF($4,''),
|
||||
$5, $6,
|
||||
$7, $8, $9,
|
||||
$10, $11, $12,
|
||||
$13, $14,
|
||||
$15, $16, $17, $18, $19
|
||||
)`,
|
||||
c.TaskID, c.Worker, c.Model, c.Profile,
|
||||
c.ToolCount, c.MessageCount,
|
||||
c.SystemStaticBytes, c.SystemDynamicBytes, c.ToolsBytes,
|
||||
c.MessagesUserBytes, c.MessagesAssistantBytes, c.MessagesOtherBytes,
|
||||
c.ToolResultBytes, c.LastMessageBytes,
|
||||
c.InputTokens, c.OutputTokens, c.CacheRead, c.CacheWrite, status)
|
||||
return err
|
||||
}
|
||||
|
||||
const llmContextTotalBytesExpr = `(system_static_bytes + system_dynamic_bytes + tools_bytes + messages_user_bytes + messages_assistant_bytes + messages_other_bytes)`
|
||||
|
||||
// llmContextSharesSelect produces the 7 pro-rata section-share SQL expressions
|
||||
// matching ComputeLLMContextShares.
|
||||
const llmContextSharesSelect = `
|
||||
COALESCE(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens * (system_static_bytes::double precision / ` + llmContextTotalBytesExpr + `::double precision) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0), 0),
|
||||
COALESCE(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens * (system_dynamic_bytes::double precision / ` + llmContextTotalBytesExpr + `::double precision) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0), 0),
|
||||
COALESCE(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens * (tools_bytes::double precision / ` + llmContextTotalBytesExpr + `::double precision) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0), 0),
|
||||
COALESCE(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens * (messages_user_bytes::double precision / ` + llmContextTotalBytesExpr + `::double precision) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0), 0),
|
||||
COALESCE(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens * (messages_assistant_bytes::double precision / ` + llmContextTotalBytesExpr + `::double precision) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0), 0),
|
||||
COALESCE(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens * (messages_other_bytes::double precision / ` + llmContextTotalBytesExpr + `::double precision) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0), 0),
|
||||
COALESCE(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens * (tool_result_bytes::double precision / ` + llmContextTotalBytesExpr + `::double precision) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN ` + llmContextTotalBytesExpr + ` > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0), 0)`
|
||||
|
||||
// LLMContextSummary aggregates llm_context by (worker, model) and returns the
|
||||
// top tasks by total input_tokens, with input percentiles (p50/p90/p99),
|
||||
// pro-rata section shares over real input_tokens, and cache_read/input ratio.
|
||||
func (d *DB) LLMContextSummary(q LLMContextQuery) (*LLMContextSummary, error) {
|
||||
topLimit := q.TopTasks
|
||||
if topLimit <= 0 {
|
||||
topLimit = 20
|
||||
}
|
||||
if topLimit > 100 {
|
||||
topLimit = 100
|
||||
}
|
||||
|
||||
var conds []string
|
||||
var args []any
|
||||
addCond := func(expr string, val any) {
|
||||
args = append(args, val)
|
||||
conds = append(conds, fmt.Sprintf(expr, len(args)))
|
||||
}
|
||||
if q.Days > 0 {
|
||||
addCond("ts >= now() - make_interval(days => $%d)", q.Days)
|
||||
}
|
||||
if tid := strings.TrimSpace(q.TaskID); tid != "" {
|
||||
addCond("COALESCE(task_id,'') = $%d", tid)
|
||||
}
|
||||
where := ""
|
||||
if len(conds) > 0 {
|
||||
where = " WHERE " + strings.Join(conds, " AND ")
|
||||
}
|
||||
|
||||
out := &LLMContextSummary{
|
||||
ByWorkerModel: []LLMContextGroupStat{},
|
||||
TopTasks: []LLMContextTaskStat{},
|
||||
}
|
||||
|
||||
groupSQL := `
|
||||
SELECT COALESCE(NULLIF(worker,''),'(unknown)') AS w,
|
||||
COALESCE(NULLIF(model,''),'(unknown)') AS m,
|
||||
COUNT(*),
|
||||
COALESCE(SUM(input_tokens), 0),
|
||||
COALESCE(SUM(output_tokens), 0),
|
||||
COALESCE(SUM(cache_read), 0),
|
||||
COALESCE(SUM(cache_write), 0),
|
||||
COALESCE(SUM(cache_read)::double precision / NULLIF(SUM(input_tokens)::double precision, 0), 0),
|
||||
COALESCE(PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY input_tokens), 0),
|
||||
COALESCE(PERCENTILE_CONT(0.90) WITHIN GROUP (ORDER BY input_tokens), 0),
|
||||
COALESCE(PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY input_tokens), 0),
|
||||
COALESCE(MAX(input_tokens), 0),
|
||||
COALESCE(AVG(tool_count)::double precision, 0),
|
||||
COALESCE(AVG(message_count)::double precision, 0),
|
||||
COALESCE(AVG(last_message_bytes)::double precision, 0),` + llmContextSharesSelect + `
|
||||
FROM llm_context` + where + `
|
||||
GROUP BY w, m
|
||||
ORDER BY SUM(input_tokens) DESC, w ASC, m ASC`
|
||||
|
||||
rows, err := d.Query(groupSQL, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var g LLMContextGroupStat
|
||||
if err := rows.Scan(
|
||||
&g.Worker, &g.Model, &g.Calls,
|
||||
&g.InputTokensTotal, &g.OutputTokensTotal, &g.CacheReadTotal, &g.CacheWriteTotal,
|
||||
&g.CacheReadRatio,
|
||||
&g.InputP50, &g.InputP90, &g.InputP99, &g.InputMax,
|
||||
&g.AvgToolCount, &g.AvgMessageCount, &g.AvgLastMessageBytes,
|
||||
&g.Shares.SystemStatic, &g.Shares.SystemDynamic, &g.Shares.Tools,
|
||||
&g.Shares.MessagesUser, &g.Shares.MessagesAsst, &g.Shares.MessagesOther,
|
||||
&g.Shares.ToolResultSubset,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out.ByWorkerModel = append(out.ByWorkerModel, g)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
taskConds := append([]string{}, conds...)
|
||||
taskConds = append(taskConds, "COALESCE(task_id,'') <> ''")
|
||||
taskArgs := append([]any{}, args...)
|
||||
taskArgs = append(taskArgs, topLimit)
|
||||
limitPlaceholder := fmt.Sprintf("$%d", len(taskArgs))
|
||||
|
||||
taskSQL := `
|
||||
SELECT task_id,
|
||||
COUNT(*),
|
||||
COALESCE(SUM(input_tokens), 0),
|
||||
COALESCE(SUM(cache_read), 0),
|
||||
COALESCE(SUM(cache_read)::double precision / NULLIF(SUM(input_tokens)::double precision, 0), 0),
|
||||
COALESCE(PERCENTILE_CONT(0.90) WITHIN GROUP (ORDER BY input_tokens), 0),
|
||||
COALESCE(MAX(input_tokens), 0),` + llmContextSharesSelect + `
|
||||
FROM llm_context
|
||||
WHERE ` + strings.Join(taskConds, " AND ") + `
|
||||
GROUP BY task_id
|
||||
ORDER BY SUM(input_tokens) DESC, task_id ASC
|
||||
LIMIT ` + limitPlaceholder
|
||||
|
||||
trows, err := d.Query(taskSQL, taskArgs...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer trows.Close()
|
||||
for trows.Next() {
|
||||
var ts LLMContextTaskStat
|
||||
if err := trows.Scan(
|
||||
&ts.TaskID, &ts.Calls, &ts.InputTokensTotal, &ts.CacheReadTotal,
|
||||
&ts.CacheReadRatio, &ts.InputP90, &ts.InputMax,
|
||||
&ts.Shares.SystemStatic, &ts.Shares.SystemDynamic, &ts.Shares.Tools,
|
||||
&ts.Shares.MessagesUser, &ts.Shares.MessagesAsst, &ts.Shares.MessagesOther,
|
||||
&ts.Shares.ToolResultSubset,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out.TopTasks = append(out.TopTasks, ts)
|
||||
}
|
||||
return out, trows.Err()
|
||||
}
|
||||
@@ -0,0 +1,287 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"math"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestComputeLLMContextSharesDeterministic(t *testing.T) {
|
||||
// Call 1: 1000 bytes (400 sys_static, 100 sys_dyn, 200 tools, 200 user [150 tool_result], 100 asst), 1000 input tokens.
|
||||
// Call 2: 2000 bytes (200 sys_static, 0 sys_dyn, 200 tools, 1400 user [1200 tool_result], 200 asst), 3000 input tokens.
|
||||
// Call 3: zero tokens / zero bytes -> ignored safely (no NaN).
|
||||
rows := []LLMContext{
|
||||
{
|
||||
SystemStaticBytes: 400, SystemDynamicBytes: 100, ToolsBytes: 200,
|
||||
MessagesUserBytes: 200, MessagesAssistantBytes: 100, ToolResultBytes: 150,
|
||||
InputTokens: 1000,
|
||||
},
|
||||
{
|
||||
SystemStaticBytes: 200, SystemDynamicBytes: 0, ToolsBytes: 200,
|
||||
MessagesUserBytes: 1400, MessagesAssistantBytes: 200, ToolResultBytes: 1200,
|
||||
InputTokens: 3000,
|
||||
},
|
||||
{InputTokens: 0, SystemStaticBytes: 0},
|
||||
}
|
||||
|
||||
got := ComputeLLMContextShares(rows)
|
||||
// Exact hand-computed attributed tokens (total = 4000):
|
||||
// sys_static: 400 + 300 = 700 -> 0.175
|
||||
// sys_dynamic: 100 + 0 = 100 -> 0.025
|
||||
// tools: 200 + 300 = 500 -> 0.125
|
||||
// msg_user: 200 + 2100 = 2300 -> 0.575
|
||||
// msg_asst: 100 + 300 = 400 -> 0.100
|
||||
// msg_other: 0 = 0 -> 0.000
|
||||
// tool_result: 150 + 1800 = 1950 -> 0.4875
|
||||
assertNear := func(name string, v, want float64) {
|
||||
t.Helper()
|
||||
if math.IsNaN(v) || math.IsInf(v, 0) || math.Abs(v-want) > 1e-9 {
|
||||
t.Errorf("%s = %.12f, want %.12f", name, v, want)
|
||||
}
|
||||
}
|
||||
assertNear("SystemStatic", got.SystemStatic, 0.175)
|
||||
assertNear("SystemDynamic", got.SystemDynamic, 0.025)
|
||||
assertNear("Tools", got.Tools, 0.125)
|
||||
assertNear("MessagesUser", got.MessagesUser, 0.575)
|
||||
assertNear("MessagesAsst", got.MessagesAsst, 0.100)
|
||||
assertNear("MessagesOther", got.MessagesOther, 0.0)
|
||||
assertNear("ToolResultSubset", got.ToolResultSubset, 0.4875)
|
||||
|
||||
sum := got.SystemStatic + got.SystemDynamic + got.Tools + got.MessagesUser + got.MessagesAsst + got.MessagesOther
|
||||
assertNear("partition sum", sum, 1.0)
|
||||
|
||||
// Empty input produces zero struct without NaN.
|
||||
zero := ComputeLLMContextShares(nil)
|
||||
if zero != (LLMContextSectionShares{}) {
|
||||
t.Fatalf("empty shares = %+v, want zero", zero)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLLMContextMigration0009AndSummaryAggregation(t *testing.T) {
|
||||
dsn := scratchDSN(t, "ctx_mig_0009")
|
||||
|
||||
// 1. Open database with migrations 0001..0008 only (pre-0009 database).
|
||||
pre0009 := fstest.MapFS{}
|
||||
entries, err := fs.ReadDir(migrationsFS(), ".")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, e := range entries {
|
||||
if e.IsDir() || e.Name() >= "0009_" {
|
||||
continue
|
||||
}
|
||||
data, err := fs.ReadFile(migrationsFS(), e.Name())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pre0009[e.Name()] = &fstest.MapFile{Data: data}
|
||||
}
|
||||
d0, err := openWithMigrations(dsn, pre0009)
|
||||
if err != nil {
|
||||
t.Fatalf("open pre-0009: %v", err)
|
||||
}
|
||||
var existsBefore bool
|
||||
if err := d0.QueryRow(`SELECT to_regclass('llm_context') IS NOT NULL`).Scan(&existsBefore); err != nil || existsBefore {
|
||||
d0.Close()
|
||||
t.Fatalf("llm_context should not exist before 0009: exists=%v err=%v", existsBefore, err)
|
||||
}
|
||||
d0.Close()
|
||||
|
||||
// 2. Re-open with full embedded migrations: 0009 must apply cleanly.
|
||||
d, err := Open(dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("Open with 0009: %v", err)
|
||||
}
|
||||
defer d.Close()
|
||||
|
||||
// 3. Seed deterministic rows across two workers and two tasks.
|
||||
seed := []LLMContext{
|
||||
{
|
||||
TaskID: "t-heavy", Worker: "worker", Model: "claude-sonnet", Profile: "main",
|
||||
ToolCount: 15, MessageCount: 4,
|
||||
SystemStaticBytes: 400, SystemDynamicBytes: 100, ToolsBytes: 200,
|
||||
MessagesUserBytes: 200, MessagesAssistantBytes: 100, ToolResultBytes: 150,
|
||||
LastMessageBytes: 120,
|
||||
InputTokens: 1000, OutputTokens: 200, CacheRead: 600, CacheWrite: 100, Status: "ok",
|
||||
},
|
||||
{
|
||||
TaskID: "t-heavy", Worker: "worker", Model: "claude-sonnet", Profile: "main",
|
||||
ToolCount: 15, MessageCount: 12,
|
||||
SystemStaticBytes: 200, SystemDynamicBytes: 0, ToolsBytes: 200,
|
||||
MessagesUserBytes: 1400, MessagesAssistantBytes: 200, ToolResultBytes: 1200,
|
||||
LastMessageBytes: 800,
|
||||
InputTokens: 3000, OutputTokens: 300, CacheRead: 1800, CacheWrite: 200, Status: "ok",
|
||||
},
|
||||
{
|
||||
TaskID: "t-light", Worker: "planner", Model: "claude-sonnet", Profile: "main",
|
||||
ToolCount: 10, MessageCount: 2,
|
||||
SystemStaticBytes: 600, SystemDynamicBytes: 0, ToolsBytes: 200,
|
||||
MessagesUserBytes: 200, MessagesAssistantBytes: 0, ToolResultBytes: 0,
|
||||
LastMessageBytes: 200,
|
||||
InputTokens: 500, OutputTokens: 100, CacheRead: 250, CacheWrite: 50, Status: "ok",
|
||||
},
|
||||
}
|
||||
for i := range seed {
|
||||
if err := d.InsertLLMContext(&seed[i]); err != nil {
|
||||
t.Fatalf("InsertLLMContext[%d]: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
summary, err := d.LLMContextSummary(LLMContextQuery{Days: 7, TopTasks: 10})
|
||||
if err != nil {
|
||||
t.Fatalf("LLMContextSummary: %v", err)
|
||||
}
|
||||
if len(summary.ByWorkerModel) != 2 {
|
||||
t.Fatalf("ByWorkerModel = %d, want 2: %+v", len(summary.ByWorkerModel), summary.ByWorkerModel)
|
||||
}
|
||||
|
||||
// First group is worker/claude-sonnet (total input = 4000).
|
||||
w := summary.ByWorkerModel[0]
|
||||
wantShares := ComputeLLMContextShares(seed[:2])
|
||||
near := func(name string, got, want float64) {
|
||||
t.Helper()
|
||||
if math.Abs(got-want) > 1e-9 {
|
||||
t.Errorf("%s = %.12f, want %.12f", name, got, want)
|
||||
}
|
||||
}
|
||||
if w.Worker != "worker" || w.Model != "claude-sonnet" || w.Calls != 2 ||
|
||||
w.InputTokensTotal != 4000 || w.OutputTokensTotal != 500 ||
|
||||
w.CacheReadTotal != 2400 || w.CacheWriteTotal != 300 || w.InputMax != 3000 {
|
||||
t.Fatalf("worker group = %+v", w)
|
||||
}
|
||||
near("CacheReadRatio", w.CacheReadRatio, 2400.0/4000.0)
|
||||
near("InputP50", w.InputP50, 2000.0)
|
||||
near("InputP90", w.InputP90, 2800.0)
|
||||
near("InputP99", w.InputP99, 2980.0)
|
||||
near("Shares.SystemStatic", w.Shares.SystemStatic, wantShares.SystemStatic)
|
||||
near("Shares.SystemDynamic", w.Shares.SystemDynamic, wantShares.SystemDynamic)
|
||||
near("Shares.Tools", w.Shares.Tools, wantShares.Tools)
|
||||
near("Shares.MessagesUser", w.Shares.MessagesUser, wantShares.MessagesUser)
|
||||
near("Shares.MessagesAsst", w.Shares.MessagesAsst, wantShares.MessagesAsst)
|
||||
near("Shares.ToolResultSubset", w.Shares.ToolResultSubset, wantShares.ToolResultSubset)
|
||||
|
||||
// TopTasks ordered by InputTokensTotal DESC: t-heavy (4000) then t-light (500).
|
||||
if len(summary.TopTasks) != 2 || summary.TopTasks[0].TaskID != "t-heavy" || summary.TopTasks[1].TaskID != "t-light" {
|
||||
t.Fatalf("TopTasks = %+v", summary.TopTasks)
|
||||
}
|
||||
|
||||
// Task filter narrows to t-light only.
|
||||
filtered, err := d.LLMContextSummary(LLMContextQuery{Days: 7, TaskID: "t-light", TopTasks: 5})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(filtered.ByWorkerModel) != 1 || filtered.ByWorkerModel[0].Worker != "planner" || len(filtered.TopTasks) != 1 {
|
||||
t.Fatalf("filtered summary = %+v", filtered)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLLMContextRepresentativeFixtureReport runs a realistic multi-turn ARTEX
|
||||
// task fixture (planner rounds + deep worker intent execution + main-agent
|
||||
// follow-up) through InsertLLMContext and LLMContextSummary, verifying the
|
||||
// aggregation pipeline end-to-end and logging reproducible Phase A numbers.
|
||||
func TestLLMContextRepresentativeFixtureReport(t *testing.T) {
|
||||
d, err := Open(testDSN(t))
|
||||
if err != nil {
|
||||
SkipPGUnavailable(t, err)
|
||||
}
|
||||
defer d.Close()
|
||||
|
||||
taskID := fmt.Sprintf("fixture-p35a-%d", time.Now().UnixNano())
|
||||
t.Cleanup(func() { _, _ = d.Exec(`DELETE FROM llm_context WHERE task_id=$1`, taskID) })
|
||||
|
||||
// Representative byte sizes modeled from ARTEX's actual plannerSystem (~12 KiB),
|
||||
// workerSystem (~11 KiB), tool schemas (~8-10 KiB for 12-16 tools), and real
|
||||
// multi-turn exploration trajectories (token estimate ~3.8 bytes/token).
|
||||
var fixture []LLMContext
|
||||
|
||||
// 5 planner rounds: ~12.5 KiB static system (includes {{.Now}} so cross-round cache misses
|
||||
// on first turn of each round), ~9.2 KiB tools (14 tools), situational graph overview growing
|
||||
// from 4 KiB to 14 KiB across rounds, 2 turns per round.
|
||||
for round := range 5 {
|
||||
overviewBytes := int64(4000 + round*2500)
|
||||
// Turn 1 of round: fresh session, cache miss (cache_write ~system+tools, cache_read 0)
|
||||
sysBytes := int64(12500)
|
||||
toolsBytes := int64(9200)
|
||||
in1 := int((sysBytes + toolsBytes + overviewBytes) / 4)
|
||||
fixture = append(fixture, LLMContext{
|
||||
TaskID: taskID, Worker: "planner", Model: "claude-sonnet-4", Profile: "fixture",
|
||||
ToolCount: 14, MessageCount: 1,
|
||||
SystemStaticBytes: sysBytes, ToolsBytes: toolsBytes,
|
||||
MessagesUserBytes: overviewBytes, LastMessageBytes: overviewBytes,
|
||||
InputTokens: in1, OutputTokens: 450, CacheRead: 0, CacheWrite: int((sysBytes + toolsBytes) / 4), Status: "ok",
|
||||
})
|
||||
// Turn 2 of round: after add_intent tool_result, prefix hits cache
|
||||
asstBytes := int64(1400)
|
||||
toolResBytes := int64(450)
|
||||
in2 := int((sysBytes + toolsBytes + overviewBytes + asstBytes + toolResBytes) / 4)
|
||||
fixture = append(fixture, LLMContext{
|
||||
TaskID: taskID, Worker: "planner", Model: "claude-sonnet-4", Profile: "fixture",
|
||||
ToolCount: 14, MessageCount: 3,
|
||||
SystemStaticBytes: sysBytes, ToolsBytes: toolsBytes,
|
||||
MessagesUserBytes: overviewBytes + toolResBytes, MessagesAssistantBytes: asstBytes,
|
||||
ToolResultBytes: toolResBytes, LastMessageBytes: toolResBytes,
|
||||
InputTokens: in2, OutputTokens: 320, CacheRead: in1, CacheWrite: 0, Status: "ok",
|
||||
})
|
||||
}
|
||||
|
||||
// 2 worker intents x 10 turns each: ~11 KiB system (embeds per-intent runDir + {{.Now}}),
|
||||
// ~10.5 KiB tools (16 tools), growing Bash/WebFetch tool_result outputs (avg ~5 KiB/turn).
|
||||
for range 2 {
|
||||
sysBytes := int64(11000)
|
||||
toolsBytes := int64(10500)
|
||||
userBytes := int64(6500) // launch overview + claimed intent
|
||||
asstBytes := int64(0)
|
||||
toolResTotal := int64(0)
|
||||
prevInput := 0
|
||||
for turn := range 10 {
|
||||
lastBytes := int64(6500)
|
||||
if turn > 0 {
|
||||
trChunk := int64(3500 + (turn%3)*2500) // realistic nmap/curl/ffuf tool outputs
|
||||
asstChunk := int64(900)
|
||||
userBytes += trChunk
|
||||
toolResTotal += trChunk
|
||||
asstBytes += asstChunk
|
||||
lastBytes = trChunk
|
||||
}
|
||||
totalB := sysBytes + toolsBytes + userBytes + asstBytes
|
||||
inTok := int(totalB / 4)
|
||||
cacheRead := prevInput
|
||||
cacheWrite := 0
|
||||
if turn == 0 {
|
||||
cacheRead = 0
|
||||
cacheWrite = int((sysBytes + toolsBytes) / 4)
|
||||
}
|
||||
fixture = append(fixture, LLMContext{
|
||||
TaskID: taskID, Worker: "worker", Model: "claude-sonnet-4", Profile: "fixture",
|
||||
ToolCount: 16, MessageCount: 1 + turn*2,
|
||||
SystemStaticBytes: sysBytes, ToolsBytes: toolsBytes,
|
||||
MessagesUserBytes: userBytes, MessagesAssistantBytes: asstBytes,
|
||||
ToolResultBytes: toolResTotal, LastMessageBytes: lastBytes,
|
||||
InputTokens: inTok, OutputTokens: 380, CacheRead: cacheRead, CacheWrite: cacheWrite, Status: "ok",
|
||||
})
|
||||
prevInput = inTok
|
||||
}
|
||||
}
|
||||
|
||||
for i := range fixture {
|
||||
if err := d.InsertLLMContext(&fixture[i]); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
summary, err := d.LLMContextSummary(LLMContextQuery{Days: 1, TaskID: taskID, TopTasks: 5})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(summary.ByWorkerModel) != 2 || len(summary.TopTasks) != 1 {
|
||||
t.Fatalf("unexpected fixture summary shape: %+v", summary)
|
||||
}
|
||||
for _, g := range summary.ByWorkerModel {
|
||||
t.Logf("FIXTURE [%s/%s] calls=%d in_total=%d p50=%.0f p90=%.0f max=%d cache_read_ratio=%.3f | sys_static=%.1f%% sys_dyn=%.1f%% tools=%.1f%% user=%.1f%% (tool_result=%.1f%%) asst=%.1f%%",
|
||||
g.Worker, g.Model, g.Calls, g.InputTokensTotal, g.InputP50, g.InputP90, g.InputMax, g.CacheReadRatio,
|
||||
g.Shares.SystemStatic*100, g.Shares.SystemDynamic*100, g.Shares.Tools*100,
|
||||
g.Shares.MessagesUser*100, g.Shares.ToolResultSubset*100, g.Shares.MessagesAsst*100)
|
||||
}
|
||||
}
|
||||
+3
-3
@@ -123,7 +123,7 @@ func readApplied(t *testing.T, d *DB) []appliedMigration {
|
||||
// embedded migration is recorded with the SHA-256 of its file.
|
||||
func assertEmbeddedApplied(t *testing.T, d *DB) []appliedMigration {
|
||||
t.Helper()
|
||||
for _, table := range []string{"llm_records", "llm_usage"} {
|
||||
for _, table := range []string{"llm_records", "llm_usage", "llm_context"} {
|
||||
var exists bool
|
||||
if err := d.QueryRow(`SELECT to_regclass($1) IS NOT NULL`, table).Scan(&exists); err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -138,9 +138,9 @@ func assertEmbeddedApplied(t *testing.T, d *DB) []appliedMigration {
|
||||
name string
|
||||
}{{1, llmRecordsMigration}, {2, llmUsageMigration}, {3, "0003_task_scope_icp_index.sql"}, {4, "0004_llm_usage_ts_index.sql"},
|
||||
{5, "0005_traffic_rewrite_rules.sql"}, {6, "0006_activity_outbox.sql"}, {7, "0007_finding_report_state.sql"},
|
||||
{8, "0008_traffic_evidence_rewrite_flag.sql"}}
|
||||
{8, "0008_traffic_evidence_rewrite_flag.sql"}, {9, "0009_llm_context.sql"}}
|
||||
if len(applied) != len(want) {
|
||||
t.Fatalf("schema_migrations = %+v, want versions 1 to 8", applied)
|
||||
t.Fatalf("schema_migrations = %+v, want versions 1 to 9", applied)
|
||||
}
|
||||
for i, w := range want {
|
||||
content, err := fs.ReadFile(migrationsFS(), w.name)
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
-- Opt-in per-call LLM context size telemetry (ARTEX_CONTEXT_TELEMETRY=1).
|
||||
-- Records only byte sizes, counts, token usage, and metadata — never any
|
||||
-- prompt, tool, or message content.
|
||||
CREATE TABLE IF NOT EXISTS llm_context (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
task_id TEXT,
|
||||
worker TEXT,
|
||||
model TEXT,
|
||||
profile TEXT,
|
||||
tool_count INTEGER NOT NULL DEFAULT 0,
|
||||
message_count INTEGER NOT NULL DEFAULT 0,
|
||||
system_static_bytes BIGINT NOT NULL DEFAULT 0,
|
||||
system_dynamic_bytes BIGINT NOT NULL DEFAULT 0,
|
||||
tools_bytes BIGINT NOT NULL DEFAULT 0,
|
||||
messages_user_bytes BIGINT NOT NULL DEFAULT 0,
|
||||
messages_assistant_bytes BIGINT NOT NULL DEFAULT 0,
|
||||
messages_other_bytes BIGINT NOT NULL DEFAULT 0,
|
||||
tool_result_bytes BIGINT NOT NULL DEFAULT 0,
|
||||
last_message_bytes BIGINT NOT NULL DEFAULT 0,
|
||||
input_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
output_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
cache_read INTEGER NOT NULL DEFAULT 0,
|
||||
cache_write INTEGER NOT NULL DEFAULT 0,
|
||||
status TEXT NOT NULL DEFAULT 'ok'
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_llm_context_ts ON llm_context(ts);
|
||||
CREATE INDEX IF NOT EXISTS idx_llm_context_task ON llm_context(task_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_llm_context_worker_model ON llm_context(worker, model);
|
||||
@@ -0,0 +1,148 @@
|
||||
# LLM Context Size Telemetry (P3.5 Phase A)
|
||||
|
||||
`ARTEX_CONTEXT_TELEMETRY=1` enables opt-in, size-only per-call telemetry for every `llmrec` `Stream` and `Complete` call. It is **disabled by default**: when off, no request sizing runs and no rows are written to PostgreSQL.
|
||||
|
||||
This phase measures context composition without changing any prompt, compaction policy, or model behavior, so Phases B (`ARTEX_BOUNDED_CONTEXT`) and C (`ARTEX_CACHE_PREFIX` / `ARTEX_COMPACTION_MEMO`) can be evaluated against real numbers first.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
| Environment variable | Values | Default |
|
||||
|---|---|---|
|
||||
| `ARTEX_CONTEXT_TELEMETRY` | `1`, `true`, `on`, `yes` (case-insensitive) | off |
|
||||
|
||||
Telemetry runs independently of `llm_record` (`recordBodies`): enabling `ARTEX_CONTEXT_TELEMETRY=1` works even when full request/response body recording is disabled, and stores **no prompt, tool, or message content** — only byte lengths, counts, token usage, and metadata.
|
||||
|
||||
### Stopping, retention, and write cost
|
||||
|
||||
When `ARTEX_CONTEXT_TELEMETRY=1` is active, each LLM call incurs one lightweight `INSERT` into `llm_context` (~200 bytes/row plus index entries, no bodies), so the table grows by one row per completion call. Unset `ARTEX_CONTEXT_TELEMETRY` (and restart) once the measurement campaign is finished to stop all writes and sizing work, or purge old rows periodically:
|
||||
|
||||
```sql
|
||||
DELETE FROM llm_context WHERE ts < now() - interval '30 days';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Measurement Semantics (`llmrec.MeasureContext`)
|
||||
|
||||
Every enabled `Stream` and `Complete` call measures the assembled `llm.CompletionRequest` right before it is handed to the inner provider, and persists one row to `llm_context` (migration `0009_llm_context.sql`) on completion or error:
|
||||
|
||||
- **`system_static_bytes` & `system_dynamic_bytes`**:
|
||||
- `norma`'s `CompletionRequest.System` is a slice of segments that `llm.joinSystem` trims (`strings.TrimSpace`), drops when empty, and joins with blank lines (`"\n\n"`). `MeasureContext` applies the exact same trim, drop-empty, and `"\n\n"` join rules per slice.
|
||||
- When `DynamicBoundary > 0`, `System` is split at `b = min(DynamicBoundary, len(System))`: `System[:b]` is measured into `system_static_bytes` and `System[b:]` into `system_dynamic_bytes`.
|
||||
- When `DynamicBoundary <= 0`, there is no provider-visible cache split, so the entire `System` slice is measured as one block in `system_static_bytes` and `system_dynamic_bytes` is `0`.
|
||||
- **Why this choice**: `agent/deferred.go` returns `boundary = 0` when no global deferred-MCP block exists and `boundary = len(system)` when one does. Placing the single unsplit block in `system_static_bytes` ensures `system_static_bytes + system_dynamic_bytes` always equals total system bytes and `system_dynamic_bytes > 0` strictly signals a dynamic suffix placed after a cache breakpoint.
|
||||
- **`tool_count` & `tools_bytes`**: `len(req.Tools)` and `len(json.Marshal(req.Tools))` (`0` when no tools). Go's `encoding/json` sorts map keys deterministically.
|
||||
- **`message_count`, `messages_user_bytes`, `messages_assistant_bytes`, `messages_other_bytes`, `last_message_bytes`**: each `llm.Message` is measured by `len(json.Marshal(msg))` and accumulated by role (`user`, `assistant`, or other); `last_message_bytes` is the serialized byte size of the final message.
|
||||
- **`tool_result_bytes`**: serialized byte length of `tool_result` content blocks inside messages. Reported as an **informational subset** of `messages_user_bytes` (not added to the partition denominator, avoiding double counting).
|
||||
|
||||
### Pro-rata token attribution
|
||||
|
||||
Rather than shipping per-provider tokenizers, each section's share of real `input_tokens` is computed per call by byte proportion:
|
||||
|
||||
```text
|
||||
total_bytes(call) = system_static_bytes + system_dynamic_bytes + tools_bytes
|
||||
+ messages_user_bytes + messages_assistant_bytes + messages_other_bytes
|
||||
|
||||
share(section) = SUM(input_tokens * (section_bytes / total_bytes))
|
||||
/ SUM(input_tokens) -- over calls with total_bytes > 0 and input_tokens > 0
|
||||
```
|
||||
|
||||
The six partition shares sum to `1.0`. `cache_read_ratio = SUM(cache_read) / NULLIF(SUM(input_tokens), 0)` (in ARTEX/Norma, `input_tokens` already includes cached tokens).
|
||||
|
||||
---
|
||||
|
||||
## Querying & Aggregation
|
||||
|
||||
### HTTP endpoint (admin JWT required, no UI in Phase A)
|
||||
|
||||
```text
|
||||
GET /api/llm/context/summary?days=30&top=20&task=<optional-task-id>
|
||||
```
|
||||
|
||||
Returns `enabled`, `by_worker_model` (calls, totals, `cache_read_ratio`, `input_p50`/`p90`/`p99`/`max`, avg tool/message/last-message size, and pro-rata `shares`), and `top_tasks` ordered by total `input_tokens`.
|
||||
|
||||
### Standalone SQL query (`psql`)
|
||||
|
||||
```sql
|
||||
WITH base AS (
|
||||
SELECT *,
|
||||
(system_static_bytes + system_dynamic_bytes + tools_bytes
|
||||
+ messages_user_bytes + messages_assistant_bytes + messages_other_bytes) AS total_bytes
|
||||
FROM llm_context
|
||||
WHERE ts >= now() - interval '30 days'
|
||||
)
|
||||
SELECT COALESCE(NULLIF(worker,''), '(unknown)') AS worker,
|
||||
COALESCE(NULLIF(model,''), '(unknown)') AS model,
|
||||
COUNT(*) AS calls,
|
||||
SUM(input_tokens) AS input_tokens,
|
||||
ROUND((SUM(cache_read)::numeric / NULLIF(SUM(input_tokens), 0))::numeric, 3) AS cache_read_ratio,
|
||||
PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY input_tokens) AS p50_input,
|
||||
PERCENTILE_CONT(0.90) WITHIN GROUP (ORDER BY input_tokens) AS p90_input,
|
||||
PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY input_tokens) AS p99_input,
|
||||
ROUND((SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens * (system_static_bytes::double precision / total_bytes) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0))::numeric, 3) AS share_sys_static,
|
||||
ROUND((SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens * (system_dynamic_bytes::double precision / total_bytes) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0))::numeric, 3) AS share_sys_dynamic,
|
||||
ROUND((SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens * (tools_bytes::double precision / total_bytes) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0))::numeric, 3) AS share_tools,
|
||||
ROUND((SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens * (messages_user_bytes::double precision / total_bytes) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0))::numeric, 3) AS share_user,
|
||||
ROUND((SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens * (tool_result_bytes::double precision / total_bytes) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0))::numeric, 3) AS share_tool_result_subset,
|
||||
ROUND((SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens * (messages_assistant_bytes::double precision / total_bytes) ELSE 0 END)
|
||||
/ NULLIF(SUM(CASE WHEN total_bytes > 0 AND input_tokens > 0 THEN input_tokens::double precision ELSE 0 END), 0))::numeric, 3) AS share_assistant
|
||||
FROM base
|
||||
GROUP BY 1, 2
|
||||
ORDER BY input_tokens DESC;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase A Baseline Report & Recommendation on Phases B and C
|
||||
|
||||
> **Measurement scope note**: No live production LLM run was executed in this local verification environment. The figures below come from the reproducible representative fixture in `db/llm_context_test.go` (`TestLLMContextRepresentativeFixtureReport`), parameterized from ARTEX's actual `plannerSystem` (~12.5 KiB), `workerSystem` (~11 KiB), tool schema sets (14–16 tools, ~9–10.5 KiB), 5 planner rounds (2 turns each), and 2 worker intents (10 turns each with 3.5–8.5 KiB Bash/tool outputs), combined with a static audit of `agent/planner.go`, `agent/worker.go`, `agent/prompt.go`, and `agent/deferred.go`. Operators should capture 3–5 real tasks with `ARTEX_CONTEXT_TELEMETRY=1` before flipping Phases B or C on by default.
|
||||
|
||||
### Representative Fixture Breakdown
|
||||
|
||||
| Lane / Model | Calls | Total Input | p50 / p90 / Max | Cache Read Ratio | `sys_static` | `sys_dynamic` | `tools` | `user` (`tool_result` subset) | `assistant` |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| `worker` / `claude-sonnet-4` | 20 | 299,000 | 15,075 / 21,535 / 22,525 | 84.9% | 18.4% | 0.0% | 17.6% | 57.3% (`46.4%`) | 6.8% |
|
||||
| `planner` / `claude-sonnet-4` | 10 | 79,060 | 7,906 / 8,971 / 9,387 | 48.5% | 39.5% | 0.0% | 29.1% | 29.2% (`0.7%`) | 2.2% |
|
||||
|
||||
### Explicit Recommendations
|
||||
|
||||
#### Phase B — Bounded Context (`ARTEX_BOUNDED_CONTEXT`)
|
||||
- **Recommendation: Proceed (opt-in first), highest ROI on deep `worker` runs.**
|
||||
- **Expected gain**: In the representative worker trajectory, `messages_user` accounts for **57.3%** of total input tokens, with **`tool_result` outputs alone representing 46.4%** (and growing quadratically across turns on long Bash/scanner sessions). Bounding/compacting old tool outputs on workers (keeping recent turns verbatim and summarizing/truncating stale `tool_result` payloads) targets the largest single token sink (realistic 25–45% input-token reduction on 10–30 turn workers, and lower p90/p99 latency).
|
||||
- **Risk**: Aggressive trimming of earlier tool outputs can drop credentials, endpoints, or exact error lines needed for exploit chaining or evidence binding, and mutating earlier messages inside a live multi-turn session **invalidates the prompt cache prefix** (turning cheap `cache_read` turns into uncached re-prefills if triggered too frequently). Bound only at explicit compaction boundaries or spill large `tool_result` bodies at insertion time before they enter the cached prefix.
|
||||
|
||||
#### Phase C — Cache Prefix Stabilization (`ARTEX_CACHE_PREFIX` / `ARTEX_COMPACTION_MEMO`)
|
||||
- **Recommendation: Proceed with a surgical `worker` prefix split (`ARTEX_CACHE_PREFIX`); verify `planner` cross-round cache behavior and `ARTEX_COMPACTION_MEMO` with live `ARTEX_CONTEXT_TELEMETRY=1` data before changing them.**
|
||||
- **Actual system-prompt volatility by lane (built-in prompts)**:
|
||||
|
||||
`system_dynamic` is **0.0%** across all lanes today because `agent/deferred.go` sets `DynamicBoundary = 0` (no global deferred MCP) or `len(system)` (everything before the boundary). Auditing the built-in templates and assembly in `agent/worker.go` and `agent/planner.go` shows what actually varies:
|
||||
|
||||
| Lane | Stable per task | Volatile per intent / round |
|
||||
|---|---|---|
|
||||
| `worker` | Base `workerDefaultTmpl` (`ProxyAddr`, `DataDir`), `untrustedContentTail`, `workerTrafficBlock`, `noiseProfileBlock`, `frameLessonsBlock` (task-level), `constraintBlock` (task-level, stable unless operator edits constraints), `def.FindingGuidance`, `<available-deferred-tools>` | `workerArtifactSpec(runDir)` (`tasks/<id>/i<intentID>`), `renderIntentTask(intent)`, intent target-assets JSON block, closing launch instruction |
|
||||
| `planner` | `plannerSystem` (`goal`, `dataDir`, `workDir`/`taskDir`, `untrustedDataRules`, `artifactSpec(taskDir)`, `noiseProfileBlock`), `frameLessonsBlock`, `def.FindingGuidance`, `<available-deferred-tools>` | `constraintBlock(ts)` is recomputed each round (stable unless operator changes constraints; situational graph overview already lives in the user turn) |
|
||||
| Custom prompts only | — | `{{.Now}}` (`agent/prompt.go:46`, 1-second resolution). Built-in templates do **not** reference `{{.Now}}` (Go's `text/template` ignores the unused `Now` field); it only affects operator-customized prompts that explicitly use `{{.Now}}`. |
|
||||
|
||||
- **Expected gain**:
|
||||
- **`worker` turn-1 cache reuse (real, code-backed)**: Today `workerArtifactSpec(runDir)` is concatenated into the single `sysBody` before `noiseProfileBlock`, lessons, constraints, `renderIntentTask(intent)`, and the target-assets block, so every sibling worker spawn busts the prompt cache from `runDir` onward. The minimal `ARTEX_CACHE_PREFIX=1` split shipped in Phase C places the prefix before `workerArtifactSpec(runDir)` (`body` + `untrustedContentTail` + `workerTrafficBlock`) **before** `DynamicBoundary = 1` (`system_static`) and leaves everything from `workerArtifactSpec(runDir)` onward (`workerArtifactSpec(runDir)` + `noiseProfileBlock` + lessons + constraints + `renderIntentTask(intent)` + target assets JSON + closing launch line + `def.FindingGuidance` + `<available-deferred-tools>`) **after** `DynamicBoundary = 1` (`system_dynamic`) in its exact original order. This keeps the visible joined prompt byte-identical and keeps the intent protected from compaction inside `System` while allowing sibling workers in the same task to share the cached `system_static + tools` prefix on turn 1 (in the fixture, `system + tools` is 36.0% of aggregate worker input and ~76.8% of turn-1 input, 21.5 KiB / 28.0 KiB, though the fixture does not isolate the per-intent suffix; measure the exact shareable prefix live). Moving the remaining task-stable blocks (`noiseProfileBlock`, lessons, constraints, `def.FindingGuidance`, `<available-deferred-tools>`) before the boundary would require reordering sections relative to `workerArtifactSpec(runDir)` and is left as a potential future optimization outside this PR.
|
||||
- **`planner` cross-round cache ratio (unverified hypothesis)**: The fixture above assumed a turn-1 cache miss on every planner round (yielding 48.5%), which is **not** demonstrated by the built-in code because `plannerSystem` is stable per task. Real cross-round misses could still happen from provider cache TTL expiry between rounds, constraint edits, or tool-RAG tool-list churn — treat any planner cache gain as a hypothesis to measure with `ARTEX_CONTEXT_TELEMETRY=1` on real tasks rather than a proven savings.
|
||||
- **Risk**: Low for the `worker` `ARTEX_CACHE_PREFIX` split (pure boundary split using `norma`'s existing `DynamicBoundary` contract, with zero text reordering; custom prompts using `{{.Now}}` stay an explicit edge case). Medium for `ARTEX_COMPACTION_MEMO` (injecting compaction memos alters model reasoning continuity and only pays off on tasks that repeatedly trigger compaction — verify with live `ARTEX_CONTEXT_TELEMETRY=1` data first).
|
||||
|
||||
---
|
||||
|
||||
## Phase C: Worker Cache Prefix Split (`ARTEX_CACHE_PREFIX=1`)
|
||||
|
||||
`ARTEX_CACHE_PREFIX=1` (accepts `1`, `true`, `on`, `yes`, case-insensitive; **off by default**) enables a minimal, zero-reordering system-prompt split for the **`worker`** lane only so sibling workers in the same task share a cacheable static prefix without changing the text visible to the model:
|
||||
|
||||
- **Static segment (`System[0]`, before `DynamicBoundary = 1`)**: `body + untrustedContentTail + workerTrafficBlock`.
|
||||
- **Dynamic segment (`System[1]`, after `DynamicBoundary = 1`)**: `workerArtifactSpec(runDir) + noiseProfileBlock + lessons + constraints + renderIntentTask(intent) + target assets + launch instruction + FindingGuidance + <available-deferred-tools>` (in exact original order). The leading `"\n\n"` from `workerArtifactSpec` is absorbed because providers join segments with `"\n\n"`, keeping `strings.Join(System, "\n\n")` byte-identical to the legacy unsplit prompt.
|
||||
- **Unchanged**: When `ARTEX_CACHE_PREFIX` is unset/false, worker segments and `DynamicBoundary` are byte-identical to the legacy layout. `planner`, `mainagent`, `chat`, and `goals` are completely unchanged regardless of the flag.
|
||||
- **Custom prompts caveat**: The split covers only the built-in suffix starting at `workerArtifactSpec(runDir)`. If an operator customizes the `worker` prompt in the database to include `{{.Now}}` (1-second resolution), that timestamp remains inside `System[0]` and will keep the static prefix volatile across spawns.
|
||||
- **Measuring live impact**: Enable both `ARTEX_CONTEXT_TELEMETRY=1` and `ARTEX_CACHE_PREFIX=1` on real tasks and compare `system_static` / `system_dynamic` shares and `cache_read_ratio` via `GET /api/llm/context/summary`.
|
||||
@@ -0,0 +1,169 @@
|
||||
package llmrec
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"log"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/norma/llm"
|
||||
)
|
||||
|
||||
// ContextTelemetryEnv is the opt-in environment flag enabling per-call context
|
||||
// size telemetry into llm_context. Off by default: when unset/false, no sizing
|
||||
// work runs and no rows are written.
|
||||
const ContextTelemetryEnv = "ARTEX_CONTEXT_TELEMETRY"
|
||||
|
||||
// ContextTelemetryEnabled reports whether ARTEX_CONTEXT_TELEMETRY is enabled
|
||||
// (1, true, on, yes; case-insensitive). Default is false.
|
||||
func ContextTelemetryEnabled() bool {
|
||||
switch strings.ToLower(strings.TrimSpace(os.Getenv(ContextTelemetryEnv))) {
|
||||
case "1", "true", "on", "yes":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// ContextMetrics holds size-only measurements of an assembled CompletionRequest
|
||||
// as it leaves for the provider. Never retains any prompt, tool, or message
|
||||
// content.
|
||||
type ContextMetrics struct {
|
||||
ToolCount int
|
||||
MessageCount int
|
||||
SystemStaticBytes int64
|
||||
SystemDynamicBytes int64
|
||||
ToolsBytes int64
|
||||
MessagesUserBytes int64
|
||||
MessagesAssistantBytes int64
|
||||
MessagesOtherBytes int64
|
||||
ToolResultBytes int64
|
||||
LastMessageBytes int64
|
||||
}
|
||||
|
||||
// MeasureContext computes deterministic byte sizes and counts for req without
|
||||
// retaining any content:
|
||||
//
|
||||
// - System: when DynamicBoundary > 0, req.System is split at
|
||||
// min(DynamicBoundary, len(req.System)) into SystemStaticBytes ([:b]) and
|
||||
// SystemDynamicBytes ([b:]). Within each slice, each segment is trimmed with
|
||||
// strings.TrimSpace, empty segments are dropped, and remaining segments are
|
||||
// joined with "\n\n", matching norma's llm.joinSystem. When
|
||||
// DynamicBoundary <= 0, the whole req.System is measured as a single static
|
||||
// block (SystemStaticBytes) and SystemDynamicBytes is 0.
|
||||
// - Tools: ToolCount = len(req.Tools); ToolsBytes = len(json.Marshal(req.Tools))
|
||||
// (0 when no tools). Go's encoding/json sorts map keys deterministically.
|
||||
// - Messages: MessageCount = len(req.Messages). Each message is measured by
|
||||
// len(json.Marshal(msg)) and summed into MessagesUserBytes (role "user"),
|
||||
// MessagesAssistantBytes (role "assistant"), or MessagesOtherBytes (any
|
||||
// other role). ToolResultBytes records the serialized size of tool_result
|
||||
// content blocks inside messages as an informational subset (not added to
|
||||
// the partition denominator). LastMessageBytes is the serialized byte size
|
||||
// of the final message (0 when no messages).
|
||||
func MeasureContext(req llm.CompletionRequest) ContextMetrics {
|
||||
var m ContextMetrics
|
||||
|
||||
// 1. System prompt bytes (split at DynamicBoundary when > 0, else single static block).
|
||||
if len(req.System) > 0 {
|
||||
if req.DynamicBoundary > 0 {
|
||||
b := req.DynamicBoundary
|
||||
if b > len(req.System) {
|
||||
b = len(req.System)
|
||||
}
|
||||
m.SystemStaticBytes = joinedSegmentBytes(req.System[:b])
|
||||
m.SystemDynamicBytes = joinedSegmentBytes(req.System[b:])
|
||||
} else {
|
||||
m.SystemStaticBytes = joinedSegmentBytes(req.System)
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Serialized tool schemas and count.
|
||||
m.ToolCount = len(req.Tools)
|
||||
if len(req.Tools) > 0 {
|
||||
if b, err := json.Marshal(req.Tools); err == nil {
|
||||
m.ToolsBytes = int64(len(b))
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Messages by role, tool_result subset, and last message size.
|
||||
m.MessageCount = len(req.Messages)
|
||||
for i, msg := range req.Messages {
|
||||
var msgBytes int64
|
||||
if b, err := json.Marshal(msg); err == nil {
|
||||
msgBytes = int64(len(b))
|
||||
}
|
||||
switch msg.Role {
|
||||
case llm.RoleUser:
|
||||
m.MessagesUserBytes += msgBytes
|
||||
case llm.RoleAssistant:
|
||||
m.MessagesAssistantBytes += msgBytes
|
||||
default:
|
||||
m.MessagesOtherBytes += msgBytes
|
||||
}
|
||||
if i == len(req.Messages)-1 {
|
||||
m.LastMessageBytes = msgBytes
|
||||
}
|
||||
for _, blk := range msg.Content {
|
||||
if blk.Type == llm.BlockToolResult {
|
||||
if b, err := json.Marshal(blk); err == nil {
|
||||
m.ToolResultBytes += int64(len(b))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return m
|
||||
}
|
||||
|
||||
// joinedSegmentBytes returns the byte length of segs after trimming whitespace
|
||||
// from each segment, dropping empty segments, and joining the remaining ones
|
||||
// with "\n\n" — matching norma's llm.joinSystem without allocating a joined
|
||||
// string.
|
||||
func joinedSegmentBytes(segs []string) int64 {
|
||||
var total int64
|
||||
nonEmpty := 0
|
||||
for _, s := range segs {
|
||||
s = strings.TrimSpace(s)
|
||||
if s == "" {
|
||||
continue
|
||||
}
|
||||
if nonEmpty > 0 {
|
||||
total += 2 // "\n\n"
|
||||
}
|
||||
total += int64(len(s))
|
||||
nonEmpty++
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// recordContext appends one row to llm_context when r.pg is configured.
|
||||
// Best-effort: logs and continues on error so telemetry never disrupts calls.
|
||||
func (r *Recorder) recordContext(taskID, worker string, m ContextMetrics, usage llm.Usage, status string) {
|
||||
if r.pg == nil {
|
||||
return
|
||||
}
|
||||
err := r.pg.InsertLLMContext(&db.LLMContext{
|
||||
TaskID: taskID,
|
||||
Worker: worker,
|
||||
Model: r.model,
|
||||
Profile: r.prof,
|
||||
ToolCount: m.ToolCount,
|
||||
MessageCount: m.MessageCount,
|
||||
SystemStaticBytes: m.SystemStaticBytes,
|
||||
SystemDynamicBytes: m.SystemDynamicBytes,
|
||||
ToolsBytes: m.ToolsBytes,
|
||||
MessagesUserBytes: m.MessagesUserBytes,
|
||||
MessagesAssistantBytes: m.MessagesAssistantBytes,
|
||||
MessagesOtherBytes: m.MessagesOtherBytes,
|
||||
ToolResultBytes: m.ToolResultBytes,
|
||||
LastMessageBytes: m.LastMessageBytes,
|
||||
InputTokens: usage.InputTokens,
|
||||
OutputTokens: usage.OutputTokens,
|
||||
CacheRead: usage.CacheReadTokens,
|
||||
CacheWrite: usage.CacheWriteTokens,
|
||||
Status: status,
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("[llmcontext] insert: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,371 @@
|
||||
package llmrec
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"iter"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/norma/llm"
|
||||
"github.com/Autumn-27/norma/transcript"
|
||||
)
|
||||
|
||||
type syntheticTelemetryProvider struct {
|
||||
usage llm.Usage
|
||||
streamErr error
|
||||
completeEr error
|
||||
}
|
||||
|
||||
func (p syntheticTelemetryProvider) Stream(context.Context, llm.CompletionRequest) iter.Seq2[llm.StreamEvent, error] {
|
||||
return func(yield func(llm.StreamEvent, error) bool) {
|
||||
if !yield(llm.StreamEvent{Type: llm.SEMessageStart, Usage: llm.Usage{
|
||||
InputTokens: p.usage.InputTokens,
|
||||
CacheReadTokens: p.usage.CacheReadTokens,
|
||||
CacheWriteTokens: p.usage.CacheWriteTokens,
|
||||
}}, nil) {
|
||||
return
|
||||
}
|
||||
if p.streamErr != nil {
|
||||
yield(llm.StreamEvent{}, p.streamErr)
|
||||
return
|
||||
}
|
||||
if !yield(llm.StreamEvent{Type: llm.SETextDelta, Text: "ok"}, nil) {
|
||||
return
|
||||
}
|
||||
yield(llm.StreamEvent{
|
||||
Type: llm.SEMessageDelta,
|
||||
StopReason: "end_turn",
|
||||
Usage: llm.Usage{OutputTokens: p.usage.OutputTokens},
|
||||
}, nil)
|
||||
}
|
||||
}
|
||||
|
||||
func (p syntheticTelemetryProvider) Complete(context.Context, llm.CompletionRequest) (llm.Message, string, llm.Usage, error) {
|
||||
if p.completeEr != nil {
|
||||
return llm.Message{}, "", p.usage, p.completeEr
|
||||
}
|
||||
return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{llm.TextBlock("done")}}, "end_turn", p.usage, nil
|
||||
}
|
||||
|
||||
func TestMeasureContextExactSizesAndBoundaryModes(t *testing.T) {
|
||||
tools := []llm.ToolSchema{
|
||||
{
|
||||
Name: "Bash",
|
||||
Description: "Run shell command",
|
||||
InputSchema: map[string]any{
|
||||
"type": "object",
|
||||
"properties": map[string]any{
|
||||
"command": map[string]any{"type": "string"},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
Name: "record_fact",
|
||||
Description: "Store a verified fact",
|
||||
InputSchema: map[string]any{"type": "object"},
|
||||
},
|
||||
}
|
||||
wantToolsBytes, err := json.Marshal(tools)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
msgUser1 := llm.UserText("first user prompt")
|
||||
msgAsst := llm.Message{
|
||||
Role: llm.RoleAssistant,
|
||||
Content: []llm.ContentBlock{
|
||||
{Type: llm.BlockToolUse, ID: "call_1", Name: "Bash", Input: json.RawMessage(`{"command":"id"}`)},
|
||||
},
|
||||
}
|
||||
toolResBlock := llm.ToolResultText("call_1", "uid=0(root) gid=0(root)", false)
|
||||
msgUser2 := llm.Message{
|
||||
Role: llm.RoleUser,
|
||||
Content: []llm.ContentBlock{toolResBlock},
|
||||
}
|
||||
msgOther := llm.Message{
|
||||
Role: llm.Role("tool"),
|
||||
Content: []llm.ContentBlock{llm.TextBlock("other role text")},
|
||||
}
|
||||
|
||||
mustLen := func(v any) int64 {
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return int64(len(b))
|
||||
}
|
||||
|
||||
// Case 1: DynamicBoundary == 0 -> all system segments as single static block (joined with "\n\n").
|
||||
reqNoSplit := llm.CompletionRequest{
|
||||
System: []string{"STATIC_ONE", "STATIC_TWO"},
|
||||
DynamicBoundary: 0,
|
||||
Tools: tools,
|
||||
Messages: []llm.Message{msgUser1, msgAsst, msgUser2, msgOther},
|
||||
}
|
||||
got1 := MeasureContext(reqNoSplit)
|
||||
if got1.SystemStaticBytes != int64(len("STATIC_ONE\n\nSTATIC_TWO")) || got1.SystemDynamicBytes != 0 {
|
||||
t.Fatalf("DynamicBoundary=0 system = (%d, %d), want (%d, 0)", got1.SystemStaticBytes, got1.SystemDynamicBytes, len("STATIC_ONE\n\nSTATIC_TWO"))
|
||||
}
|
||||
if got1.ToolCount != 2 || got1.ToolsBytes != int64(len(wantToolsBytes)) {
|
||||
t.Fatalf("tools = (count=%d, bytes=%d), want (2, %d)", got1.ToolCount, got1.ToolsBytes, len(wantToolsBytes))
|
||||
}
|
||||
wantUser := mustLen(msgUser1) + mustLen(msgUser2)
|
||||
wantAsst := mustLen(msgAsst)
|
||||
wantOther := mustLen(msgOther)
|
||||
wantToolRes := mustLen(toolResBlock)
|
||||
if got1.MessageCount != 4 ||
|
||||
got1.MessagesUserBytes != wantUser ||
|
||||
got1.MessagesAssistantBytes != wantAsst ||
|
||||
got1.MessagesOtherBytes != wantOther ||
|
||||
got1.ToolResultBytes != wantToolRes ||
|
||||
got1.LastMessageBytes != wantOther {
|
||||
t.Fatalf("messages = %+v, want count=4 user=%d asst=%d other=%d toolRes=%d last=%d",
|
||||
got1, wantUser, wantAsst, wantOther, wantToolRes, wantOther)
|
||||
}
|
||||
|
||||
// Case 2: 0 < DynamicBoundary < len(System) -> split at boundary.
|
||||
reqSplit := llm.CompletionRequest{
|
||||
System: []string{"AAA", "BBBB", "CCCCC", "DD"},
|
||||
DynamicBoundary: 2,
|
||||
}
|
||||
got2 := MeasureContext(reqSplit)
|
||||
if got2.SystemStaticBytes != int64(len("AAA\n\nBBBB")) || got2.SystemDynamicBytes != int64(len("CCCCC\n\nDD")) {
|
||||
t.Fatalf("DynamicBoundary=2 system = (%d, %d), want (%d, %d)",
|
||||
got2.SystemStaticBytes, got2.SystemDynamicBytes, len("AAA\n\nBBBB"), len("CCCCC\n\nDD"))
|
||||
}
|
||||
|
||||
// Case 3: DynamicBoundary >= len(System) -> whole prompt static, dynamic == 0.
|
||||
reqAllStatic := llm.CompletionRequest{
|
||||
System: []string{"AAA", "BBBB"},
|
||||
DynamicBoundary: 5,
|
||||
}
|
||||
got3 := MeasureContext(reqAllStatic)
|
||||
if got3.SystemStaticBytes != int64(len("AAA\n\nBBBB")) || got3.SystemDynamicBytes != 0 {
|
||||
t.Fatalf("DynamicBoundary=5 system = (%d, %d), want (%d, 0)",
|
||||
got3.SystemStaticBytes, got3.SystemDynamicBytes, len("AAA\n\nBBBB"))
|
||||
}
|
||||
|
||||
// Case 4: whitespace-padded and empty segments match norma's llm.joinSystem
|
||||
// (strings.TrimSpace per segment, drop empty, join remaining with "\n\n").
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
segs []string
|
||||
boundary int
|
||||
wantStat int64
|
||||
wantDyn int64
|
||||
}{
|
||||
{
|
||||
name: "padded and empty unsplit",
|
||||
segs: []string{" STATIC ", "", "\n", "TAIL\n\n"},
|
||||
boundary: 0,
|
||||
wantStat: int64(len("STATIC\n\nTAIL")),
|
||||
wantDyn: 0,
|
||||
},
|
||||
{
|
||||
name: "padded split at boundary 2",
|
||||
segs: []string{" A ", " B ", " C "},
|
||||
boundary: 2,
|
||||
wantStat: int64(len("A\n\nB")),
|
||||
wantDyn: int64(len("C")),
|
||||
},
|
||||
{
|
||||
name: "only empty and whitespace",
|
||||
segs: []string{"", " "},
|
||||
boundary: 1,
|
||||
wantStat: 0,
|
||||
wantDyn: 0,
|
||||
},
|
||||
{
|
||||
name: "empty segments around boundary",
|
||||
segs: []string{" ", "HEAD ", "", "\t\n", " TAIL"},
|
||||
boundary: 2,
|
||||
wantStat: int64(len("HEAD")),
|
||||
wantDyn: int64(len("TAIL")),
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := MeasureContext(llm.CompletionRequest{System: tc.segs, DynamicBoundary: tc.boundary})
|
||||
if got.SystemStaticBytes != tc.wantStat || got.SystemDynamicBytes != tc.wantDyn {
|
||||
t.Fatalf("MeasureContext(%q, b=%d) = (%d, %d), want (%d, %d)",
|
||||
tc.segs, tc.boundary, got.SystemStaticBytes, got.SystemDynamicBytes, tc.wantStat, tc.wantDyn)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestContextTelemetryFlagOffAndOnPostgres(t *testing.T) {
|
||||
dsn := os.Getenv("ARTEX_PG_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("ARTEX_PG_DSN not set")
|
||||
}
|
||||
pg, err := db.Open(dsn)
|
||||
if err != nil {
|
||||
db.SkipPGUnavailable(t, err)
|
||||
}
|
||||
defer pg.Close()
|
||||
|
||||
stamp := time.Now().UnixNano()
|
||||
taskOff := fmt.Sprintf("ctx-off-%d", stamp)
|
||||
taskOn := fmt.Sprintf("ctx-on-%d", stamp)
|
||||
t.Cleanup(func() {
|
||||
_, _ = pg.Exec(`DELETE FROM llm_context WHERE task_id IN ($1,$2)`, taskOff, taskOn)
|
||||
_, _ = pg.Exec(`DELETE FROM llm_usage WHERE task_id IN ($1,$2)`, taskOff, taskOn)
|
||||
_, _ = pg.Exec(`DELETE FROM llm_records WHERE task_id IN ($1,$2)`, taskOff, taskOn)
|
||||
})
|
||||
|
||||
sentinelPrompt := "SECRET_PROMPT_SENTINEL_NEVER_STORE"
|
||||
sentinelMsg := "SECRET_USER_MSG_SENTINEL_NEVER_STORE"
|
||||
req := llm.CompletionRequest{
|
||||
System: []string{sentinelPrompt, "DYNAMIC_TAIL"},
|
||||
DynamicBoundary: 1,
|
||||
Tools: []llm.ToolSchema{
|
||||
{Name: "Bash", Description: "SECRET_TOOL_DESC", InputSchema: map[string]any{"type": "object"}},
|
||||
},
|
||||
Messages: []llm.Message{
|
||||
llm.UserText(sentinelMsg),
|
||||
},
|
||||
}
|
||||
wantMetrics := MeasureContext(req)
|
||||
wantUsage := llm.Usage{InputTokens: 1200, OutputTokens: 150, CacheReadTokens: 800, CacheWriteTokens: 100}
|
||||
|
||||
// 1. Flag OFF (default): zero rows in llm_context on both Stream and Complete,
|
||||
// identical behavior regardless of recordBodies.
|
||||
t.Setenv(ContextTelemetryEnv, "")
|
||||
if ContextTelemetryEnabled() {
|
||||
t.Fatal("expected telemetry disabled when env unset")
|
||||
}
|
||||
recOff := Wrap(syntheticTelemetryProvider{usage: wantUsage}, pg, "claude-test", "prof-a", "", "", func() bool { return false })
|
||||
ctxOff := transcript.WithSessionID(WithTaskID(context.Background(), taskOff), "exp1-worker-i1")
|
||||
var streamText string
|
||||
for ev, err := range recOff.Stream(ctxOff, req) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ev.Type == llm.SETextDelta {
|
||||
streamText += ev.Text
|
||||
}
|
||||
}
|
||||
if streamText != "ok" {
|
||||
t.Fatalf("flag-off stream text = %q, want ok", streamText)
|
||||
}
|
||||
msgOff, stopOff, usageOff, err := recOff.Complete(ctxOff, req)
|
||||
if err != nil || msgOff.Text() != "done" || stopOff != "end_turn" || usageOff != wantUsage {
|
||||
t.Fatalf("flag-off Complete = (%q, %q, %+v, %v)", msgOff.Text(), stopOff, usageOff, err)
|
||||
}
|
||||
var offCount int
|
||||
if err := pg.QueryRow(`SELECT COUNT(*) FROM llm_context WHERE task_id=$1`, taskOff).Scan(&offCount); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if offCount != 0 {
|
||||
t.Fatalf("flag-off wrote %d rows to llm_context, want 0", offCount)
|
||||
}
|
||||
|
||||
// 2. Flag ON with recordBodies=FALSE: rows ARE written on Stream (ok + error)
|
||||
// and Complete, with exact byte sizes and zero llm_records rows.
|
||||
t.Setenv(ContextTelemetryEnv, "1")
|
||||
if !ContextTelemetryEnabled() {
|
||||
t.Fatal("expected telemetry enabled when ARTEX_CONTEXT_TELEMETRY=1")
|
||||
}
|
||||
recOn := Wrap(syntheticTelemetryProvider{usage: wantUsage}, pg, "claude-test", "prof-a", "", "", func() bool { return false })
|
||||
ctxOn := transcript.WithSessionID(WithTaskID(context.Background(), taskOn), "exp2-planner")
|
||||
for _, err := range recOn.Stream(ctxOn, req) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if _, _, _, err := recOn.Complete(ctxOn, req); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Error path also records status='error'.
|
||||
recErr := Wrap(syntheticTelemetryProvider{usage: llm.Usage{InputTokens: 400}, streamErr: errors.New("upstream 500")}, pg, "claude-test", "prof-a", "", "", func() bool { return false })
|
||||
for _, err := range recErr.Stream(ctxOn, req) {
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
var recBodiesCount int
|
||||
if err := pg.QueryRow(`SELECT COUNT(*) FROM llm_records WHERE task_id=$1`, taskOn).Scan(&recBodiesCount); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if recBodiesCount != 0 {
|
||||
t.Fatalf("llm_records count = %d with recordBodies=false, want 0", recBodiesCount)
|
||||
}
|
||||
|
||||
rows, err := pg.Query(`
|
||||
SELECT worker, model, profile, tool_count, message_count,
|
||||
system_static_bytes, system_dynamic_bytes, tools_bytes,
|
||||
messages_user_bytes, messages_assistant_bytes, messages_other_bytes,
|
||||
tool_result_bytes, last_message_bytes,
|
||||
input_tokens, output_tokens, cache_read, cache_write, status
|
||||
FROM llm_context WHERE task_id=$1 ORDER BY id`, taskOn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var got []db.LLMContext
|
||||
for rows.Next() {
|
||||
var c db.LLMContext
|
||||
if err := rows.Scan(
|
||||
&c.Worker, &c.Model, &c.Profile, &c.ToolCount, &c.MessageCount,
|
||||
&c.SystemStaticBytes, &c.SystemDynamicBytes, &c.ToolsBytes,
|
||||
&c.MessagesUserBytes, &c.MessagesAssistantBytes, &c.MessagesOtherBytes,
|
||||
&c.ToolResultBytes, &c.LastMessageBytes,
|
||||
&c.InputTokens, &c.OutputTokens, &c.CacheRead, &c.CacheWrite, &c.Status,
|
||||
); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got = append(got, c)
|
||||
}
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("llm_context rows = %d, want 3 (stream ok, complete ok, stream error)", len(got))
|
||||
}
|
||||
for i, c := range got {
|
||||
if c.Worker != "planner" || c.Model != "claude-test" || c.Profile != "prof-a" ||
|
||||
c.ToolCount != wantMetrics.ToolCount || c.MessageCount != wantMetrics.MessageCount ||
|
||||
c.SystemStaticBytes != wantMetrics.SystemStaticBytes ||
|
||||
c.SystemDynamicBytes != wantMetrics.SystemDynamicBytes ||
|
||||
c.ToolsBytes != wantMetrics.ToolsBytes ||
|
||||
c.MessagesUserBytes != wantMetrics.MessagesUserBytes ||
|
||||
c.MessagesAssistantBytes != wantMetrics.MessagesAssistantBytes ||
|
||||
c.MessagesOtherBytes != wantMetrics.MessagesOtherBytes ||
|
||||
c.ToolResultBytes != wantMetrics.ToolResultBytes ||
|
||||
c.LastMessageBytes != wantMetrics.LastMessageBytes {
|
||||
t.Fatalf("row %d metrics = %+v, want %+v", i, c, wantMetrics)
|
||||
}
|
||||
}
|
||||
if got[0].Status != "ok" || got[0].InputTokens != 1200 || got[0].CacheRead != 800 || got[0].CacheWrite != 100 ||
|
||||
got[1].Status != "ok" || got[1].OutputTokens != 150 ||
|
||||
got[2].Status != "error" || got[2].InputTokens != 400 {
|
||||
t.Fatalf("unexpected usage/status rows: %+v", got)
|
||||
}
|
||||
|
||||
// Verify schema contains zero content columns and stored rows hold none of the sentinels.
|
||||
var textCols int
|
||||
if err := pg.QueryRow(`SELECT COUNT(*) FROM information_schema.columns WHERE table_name='llm_context' AND column_name ~ '(body|content|prompt|text|raw|message$)'`).Scan(&textCols); err != nil || textCols != 0 {
|
||||
t.Fatalf("unexpected content-like columns in llm_context: count=%d err=%v", textCols, err)
|
||||
}
|
||||
var serializedRows string
|
||||
if err := pg.QueryRow(`SELECT COALESCE(string_agg(row_to_json(t)::text, ''), '') FROM llm_context t WHERE task_id=$1`, taskOn).Scan(&serializedRows); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, secret := range []string{sentinelPrompt, sentinelMsg, "SECRET_TOOL_DESC", "DYNAMIC_TAIL"} {
|
||||
if containsSubstr(serializedRows, secret) {
|
||||
t.Fatalf("llm_context leaked request content %q", secret)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func containsSubstr(s, sub string) bool {
|
||||
for i := 0; i+len(sub) <= len(s); i++ {
|
||||
if s[i:i+len(sub)] == sub {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -151,6 +151,7 @@ func (r *Recorder) Stream(ctx context.Context, req llm.CompletionRequest) iter.S
|
||||
// stats and must be complete even for interrupted/failed runs, so it is NOT
|
||||
// gated. Only body serialization + response accumulation are skipped when off.
|
||||
recordBodies := r.enabled == nil || r.enabled()
|
||||
collectContext := ContextTelemetryEnabled()
|
||||
start := time.Now()
|
||||
session := transcript.SessionIDFrom(ctx)
|
||||
parsedID, worker := parseSession(session)
|
||||
@@ -165,6 +166,11 @@ func (r *Recorder) Stream(ctx context.Context, req llm.CompletionRequest) iter.S
|
||||
taskID = parsedID
|
||||
}
|
||||
|
||||
var ctxMetrics ContextMetrics
|
||||
if collectContext {
|
||||
ctxMetrics = MeasureContext(req)
|
||||
}
|
||||
|
||||
// Serialize the request only when storing bodies (this is the expensive part).
|
||||
// Alongside it, attach a Capture so the HTTP transport can hand back the
|
||||
// untouched wire bodies — the normalized view below cannot reconstruct them
|
||||
@@ -198,6 +204,10 @@ func (r *Recorder) Stream(ctx context.Context, req llm.CompletionRequest) iter.S
|
||||
}
|
||||
// Lightweight metering row — always written.
|
||||
r.recordUsage(ctx, taskID, expID, worker, usage, int(time.Since(start).Milliseconds()), status)
|
||||
// Opt-in context size telemetry — independent of recordBodies.
|
||||
if collectContext {
|
||||
r.recordContext(taskID, worker, ctxMetrics, usage, status)
|
||||
}
|
||||
// Heavy trace row — only when body recording is on.
|
||||
if recordBodies {
|
||||
r.record(req, session, taskID, worker, reqBody, capt, start, textBuf.String(), thinkingBuf.String(), usage, stopReason, err)
|
||||
@@ -252,6 +262,7 @@ func (r *Recorder) Stream(ctx context.Context, req llm.CompletionRequest) iter.S
|
||||
// usage metering, normalized body recording and raw wire capture as Stream.
|
||||
func (r *Recorder) Complete(ctx context.Context, req llm.CompletionRequest) (llm.Message, string, llm.Usage, error) {
|
||||
recordBodies := r.enabled == nil || r.enabled()
|
||||
collectContext := ContextTelemetryEnabled()
|
||||
start := time.Now()
|
||||
session := transcript.SessionIDFrom(ctx)
|
||||
parsedID, worker := parseSession(session)
|
||||
@@ -264,6 +275,11 @@ func (r *Recorder) Complete(ctx context.Context, req llm.CompletionRequest) (llm
|
||||
taskID = parsedID
|
||||
}
|
||||
|
||||
var ctxMetrics ContextMetrics
|
||||
if collectContext {
|
||||
ctxMetrics = MeasureContext(req)
|
||||
}
|
||||
|
||||
reqBody := ""
|
||||
var capt *Capture
|
||||
if recordBodies {
|
||||
@@ -279,6 +295,9 @@ func (r *Recorder) Complete(ctx context.Context, req llm.CompletionRequest) (llm
|
||||
reportStop(ctx, stopReason)
|
||||
}
|
||||
r.recordUsage(ctx, taskID, expID, worker, usage, int(time.Since(start).Milliseconds()), status)
|
||||
if collectContext {
|
||||
r.recordContext(taskID, worker, ctxMetrics, usage, status)
|
||||
}
|
||||
if recordBodies {
|
||||
r.record(req, session, taskID, worker, reqBody, capt, start, msg.Text(), thinkingText(msg), usage, stopReason, err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/llmrec"
|
||||
)
|
||||
|
||||
// pgLLMContextSummary returns aggregated context-size telemetry from
|
||||
// llm_context: per-(worker, model) input percentiles, pro-rata section shares
|
||||
// over real input_tokens, cache_read/input ratio, and top tasks by input
|
||||
// tokens. Read-only reporting endpoint (no UI in Phase A).
|
||||
func (s *Server) pgLLMContextSummary(w http.ResponseWriter, r *http.Request) {
|
||||
pg := s.m.PG()
|
||||
if pg == nil {
|
||||
writeJSON(w, http.StatusOK, map[string]any{
|
||||
"enabled": llmrec.ContextTelemetryEnabled(),
|
||||
"by_worker_model": []db.LLMContextGroupStat{},
|
||||
"top_tasks": []db.LLMContextTaskStat{},
|
||||
})
|
||||
return
|
||||
}
|
||||
q := r.URL.Query()
|
||||
summary, err := pg.LLMContextSummary(db.LLMContextQuery{
|
||||
Days: atoiDefault(q.Get("days"), 30),
|
||||
TaskID: strings.TrimSpace(q.Get("task")),
|
||||
TopTasks: atoiDefault(q.Get("top"), 20),
|
||||
})
|
||||
if err != nil {
|
||||
writeErr(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{
|
||||
"enabled": llmrec.ContextTelemetryEnabled(),
|
||||
"by_worker_model": summary.ByWorkerModel,
|
||||
"top_tasks": summary.TopTasks,
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/llmrec"
|
||||
)
|
||||
|
||||
func TestLLMContextSummaryEndpointAuthAndAggregation(t *testing.T) {
|
||||
s, request := ingestServer(t)
|
||||
|
||||
// Unauthenticated request -> 401.
|
||||
rNoAuth := httptest.NewRequest(http.MethodGet, "/api/llm/context/summary", nil)
|
||||
wNoAuth := httptest.NewRecorder()
|
||||
s.Handler().ServeHTTP(wNoAuth, rNoAuth)
|
||||
if wNoAuth.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("unauthenticated /api/llm/context/summary status=%d, want 401", wNoAuth.Code)
|
||||
}
|
||||
|
||||
taskID := fmt.Sprintf("srv-ctx-%d", time.Now().UnixNano())
|
||||
t.Cleanup(func() { _, _ = s.m.PG().Exec(`DELETE FROM llm_context WHERE task_id=$1`, taskID) })
|
||||
|
||||
if err := s.m.PG().InsertLLMContext(&db.LLMContext{
|
||||
TaskID: taskID, Worker: "worker", Model: "test-model", Profile: "p1",
|
||||
ToolCount: 8, MessageCount: 5,
|
||||
SystemStaticBytes: 250, ToolsBytes: 250,
|
||||
MessagesUserBytes: 400, MessagesAssistantBytes: 100, ToolResultBytes: 300,
|
||||
LastMessageBytes: 200,
|
||||
InputTokens: 1000, OutputTokens: 120, CacheRead: 700, CacheWrite: 100, Status: "ok",
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
t.Setenv(llmrec.ContextTelemetryEnv, "1")
|
||||
w := request(http.MethodGet, "/api/llm/context/summary?days=7&top=5&task="+taskID, "")
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("GET /api/llm/context/summary status=%d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
var resp struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
ByWorkerModel []db.LLMContextGroupStat `json:"by_worker_model"`
|
||||
TopTasks []db.LLMContextTaskStat `json:"top_tasks"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !resp.Enabled || len(resp.ByWorkerModel) != 1 || len(resp.TopTasks) != 1 {
|
||||
t.Fatalf("unexpected summary response: %+v", resp)
|
||||
}
|
||||
g := resp.ByWorkerModel[0]
|
||||
if g.Worker != "worker" || g.Model != "test-model" || g.Calls != 1 || g.InputTokensTotal != 1000 ||
|
||||
g.CacheReadRatio < 0.699 || g.CacheReadRatio > 0.701 ||
|
||||
g.Shares.SystemStatic < 0.249 || g.Shares.Tools < 0.249 || g.Shares.MessagesUser < 0.399 {
|
||||
t.Fatalf("unexpected group stat: %+v", g)
|
||||
}
|
||||
}
|
||||
@@ -903,6 +903,7 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("GET /api/commands", s.pgListCommands)
|
||||
mux.HandleFunc("GET /api/commands/stats", s.pgToolStats) // aggregate call counts per tool
|
||||
mux.HandleFunc("GET /api/llm/records", s.pgListLLMRecords)
|
||||
mux.HandleFunc("GET /api/llm/context/summary", s.pgLLMContextSummary)
|
||||
mux.HandleFunc("DELETE /api/llm/records", s.pgDeleteLLMRecords)
|
||||
mux.HandleFunc("GET /api/llm/records/tasks", s.pgLLMTasks)
|
||||
mux.HandleFunc("GET /api/llm/records/by-model", s.pgTokenByModel) // aggregate this task's token usage by model
|
||||
|
||||
Reference in New Issue
Block a user