Commit Graph
207 Commits
Author SHA1 Message Date
GRAB Racem 2aab3e8210 fix(auth): require a setup token for /api/auth/init and bind 8787 to localhost
POST /api/auth/init accepted any caller while auth.password_hash was empty,
so whoever reached port 8787 first could set the admin password and take the
instance. When no password is set at startup, the server now generates a
random one-time setup token and writes it to the server log. authInit
compares it in constant time, rejects a missing or wrong token with 403, and
clears it once the password is stored.

The setup page has a field for the token and api.initPassword sends it.
docker-compose publishes 8787 on 127.0.0.1 only, like 8788. README says where
to find the token.
2026-11-03 22:18:27 +01:00
BaaRai e6f4de5ddc fix(security): scrub startup secrets before any child can spawn
The scrub ran in main after server.New returned, but New resumes every
non-terminal task before returning, and those loops start workers that run
shells. Move the scrub into New, right after both environment readers, and
start MCP discovery after it as well: both paths spawn child processes.

New takes an option rather than scrubbing unconditionally. The test suite
builds several servers in one process and still needs ARTEX_PG_DSN on later
calls, so only the binary opts in.
2026-11-03 22:14:35 +01:00
BaaRai 1c35c9cf68 fix(ollama): normalize the base URL on the environment path too
The saved-profile path normalized /v1 but FromEnv did not, so
ARTEX_LLM_BASE_URL=http://host:11434 sent chat to /chat/completions and got a
404 on a LAN Ollama server. Share one helper between both entry points, and add
an env-path test with a bare host and with a full endpoint URL.

Also soften the keyless comment: a local Ollama ignores the bearer token, but a
gateway in front of it can require one, which belongs in a saved profile.
2026-11-02 17:03:57 +01:00
BaaRai 978f308a9e fix(ollama): keyless env config and /v1 base URL normalization
Two review fixes on top of the Ollama provider:

FromEnv no longer inherits OPENAI_API_KEY for the ollama format. Ollama
ignores the bearer token, so inheriting the cloud key could only ship it to
the local or LAN endpoint, contradicting the comment in NewProvider and the
documented behavior.

ConfigFrom now normalizes the base URL to the /v1 root, so the natural
Ollama URL (host:11434 without /v1) works for chat as well. Previously the
provider appended /chat/completions to it and the request landed on a path
Ollama does not serve, while pgListModels already appended /v1.

Tests: bare-host normalization assertion, the /v1/chat/completions path
assertion, and a new test that the env config carries no inherited key.
2026-10-25 19:18:25 +01:00
Glauber Portella 80a1ee96f6 ollama llm 2026-10-23 18:12:38 -03:00
stevenAthompson 2d5f9677f4 chore: remove automatic update check and in-app self-update
The update source is hard-coded to a repository that no longer exists and the release checksums are unsigned. Removes the selfupdate package and the /api/update/* endpoints, the top-bar check that queried GitHub on every page load, the Version & Updates card, and the exit-75 swap protocol in the start scripts.
2026-10-22 23:58:37 -04:00
BaaRai a195dfee2a fix(test): escalation test to the ctx-aware hooks.current signature
Seam fix after merging #111 (escalation) with #114 (ctx-threaded
hooks.current): the escalation test predates the signature change and kept
compiling on its own branch, so the merge left main red.
2026-10-21 15:16:25 +02:00
BaaRai ff017216bf Merge pull request #114 from BaaRai/feat/per-intent-llm-rotation
feat(engine): per-intent LLM chain-head rotation (opt-in llm_rotate)
2026-10-21 15:13:56 +02:00
BaaRai 6894c391dd fix: test isolation for the escalation streak + compliant goal fixtures
- resetLLMFailStreaksForTest helper: the package-global streak is in-memory
  state, so the no-switch-on-ordinary-error subtests now reset it before each
  case (accumulation across subtests sharing task/profile keys made the third
  case escalate, breaking the contract the test guards)
- core lifecycle and credentials fixtures: goals now name the lab target, per
  the ambiguity gate (these tests exercise lifecycle/credentials, not the gate)
- canonical reset helper moved into task_llm.go next to the counter it resets
2026-10-21 15:08:37 +02:00
BaaRai 89348b5512 feat(engine): per-intent LLM chain-head rotation (opt-in llm_rotate)
Implements the per-intent rotation shape from #85: a task created with
llm_rotate=true distributes worker intents round-robin over its LLM chain
heads, so lane A can hunt on one model while lane B hunts on another
(different lenses per intent, no extra tokens). Default stays the classic
ordered failover from the active head; planner/goals/mainagent keep the
active head always.

Mechanics: the engine pins the next chain head into the run context when a
worker claims an intent; the pin travels through the provider resolver
(hooks.current now takes ctx). A pin is only a dispatch preference - it must
belong to the task chain and never bypasses failover state (an exhausted
chain stays exhausted). Schema: tasks.llm_rotate boolean, default false,
idempotent ALTER; exposed on the create payload and the task DTO.

Closes #85 (the ensemble/P4 shape stays parked as before)
2026-10-21 15:00:16 +02:00
BaaRai b237fbaa32 feat(engine): ouroboros takes - goal ambiguity gate, failover escalation, stagnation surfacing
Three orchestration mechanisms from the Q00/ouroboros evaluation (#99):

- Goal ambiguity gate at task creation: a goal must name a target (URL,
  domain, IPv4, localhost:port or absolute path) and reach a length floor,
  else the API returns 400 before any token is spent. Deliberately
  target-less goals opt out with the literal marker [targetless]. Guards
  against the live failure where a goal naming no target sent every worker
  to an unrelated asset.

- Consecutive-failure escalation on the task LLM chain: quota exhaustion
  already advances the chain; now 3 consecutive non-quota call failures on
  the same chain profile advance it too, with the true reason recorded.
  Escalation-only by design (no auto-downgrade: mid-task model swaps risk
  behavior drift; the operator keeps the manual chain reset).

- Stagnation surfacing in the planner overview: finished intents are
  fingerprinted (case-folded, digits collapsed); a direction dispatched 3+
  times with zero findings since its first attempt is flagged in a
  self-describing stagnation_warning payload that instructs the planner to
  diversify instead of grinding. Absent when nothing stagnates.

Closes #99
2026-10-21 14:46:56 +02:00
BaaRai a8f3c2c328 feat(rsi): evidence-gated lesson library in shadow mode
Extracts candidate lessons from terminal tasks (confirmed, negative,
retest caution), gates them on cue + action + evidence, and makes
extraction idempotent through a per-source evidence key.

ARTEX_RSI_LESSONS selects off / shadow / inject. Shadow logs what would
have been shown; inject appends a code-owned, data-not-instructions block
after the planner and worker system prompts. No prompt mutation and no
adoption: status stays candidate.

Credit: the Library admission rule comes from harness2 (Google Research,
Apache-2.0, arXiv 2610.14224); the gain taxonomy from awesome-rsi
(lobehub).
2026-10-21 14:25:35 +02:00
BaaRai b95ece6418 Merge pull request #106 from BaaRai/rsi/skill-gaps
feat(server): capability-gap report from skill misses
2026-10-21 14:24:15 +02:00
BaaRai cbb6f66c4b feat(server): capability-gap report from skill misses
GET /api/skill-gaps aggregates skill_usage rows with found=false by
skill, agent, and task category. Hits are returned for the same key so a
miss rate needs no second query; hit-only skills are excluded. The source
rows already existed: a miss is written when the model names a skill the
platform does not provide.

This is the evidence-backed capability backlog, not an auto-fixer: it
creates no skill and changes no prompt. Motivated by the harness and
skill-evolution sections of awesome-rsi (lobehub).
2026-10-19 00:33:08 +02:00
BaaRai cf369dec4f feat(server): per-task outcome card for RSI baselines
Adds GET /api/tasks/{id}/outcome: finding disposition and severity,
retest verdicts, result-row token totals, tool calls, skill misses, and
wall time in one row. This is the measurement layer an RSI loop needs
before claiming a change helped; it alters no agent behavior.

Metric selection follows harness2 (Google Research, Apache-2.0,
arXiv 2610.14224) and the task-gain / retained-gain / improver-gain
taxonomy in the awesome-rsi list.
2026-10-19 00:28:27 +02:00
BaaRai b269d3e590 fix(server): fill task metrics on the detail endpoint
GET /api/tasks/{id} returned the metrics block zeroed (tokens, goal
progress, in_flight, findings, last_activity) because only listTasks
populated those DTO fields. Extract the fill into applyTaskMetrics and
call it from both handlers so the two endpoints report identical
numbers from the same TaskListMetricsAll query. The swallowed query
error in listTasks is now logged instead of silently zeroing the table.

Test: TestTaskDetailReturnsListMetrics seeds result-activity tokens and
goal nodes, asserts the detail endpoint reports them and matches the
list endpoint; fails on main, passes with the fix.

Fixes #91
2026-10-13 14:49:47 +02:00
BaaRai 9f03b8c16a feat: per-task planner quiet-mode switch, editable mid-run
Quiet-mode was compile-time only; wanting it off mid-task (empty frontier, one long intent, idle lanes) was impossible. tasks.plan_quiet_mode (default on) gates the edge-round skip in runRound, PATCH /api/tasks/{id} flips it live (the next round reads it), creation accepts plan_quiet_mode, and the DTO exposes it.
2026-10-12 21:13:28 +02:00
BaaRai 8d841a6011 Merge pull request #83 from BaaRai/fix/inheritance-test-teardown-order
fix: inheritance test teardown closes the manager after the DeleteTask cleanups
2026-10-12 18:46:55 +02:00
BaaRai 35bd98561c Merge pull request #82 from BaaRai/feat/test-db-per-package
feat: each test package runs on its own database
2026-10-12 18:46:51 +02:00
BaaRai ac8491d666 fix: inheritance test teardown closes the manager after the DeleteTask cleanups
defer m.Close() ran before the t.Cleanup DeleteTask calls, so they raced a closed database and the late writers they trigger could still touch the temp dir while RemoveAll was already walking it (the CI-only TempDir: directory not empty flake). The manager now closes via a t.Cleanup registered first (runs last, LIFO) and the temp dir is removed with a tolerated failure.
2026-10-12 18:43:41 +02:00
BaaRai aaf966872a feat: each test package runs on its own database
agent, db, server and evidence serialized on one shared test database via an advisory lock, while evidence, llmrec and traffic-style packages could still run against it in parallel; under full-suite load that window produced a real SQLSTATE 40P01 deadlock in the blackboard direct-source test. TestMain in every PG-using package now rewrites ARTEX_PG_DSN to a per-package database (created automatically by ensureDatabase), so packages neither contend nor need to serialize; the advisory lock stays as an in-package safety net. llmrec gains the TestMain it lacked, which also gives its requires-isolated-DSN tests a truly isolated database inside ./... runs.
2026-10-12 18:43:29 +02:00
BaaRai f57cf659b5 feat: planner quiet-mode skips edge rounds while work runs and goals hold
Every worker write-back fired a full planning round even when the open-goal set had not changed; on a live three-worker task that was 7 rounds in 45 minutes. Edge-triggered rounds are now dropped while active intents exist and the open-goal fingerprint recorded at the end of the last round is unchanged; heartbeat rounds, goal-set changes and frontier drains still plan immediately. Fails open on store errors.
2026-10-12 18:36:44 +02:00
BaaRai 2cdedd8eee Merge pull request #75 from BaaRai/fix/llm-test-stored-profile-endpoint
fix: llm/test stored profile fills endpoint fields from the row
2026-10-12 08:55:32 +02:00
BaaRai 6b1f04b1ba fix: llm/test stored profile fills endpoint fields from the row
The fallback only borrowed the stored key; provider, model, base_url, proxy, streaming and thinking settings still came from the request form. An empty form (the list-view test button) then hit ConfigFrom defaults, sending the profile key to the canonical provider endpoint and 401ing. Fields the form leaves empty now come from the stored profile; explicit form values still win.
2026-10-12 08:54:29 +02:00
BaaRai c96c3ea3a9 fix: cost ledger charges fresh input only (cache read is part of input tokens)
input_tokens includes the cached subset under Anthropic and OpenAI usage semantics, so pricing input and cache read at their full prices double-counted the cache. The fresh share is now input minus cache reads, clamped at zero for gateways reporting disjoint buckets.
2026-10-12 08:53:20 +02:00
BaaRai 67ff6ffffa feat: cost ledger (per-profile EUR pricing on token usage)
The token metering is real now (input, output, cache read/write per
profile), but it answers "how many tokens" while the question an operator
actually asks is "what did this run cost". Converting requires per-profile
prices, which nowhere exist.

Changes:
- llm_profiles gains four optional price columns (EUR per 1M tokens:
  input, output, cache read, cache write; 0/NULL = unset), migrated
  idempotently and round-tripped through SaveProfile / the profile API
  in one small auditable write
- the usage aggregation computes cost_eur per profile from the stored
  prices, in Go after the archived-task merge so cold-storage tokens are
  priced identically; profiles without prices keep cost_eur nil instead
  of showing a fake zero
- web types carry cost_eur and the price fields; the dashboard card and
  profile-dialog inputs consume them in a follow-up (the API serves the
  data already)
- integration test: priced profile + usage rows -> expected EUR to the
  cent (2M in @1.5 + 0.5M out @4 + 1M cache-read @0.15 = 5.15), plus a
  prices round-trip check

Note: the Go side activates on the next build; the schema migration is
idempotent at startup.
2026-10-12 08:20:37 +02:00
BaaRai 2853f085e5 fix: llm/test stored-key fallback skips the runtime's key normalization
The connection test resolves its API key as: form input, else the stored
profile's key, else the global key. Form keys pass through ConfigFrom,
which trims surrounding whitespace; the stored-key fallback assigned
p.APIKey raw. A key pasted with a trailing space or newline therefore
behaved asymmetrically: conversations worked (the runtime path loads the
profile through ConfigFrom and trims) while the connection test sent the
malformed header and failed auth with 401. SaveProfile had the same gap
one layer down, storing the raw pasted value.

Changes:
- the llm/test fallback now trims the stored key and session header name,
  matching ConfigFrom's normalization so the test sends byte-identical
  headers to what the runtime sends for the same profile
- SaveProfile trims the key before storing (and before deriving the
  hint), so surrounding whitespace never reaches the database
- tests: an integration test saves a profile with a whitespace-padded
  key, runs the real handler against a fake anthropic endpoint, and
  asserts the endpoint received the trimmed key; a second test pins the
  ConfigFrom trim the fallback must match
2026-10-12 07:37:32 +02:00
BaaRai 0e53de78c2 Merge pull request #57 from BaaRai/feat/ask-operator-env-handoff
feat: ask_operator environmental handoff discipline
2026-10-12 06:46:25 +02:00
BaaRai 4b815d5ee0 feat: ask_operator environmental handoff discipline
When a worker hits an environmental failure (sandboxed or filtered
network path, missing binary or key, TLS or proxy interception, missing
runtime permission), nothing today tells it how to stop: transcripts
show turn after turn of retries with different flags, equivalent tools,
or fallback transports, all burning budget against a wall the agent
cannot move.

Changes:

- the ask_operator tool description now covers environmental blocks
  explicitly: do not retry or work around; include the exact single
  command the operator should run in their own terminal as one
  copy-pastable line, plus what to paste back, then wait. The question
  parameter description gains the same guidance.
- worker and pentest seeds gain an "Operator handoff on environmental
  blocks" block: stop after the first environmental failure, no
  flag-hopping, no bypass hunting, one command per ask_operator request,
  never batch.
- tests: the tool description must keep the handoff markers
  (TestAskOperatorEnvHandoffGuidance) and the executor seeds must keep
  the block (TestPromptSeedsCarryOperatorHandoff).

Pattern adapted from bug-bounty-hunting-prompts (MIT), whose operator
handoff rule produced exactly this single-command format.
2026-10-12 06:45:21 +02:00
BaaRai b81c6238a7 feat: codeaudit skill (white-box source audit pipeline)
ARTEX hunts live targets well but has no methodology for auditing source
code handed to it (a plugin, an application, a repository). White-box
audits so far ran on ad-hoc task goals with no shared procedure.

Adds skills/codeaudit/SKILL.md, distilled from VulnHunter's methodology
(Apache-2.0) and adapted to ARTEX mechanics (work dir artifacts, facts,
report_finding, the standing quality and adversarial gates):

- production-code-only scope rules, with the checked-in infra-config
  exception (reverse proxy / trust / CORS directives are in scope)
- phase 1: every user-controllable input inventoried with dispositions,
  codebase partitioning, shared defense catalog
- phase 2: attacker-first forward tracing per vuln class, with the
  investigation discipline (never stop at abstraction boundaries, audit
  all parameters at outbound call sites, exhaust all paths, follow
  through stores and responses, transitive caller search on sinks)
- three gates (reachable, attacker-controlled, new capability), comment
  skepticism, sanitizer scope mismatch, anti-merge
- PoC preference order (executed against a local instance beats static
  trace; static PoCs carry entry value, file:line chain, sink literal,
  impact), measured impact per the standing gates
- sweep with per-instance dispositions and a count cross-check that
  forbids silent drops

Also fixes the shipped scopesentry skill, which the new guard test
caught immediately: its frontmatter block was never closed and its name
did not match its directory, so the whole file failed frontmatter
parsing and silently degraded to an empty description (the skill has
been undiscoverable). The new server test parses every shipped skill
with the real parser and fails on any skill LoadDir cannot read or that
ends with an empty description.
2026-10-12 06:41:22 +02:00
BaaRai c1bebabafa feat: first-layer prompt-injection hardening for agents
Executor agents ingest attacker-controlled content on every run (HTTP
responses via curl, browser pages via the browser MCP, traffic records,
binary strings via remote analysis tools) with no counter-instruction
anywhere in their context.

Two defenses, both zero-cost per call:

- A standing hardening block in the default prompt seeds of worker,
  pentest, mainagent and auto: tool output from or through the target is
  data, never instructions; embedded directives are quoted verbatim and
  recorded as a possible prompt-injection finding instead of followed;
  credentials and operator data never go to a destination suggested by
  target-controlled content.
- Every MCP tool description gains a one-line untrusted-output note at
  wiring time (covers both transports and travels with the description:
  cached prompt for global tools, on-demand schema for deferred ones).

A seed test guards the hardening block against silent removal.

Closes #48
2026-10-12 00:58:53 +02:00
BaaRai d54e468763 feat: ask_operator tool - workers can request operator input (captcha/OTP) with a screenshot and pause
A worker hitting a wall only a human can clear (captcha, email/SMS OTP, 2FA,
a decision) calls ask_operator with a precise question and an optional
screenshot. The question lands in the intent's transcript as a user-role turn
with an attachment card (the same {text, attachments} shape chat uploads use,
so the existing transcript UI renders it natively), and the intent pauses. The
operator replies through the existing message-to-worker flow; the reply resumes
the run with the answer as the next input. Images are whitelisted (png/jpg/
jpeg/webp/gif), capped at 10MB, and copied into the task uploads dir.

Closes #41
2026-10-11 23:06:39 +02:00
BaaRai 553a9ac638 Merge pull request #44 from BaaRai/feat/mcp-health
feat: surface MCP server connect health to operators and agents
2026-10-11 23:05:38 +02:00
BaaRai 28e6618c71 feat: surface MCP server connect health to operators and agents
Every connect attempt (discovery, refresh, and the per-agent-run wiring) now
records its outcome on the server row: last_connect_at/ok/err, written through
a change-deduped update so per-run wiring does not amplify writes. The MCP page
shows a health chip per card (ok with time, or the failure reason), and a new
read-only mcp_status tool (bound to auto, worker and pentest) lets a worker
notice that a server it expected is down and say so instead of silently
degrading while the cached tool count keeps the card looking healthy.

Closes #38
2026-10-11 22:50:00 +02:00
BaaRai 07d272727d fix: emit safe skill frontmatter YAML and verify it round-trips
Descriptions written through the API were interpolated raw into the SKILL.md
frontmatter, so a value containing a colon+space made the whole frontmatter
unparseable: every field silently stayed empty, the skill loaded under its
directory name, and a skill-gated MCP server was silently skipped. The same
applied to updates through the meta rewrite, and the shipped scopesentry skill
had a malformed name line all along.

- yamlScalar: plain-safe tokens stay bare, everything else single-quoted
- fsCreateSkill/fsUpdateSkillMeta emit safe values and re-verify the written
  file with the real skill parser (name + mcps must survive), rejecting the
  write otherwise
- warnBrokenSkills at startup logs skills whose name fell back to the
  directory (the signature of broken frontmatter)
- fix skills/scopesentry/SKILL.md name line

Closes #39
2026-10-11 22:26:22 +02:00
BaaRai 24c8ea13c0 feat: rea reverse-engineering skill + per-server MCP HTTP request timeout
Add a rea skill (skill-gated MCP): agents get binary and application package
reverse engineering through a rea MCP server, with tools kept deferred so the
172-tool catalog never bloats the prompt until the skill loads.

Remote MCP servers now take timeout_seconds (default 120): analysis servers
with minutes-long tool calls were cut off mid-call by the fixed 120s client
timeout. Applies to the Streamable HTTP transport; wired through discovery,
the ScopeSentry sync client, the API and the MCP dialog.

Closes #8
2026-10-11 19:02:23 +02:00
BaaRai d8f47d4359 Traffic ingest: capture operator browsing via a browser extension
Closes #33. The MITM proxy only sees the agent's own actions, so
anything behind a human gate (signup captcha, email OTP, 2FA, SSO)
left the authenticated surface dark. The operator's browser becomes a
capture source instead:

- traffic: exported RecordExternal storing an externally captured
  exchange through the same spill/blob/FTS path as proxy traffic
  (insert logic shared with record()); rejects unparsable URLs and
  empty methods
- server: POST /api/traffic/ingest inside the authenticated API group,
  batch capped at 50 entries, bodies truncated at 256 KiB, per-entry
  ingested/failed counts
- extension/: Manifest V3 Chrome/Edge extension driving the
  chrome.debugger Network domain on the chosen tab, batching text
  exchanges to the ingest endpoint with the operator's URL + JWT in
  chrome.storage.local; binary bodies skipped; flush on batch/stop
- tests for both layers, README section
2026-10-11 14:53:08 +02:00
BaaRai 525ad9ffde Skip the four known-broken tests in code instead of CI -skip flags
The go test -skip flag proved unreliable in the full-suite CI shape
(the same pattern skipped correctly in isolation but ran on the
runner), so each of the four tests now carries an explicit t.Skip with
the reason and a pointer to issue #9. The CI workflow drops the -skip
argument and keeps serial package execution.
2026-10-10 18:12:42 +02:00
BaaRai c177ab9d86 feat(tasks): per-task test-account credentials
Task credentials (#3): declare test accounts at task creation, and never
lose the accounts the agent registers on its own.

- task_credentials table (unique per task+domain+username; upsert
  overwrites the password in place), idempotent schema migration
- POST /api/tasks accepts credentials; GET/POST/DELETE
  /api/tasks/{id}/credentials serve the UI (manual add, e.g. relaying an
  OTP-verified account; delete)
- empty password = the "agent generates" mode: the entry surfaces in the
  worker's graph overview with an explicit instruction to register that
  account with a strong password and record it
- new tools save_task_credential / list_task_credentials for worker and
  pentest; the graph overview injects the task's credentials each round
  with a standing note: reuse matching entries, always record accounts
  you register
- New Task dialog: optional credentials section (domain, username,
  password, note) with a dedicated-test-accounts hint
- task detail: Credentials tab, passwords masked with explicit
  reveal/copy, agent-recorded badge, light polling so mid-run recordings
  appear

One-shot binding migration (settings-flag-guarded, same pattern as the
other seeds) grants the two tools to worker+pentest on databases seeded
before this change; fresh DBs get them via WorkerTools()/the pentest
catalog.

Closes #3
2026-10-09 14:28:08 +02:00
BaaRai 353a947285 fix(tasks): freeze the timeout budget while a task is paused
deadline_at is a wall-clock timestamp, so pause time counted against the
task timeout: pausing overnight (or shutting the machine down while
paused) made the deadline ring while parked, and resuming landed
straight into the settle sequence and a timeout terminal state.

ApplyTaskPause now stamps paused_at, and ApplyTaskAdmission credits the
paused interval back onto deadline_at in the same atomic UPDATE that
ends the pause (paused_at is cleared there too). The shifted deadline
flows to the live handle via the existing RETURNING mirror, so the
engine clock context picks it up. Matches the re-basing the
archive-restore path already did for paused tasks.

Migration is an idempotent ADD COLUMN; tasks paused before the upgrade
have no paused_at and keep the old behavior.

Fixes #1
2026-10-09 11:37:30 +02:00
BaaRai e233c02314 Translate all Chinese text to English
Comments, doc comments, UI labels, log and error messages, LLM prompt
strings, markdown docs, test data and assertions. Identifiers, enum
keys, routes, placeholders and code structure are unchanged.

Notes:
- docs/漏洞流量证据.md renamed to docs/vulnerability-traffic-evidence.md
- tests that exercised CJK specific handling (GBK zip names, Chinese
  full text search) now use ASCII equivalents; go build passes and the
  test suite matches the pre translation baseline (remaining failures
  need a PostgreSQL DSN and fail identically on the original)
- frontend mention syntax now uses English kinds (finding, asset, ...)
  matching the backend, chat mentions tests pass
2026-10-08 23:14:34 +02:00
Autumn-27 66b7006c69 fix bug 2026-10-07 07:45:28 -04:00
Autumn-27 8bc4d58c85 feat(intercept): 模型兜底审批单独计量 token 并在配置页展示
兜底审批(LLM judge)调模型判定漏网命令,其 token 消耗此前无法单独查看:
usage 行要么无 worker 归属、要么被并进复用的模型 profile 桶里。

- llmrec 增加 WithWorker 上下文覆盖,reviewer 调用前标注 worker='judge',
  让审批调用的计量行可被单独筛出(复用现有 llm_usage.worker 列,无需迁移)。
- db.JudgeUsageStats 返回累计总量 + 近 N 天每日序列(WHERE worker='judge')。
- 新增 GET /api/intercept/judge/usage。
- 配置页「模型兜底审批」开关下方新增用量卡片:调用次数 / 输入 / 输出 /
  缓存读写 五项统计 + 近 30 天每日消耗迷你柱图 + 刷新。
2026-10-04 02:43:26 -04:00
Autumn-27 9b88c55af0 fix(notify): 汇总模式按消息计量限流,批次大小不再被每轮请求预算掐死
令牌桶的额度此前被同时当成两个量纲用:对实时模式它是「发几条消息」,
对汇总模式却被直接当成「一批装几条漏洞」传给 ClaimDigestBatch。

notifyMaxSendsPerChannelPerTick(=5)是由租约时长倒推的 HTTP 请求预算,
而一个汇总批次只发一次请求,两者本不该相乘。稳态下更糟:rate_per_min=20
的渠道在 3 秒的 tick 里只补得到 1 个令牌,于是每条汇总消息只装 1 个漏洞,
头部还写着「近 30 分钟新增 1 个漏洞」——digest 退化成带汇总文案的实时推送,
MaxDigestBatchSize(500) 永不可达。

改为统一按**消息条数**计量:一批漏洞合成一条消息、消耗一个令牌,
rate_per_min 对 digest 依然生效(每分钟最多这么多条汇总消息),
而批次大小只受 MaxDigestBatchSize 这个内存上界约束。

ClaimDigestBatch 原先的注释声称「单批同时受限流额度与内存上界约束」,
那正是本问题的来源——限流额度本不该充当批次大小。注释一并改掉,
免得下次又按「rate_per_min 对 digest 不生效」把它改回去。

顺带让 takeTokens 接受 want 上限:原先整桶抽空后才由调用方截断,
多出来的令牌(飞书默认 100/min 时是 95 个)直接丢弃,渠道这一轮
没有待发投递时也照扣,注释声称的突发容量在任何情况下都做不到。

决策收在 digestTickPlan 里由新增用例直接钉住。现有端到端用例都手动给
stepDigest 传一个够大的 limit,绕过了 step 里的额度计算,所以覆盖不到这里。
2026-10-01 06:43:29 -04:00
luckoneandClaude Code 765c0e99e3 fix(notify): 安全审计修复 —— 掩码绕过、SSRF、静默丢失等 12 项
对推送功能做了一轮安全审计,每条都先复现、再修、再补回归测试。
以下按严重程度排列。

【严重】掩码可被「改目标地址、保留凭据」绕过
目标地址与凭据是两套独立字段,而配置合并对「未提及的键」保留原值。于是
只改地址、对凭据避而不谈,服务端就会把库里的真凭据发到任意地址。已实测
复现四个渠道:通用 Webhook 的 Authorization 头、Telegram 的 Bot Token
(进请求路径)、邮件的 SMTP 密码(STARTTLS 后交给对端)、钉钉/飞书的加签
密钥(该项不可利用,一并列出仅为说明面)。完全静默、不依赖重定向,所以
上一轮加的「拒绝跨主机跳转」挡不住它。

修法:新增 Channel.DestinationKeys() 声明「消息发往哪」,PrepareConfigUpdate
在地址变化时要求对**每一个**凭据字段显式表态(给新值,或显式留空表示不再
需要)——原样回传掩码值等于「沿用旧凭据」,正是攻击形状,一并拒绝。
刻意不做「自动丢弃凭据」:对可选字段(如 headers)那会变成「鉴权静默没了
但接口返回成功」,比报错更难排查。

【中】markdown 系三渠道对不可信内容零转义
钉钉/企微/飞书此前完全不转义(Telegram 与邮件都做了),而标题与摘要来自
模型输出、资产 url 来自被测目标。一条标题里的 `[文字](http://evil)` 会在
工程师的钉钉里变成可点击外链;`![](http://evil/beacon)` 会被客户端拉取,
泄露阅读者 IP 并暴露「这条已被看过」。现统一单行化 + markdown 元字符转义。

修的过程中还纠了自己一个设计错误:转义一度被放进 markdownTitle,而它被
markdown 正文、Telegram 的 HTML、飞书卡片、Webhook 的 JSON 与邮件主题四种
语境共用,markdown 的反斜杠泄漏到了 Telegram 消息里(用户会看到 `\(1\)`)。
转义必须发生在各自的渲染出口——测试直接抓到了这个回归。

【中】投递地址的 SSRF 面
此前只校验 scheme 与 host,云元数据 169.254.169.254、环回、内网一律可投递;
而失败响应体前 200 字节会进 last_error 并被投递历史接口回显,构成半盲读
原语。现在在**拨号阶段**设防(最终生效点,同时覆盖 DNS 重绑定与同主机
重定向)。环回/链路本地需显式 ARTEX_NOTIFY_ALLOW_LOCAL=1 才放行(本机
SMTP 中继是合法配置);RFC1918 私网**刻意放行**——内网自建 Mattermost /
SMTP 很常见,做到那个程度会把正常部署一起废掉,这条取舍写进了测试断言。

【中】汇总消息被截断后整批标记已送达 = 静默丢失
渠道都有长度上限(企微 4096 字节最紧)。此前渲染完整篇再截断、然后整批
标记成功:被截掉的那些既不在消息里、也不在失败列表里,投递历史还显示
成功。现改为按**整条**打包,只标记真正装下的,其余回队续发,消息头部
如实写明还有多少条在下一条。被推迟的条目**不消耗重试次数**(领取时乐观
+1 的那次减回去),否则 500 条的积压会在第三段就把尾部判成失败。

【中】其余修复
- min_severity 打错字让过滤器静默失效(未知级别序数为 0 → rank>=0 恒真,
  用户以为「仅高危」实际全推)。改为写入时校验取值。
- rate_per_min=0(不限流)不可达:文档/UI/令牌桶都按 0=不限流解释,唯独
  写库那层 `if <=0 取默认值` 把它改成 20 或 100。默认值改到接口层填。
- digest 渠道完全绕过令牌桶(allow 被扣却没人用)。
- 失败处置按整批 max(attempts) 判断,让老投递连坐同批的新投递。
- 汇总批次里快照无法解析的投递被静默标记成功。
- url.Parse 失败时错误文本仍带完整地址(上一轮只修了 http.Client.Do 那侧;
  且当时的用例全走 scheme 分支、没覆盖解析失败路径,是假保证)。
- 单渠道每轮投递条数可能超出租约时长(多实例下重复发送)。加了防漂移
  断言,写的时候立刻发现原取值 6 恰好用满租约、零余量,已改 5。
- Telegram 截断可能切断 HTML 实体,导致整条被拒收。
- 邮件把 SMTP 4xx(灰名单等临时失败)判成永久失败,最该重试的场景从不重试。
- 提交嵌套结构时会把掩码字面量真的写进库,导致鉴权静默失效。

测试:notify 包从 82% 提到 86%,新增 SSRF 守卫、掩码绕过(用攻击形状的
输入而非「防御逻辑的正确输入」)、分段投递收敛、markdown 转义、SMTP 4xx/5xx
分类等回归用例;端到端补了一条「60 条长标题跨多轮发完且不卡死」的收敛测试。

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-30 16:25:06 +08:00
luckoneandClaude Code 4eb5311998 feat(notify): 漏洞发现支持推送到钉钉/飞书/企业微信等六种渠道
新增「系统 → 通知推送」页,把扫描发现的漏洞推送到钉钉、飞书、企业微信、
通用 Webhook、Telegram、邮件。同一类型可配任意多个机器人实例,每个实例独立
设启停、限流与过滤规则(最低级别 / 任务资产范围 / 漏洞类型关键词 / 是否接收
处置状态变更)。推送时机分实时与汇总两种模式,「高危实时、其余汇总」建两个
渠道即可表达,策略不写死在代码里。带投递历史与失败重发。

三个关键设计:

1. 写漏洞的事务只做一次盲 INSERT。notification_events 由 RecordFindingTx
   在同一事务内写入,保证「漏洞落库」与「推送任务存在」原子一致;这条 INSERT
   刻意不读渠道表、不跑用户的过滤规则,否则一条配错的过滤条件就能污染甚至
   中止事务,让高危漏洞存不进库。为把失败隔离在这一条语句上(PostgreSQL 中
   事务内任一语句报错会让整个事务作废、连 COMMIT 都失败),它被 SAVEPOINT
   包住,失败时只记日志。

2. 投递用租约领取而非长事务。FOR UPDATE SKIP LOCKED 领取后把行置为 sending、
   把 next_attempt_at 推到未来作为租约,提交后再做网络投递,因此投递期间不
   持有数据库锁;崩溃留下的 sending 行会被下一轮重新领取,自愈且不会无限重试。

3. 限流不消耗重试预算。先按渠道令牌桶算出本轮还能发几条,再按这个数量领取;
   顺序反过来会让被限流挡下的投递白计一次尝试,三次预算被等待耗光后落入失败。

审计并修复的四个问题:

- 推送凭据经错误信息外泄:http.Client.Do 返回的 *url.Error 会把完整 URL 打进
  错误文本,而钉钉 access_token、企业微信 key、飞书 hook id、Telegram
  /bot<token>/ 都就在 URL 里。该串流入 last_error(明文落库)、投递历史接口
  (绕过渠道掩码)、服务端日志与前端错误提示。现统一脱敏为 scheme://host
  + 底层原因,并拒绝跨主机重定向。
- 最低级别门槛打错字会让过滤器静默失效:未知级别序数为 0,判定退化成
  rank >= 0 恒真,用户以为限制「仅高危」实际全推。现于写入路径校验取值。
- 汇总批次无上限:一次领取把全部待发行读进内存、消息被渠道长度上限截掉大半
  且静默丢失。现单批上限 500 条,超出部分留库成为下一批。
- 邮件渠道把 SMTP 4xx(灰名单等临时失败)判成永久失败,导致这类最该重试的
  场景从不重试。现按应答码首位区分 4xx/5xx。

测试:notify 包 82% 覆盖(含用自建最小 SMTP 服务器驱动的完整邮件协议会话、
两种签名算法对 OpenSSL 基准值的比对、UTF-8 边界截断、以及模板能力边界的
反射断言);db 与 server 侧共 29 个用例跑真实 PostgreSQL,其中端到端用例
真发 HTTP 到假接收端并断言消息内容与投递状态。

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-30 16:25:06 +08:00
Autumn-27 eb347d5965 feat(traffic): 流量列表新增「清空全部」,并借空库时机压实索引
一次删除全部流量记录,忽略当前筛选条件。与按 host 删除的两条路径相比多做
两件事:

- 清理索引已不再记录的历史 host 目录。按 host 删除只能从索引反查目录,
  行已消失的孤立目录会一直留着;「清空全部」既然语义是不留残余,就直接
  扫顶层目录(跳过 _index/_blobs/_ca/_delete_staging)。
- 收尾做一次全量压实(optimize + VACUUM + wal_checkpoint(TRUNCATE)),把
  索引占用的磁盘空间还给系统,并把实际释放量返回给前端提示。

VACUUM 的耗时与临时空间只和它保留的内容成正比,所以空库是做全量压实唯一
免费的时机——也正因如此,这一步顺带成为已有实例的转换途径:auto_vacuum
只能通过 VACUUM 生效,清空一次之后旧库即转为增量回收模式,此后日常按
host 删除就能自行回收空间(TestDeleteAllConvertsLegacyIndex 覆盖)。

已绑定到漏洞的流量证据保存在独立的证据库中,不受影响——绑定时已拷出独立
副本,正是为了让热流量可以随时清空而不带走证据。确认弹窗里明确写了这点。

前端把「清空全部」做成 outline 样式而非第二个 destructive 按钮:它忽略一切
筛选条件,不该看起来和「删除该目标」只差一次误点;无流量时禁用。

测试:新增 TestDeleteAllPurgesAndCompacts(清空后索引、正文、blob 引用与
孤立目录全清,空间回收到 1/8 以下,清空后仍可继续录制)与
TestDeleteAllConvertsLegacyIndex(旧库经清空后转为增量回收,随后普通删除
即可回收)。
2026-09-30 02:50:49 -04:00
Autumn-27 6d8c50a12e feat(traffic): 流量列表支持内容检索、路径/状态码/长度筛选与列排序 (#177)
- 后端 traffic.Page 改为 PageQuery:新增响应内容(FTS)、路径(url_template)、
  状态码(精确/Nxx 分桶)、响应长度范围筛选,并支持按 ts/status/resp_len 排序
  (尾随 id DESC 保证分页稳定);补 status/resp_len 两个索引(IF NOT EXISTS,旧库自动生效)
- server getTraffic 解析 body/path/status/resp_min/resp_max/sort/order 参数
- 前端流量页新增高级筛选行(响应内容/路径/状态码/长度范围)与可排序表头,
  排序偏好持久化到 localStorage;抽出公共组件 ui/sortable-head
- 仅改动前端页走的 Page,不动 Agent 工具的 query 契约
2026-09-29 21:58:17 -04:00
Autumn-27 c63955481d fix(tools): MCP/自定义工具输出统一走 Capture 截断,修复 mcp 截断不生效 (#176) 2026-09-29 10:09:23 -04:00
Autumn-27 1a10c153b0 feat(tasks): 漏洞列改为按严重度分档显示 严/高/中/低
- TaskListMetricsAll 用 COUNT(*) FILTER 按 severity 分四档统计(同一条查询)
- TaskListMetrics.Findings 及 TaskDTO.findings 由数字改为 {critical,high,medium,low}
- 列表页「漏洞」列渲染 x/x/x/x,配色沿用 severity 语义、为 0 的档位弱化
- severity 为空/白名单外的记录不计入任一档;mock 同步为对象结构
2026-09-24 13:32:45 -04:00