Compare commits

..
Author SHA1 Message Date
BaaRai c2d1f557c3 perf(db): per-task metrics for the task detail and an index-served latest activity
getTask computed the list metrics of every task to read one; it now calls TaskListMetrics(expID), which runs the same laterals as TaskListMetricsAll with a WHERE on the exploration. listTasks keeps the global query. The latest_activity lateral ordered by its EXTRACT(EPOCH) output alias, so PostgreSQL read and sorted every activity row of each task; it now orders by activity.created_at and idx_act_latest serves the LIMIT 1. Both queries share one SQL constant.
2026-11-15 08:20:08 +01:00
BaaRai e636408f66 Merge pull request #235 from BaaRai/docs/upstream-changelog
docs: credit the upstream and complete the unreleased changelog
2026-11-15 02:01:55 +01:00
BaaRai fcbd87445e docs: credit the upstream and complete the unreleased changelog 2026-11-15 01:54:50 +01:00
BaaRai 59f768f1ef Merge pull request #234 from BaaRai/feat/proxy-pace-bash
feat(traffic): pace Bash requests at the capture proxy
2026-11-15 01:41:56 +01:00
6 changed files with 268 additions and 41 deletions
+31 -3
View File
@@ -6,12 +6,40 @@ All notable changes to this project are documented in this file. The format is b
### Added ### Added
- **Noise profile at runtime**: System settings has a **Target noise** card. It sets the profile (`off`, `quiet`, `balanced`, `aggressive`) through the `noise_profile` setting, which takes precedence over `ARTEX_NOISE_PROFILE`. The change applies to the next WebFetch, http custom tool and enrich request, and to the next agent prompt, without a restart. The card also shows the per-host counters, which `GET /api/settings/noise` returns. - **Target noise governor**: one shared egress governor paces ARTEX's own HTTP requests (WebFetch, the http custom tools and the enrich probe) against a noise profile (`off` by default, `quiet`, `balanced`, `aggressive`): a per-host token bucket with jitter, per-host and global in-flight caps, 429/503 backoff, GET deduplication with a short cache, and a configurable target User-Agent (`ARTEX_TARGET_USER_AGENT`). Off is a strict passthrough (#205).
- **Bash requests paced through the capture proxy**: with traffic capture on, the Bash proxy URL carries a marker userinfo. The recording proxy paces each request of a marked connection by the noise profile's per-host rate and jitter, and counts it, before it goes upstream. This applies inside CONNECT tunnels too. WebFetch, the http custom tools and enrich keep the unmarked URL and are not paced twice. The proxy now drops `Proxy-Authorization` from every request it forwards or records. This is best effort: only clients that honour the proxy are covered, with capture on. - **Low-and-slow prompt block**: when a noise profile is set, the planner, worker, main agent and chat prompts carry the per-host limits of the profile and the exact rate or delay flags for nmap, nuclei, httpx, katana, ffuf, dalfox and sqlmap; with the profile off the prompts are unchanged (#212).
- **Noise profile at runtime**: System settings has a **Target noise** card. It sets the profile (`off`, `quiet`, `balanced`, `aggressive`) through the `noise_profile` setting, which takes precedence over `ARTEX_NOISE_PROFILE`. The change applies to the next WebFetch, http custom tool and enrich request, and to the next agent prompt, without a restart. The card also shows the per-host counters, which `GET /api/settings/noise` returns (#233).
- **Bash requests paced through the capture proxy**: with traffic capture on, the Bash proxy URL carries a marker userinfo. The recording proxy paces each request of a marked connection by the noise profile's per-host rate and jitter, and counts it, before it goes upstream. This applies inside CONNECT tunnels too. WebFetch, the http custom tools and enrich keep the unmarked URL and are not paced twice. The proxy now drops `Proxy-Authorization` from every request it forwards or records. This is best effort: only clients that honour the proxy are covered, with capture on (#234).
- **Hierarchical budgets**, off by default: a task budget and a per-intent budget take the minimum of their own budget and the parent's remaining budget above the wrap-up reserve, planner calls count inside the task, and reaching the reserve forces a wrap-up (#206).
- **Coverage ledger**: the worker records what it examined, ruled out or left open for each surface and risk class with `record_coverage`; gaps, unexamined classes and completeness appear in the planner context, `GET /api/tasks/{id}/coverage`, the coverage card and the Markdown report (#207).
- **Closed-question judge**: the LLM fallback judge answers eight fixed questions that the code owns and the code takes the decision; every failure now leads to ask instead of allow, and the previous behaviour stays available as `llm_judge_mode = freeform` (#208).
- **argv interception rules**: a new `match_type = argv` matches the parsed prefix of a Bash command exactly, and a pending Bash approval can create such a rule in one click; a code-owned denylist refuses programs where a prefix would allow arbitrary code (#210, #211).
- **Scanner output ingestion**: `ingest_scan_output` parses nmap XML, nuclei JSONL, httpx JSONL and ffuf JSON, and writes assets and facts deterministically, so the model never reads the raw output (#214, #215).
- **bbot ingestion**: a typed parser splits bbot JSONL events by `scope_distance`, only in-scope events go through `insert_assets`, and `ingest_bbot` reads the file line by line with a 64 KiB cap (#199, #200).
- **Tool RAG**, off by default: when the bound tool list of an agent is large, only the subset relevant to the current intent is injected, as a projection that never adds or edits a tool (`ARTEX_TOOL_RAG`) (#204).
- **Fact provenance**: `record_fact` takes a closed `source` enum (default `inference`) and every fact carries the server-set `origin`, both shown in `recent_facts`, `list_facts` and `expand_digest` (#194).
- **Per-message nonce envelope**: agent- and target-derived fields reach the planner and worker user messages inside a code-owned data envelope with a random nonce per message; operator input stays outside it (#195).
- **Intercept risk score and batch approval**: a deterministic risk score (low, medium, high) is shown on each pending approval, and up to 100 pending decisions can be settled in one call, each through the existing single-decision path (#232).
- **`ask_operator` target warning**: a request whose question names a host or URL outside the goal targets is tagged as containing target-derived text (#197).
- **Opt-in update banner**: `ARTEX_UPDATE_CHECK` makes the backend check the latest GitHub release at most once per 24 hours and the UI show a banner; off by default (#202).
- **Release provenance**: release archives carry GitHub build provenance attestations and a CycloneDX SBOM (`gh attestation verify <zip> --repo BaaRai/ARTEX`); release notes are the matching `CHANGELOG.md` section (#219, #201).
### Changed ### Changed
- **Untrusted-output boundary is now on by default**: Norma frames tool results for the worker, planner, chat and main agent unless `ARTEX_UNTRUSTED_BOUNDARY` is set to `0`, `false`, `off` or `no`. It was opt-in via `ARTEX_UNTRUSTED_BOUNDARY=1` in 0.6.0 (#166). - **Untrusted-output boundary is now on by default**: Norma frames tool results for the worker, planner, chat and main agent unless `ARTEX_UNTRUSTED_BOUNDARY` is set to `0`, `false`, `off` or `no`. It was opt-in via `ARTEX_UNTRUSTED_BOUNDARY=1` in 0.6.0 (#166); the default flips here (#203).
- **Versioned migrations**: the `db` package applies numbered `db/migrations/NNNN_slug.sql` files once, in order, under the schema advisory lock, and records a SHA-256 checksum in `schema_migrations`; missing, renamed, changed or out-of-order migrations fail closed (#226).
- **Coverage graph performance**: the coverage graph batches its connector lookups (three queries instead of one per root and per company) and adds the partial index `idx_ts_icp`; on a task of about 3,000 assets the p50 dropped from 126 ms to 27 ms and the queries per call from 221 to 3 (#230).
### Fixed
- **Auto-scope hygiene**: auto `task_scope` rows count for coverage and untested assets only under a declared root, and direct coverage counts anchored assets through fact anchors only; the scope is described as "coverage scope, not enforced" (#196).
- **Enrich HTTP probe**: it now dials through `intercept.PinnedDialer`, so the target gate applies and a probe without a target scope is refused (#198).
- **Agent hook data races**: the eight package-level hooks are synchronised, so a second wire-up no longer races tasks that already read them (#227).
### Security
- **Secrets at rest**: the credential-bearing columns and settings (LLM API keys and proxy, task credentials, MCP server env, notification channel config, search and proxy settings), the exec spec of custom tools and the evidence snapshot heads are encrypted in PostgreSQL with AES-256-GCM (`enc:v1:`) using `ARTEX_SECRET_KEY`, else `keys/secret.key`; existing plaintext rows are backfilled once at startup, and a missing or wrong key while encrypted data exists refuses to start. This protects database dumps and backups, not a running instance (#213, #216, #217, #218).
- **URL credentials are dropped before storage**: scan and bbot ingestion, the asset store and archive restore strip URL userinfo, so `https://user:pass@host/app` is stored and matched as `https://host/app` (#222, #223, #224).
## [0.6.0] - 2026-11-12 ## [0.6.0] - 2026-11-12
+8
View File
@@ -6,6 +6,8 @@
**Autonomous penetration testing that keeps its receipts.** **Autonomous penetration testing that keeps its receipts.**
(upstream: [Autumn-27/ARTEX](https://github.com/Autumn-27), AGPL-3.0)
Self-hosted multi-agent offensive security. A planner explores, workers run real tools, Self-hosted multi-agent offensive security. A planner explores, workers run real tools,
and every finding stays bound to the assets and HTTP traffic that prove it. and every finding stays bound to the assets and HTTP traffic that prove it.
@@ -693,6 +695,12 @@ Operational security properties:
--- ---
## About this fork
This repository is the English translation and continuation of ARTEX, originally by [Autumn-27](https://github.com/Autumn-27). The upstream repository is no longer public, so the link points to the author profile. Changes in this fork are tracked in [CHANGELOG.md](CHANGELOG.md), under the same AGPL-3.0 license.
---
## References and prior art ## References and prior art
- [Cairn](https://github.com/oritera/Cairn) - [Cairn](https://github.com/oritera/Cairn)
+50 -26
View File
@@ -1486,11 +1486,13 @@ type TaskListMetrics struct {
Findings FindingSeverityCounts // this task's vulnerability count in the findings table (bucketed by severity) Findings FindingSeverityCounts // this task's vulnerability count in the findings table (bucketed by severity)
} }
// TaskListMetricsAll returns list aggregates for every live task in one query. // taskListMetricsQuery is the select shared by TaskListMetricsAll and
// The lateral lookups use the per-exploration indexes instead of grouping the // TaskListMetrics; each appends its own WHERE on task. The lateral lookups use
// complete activity history on every poll. // the per-exploration indexes instead of grouping the complete activity history
func (d *DB) TaskListMetricsAll() (map[int64]TaskListMetrics, error) { // on every poll. latest_activity orders by the column activity.created_at, so
rows, err := d.Query(` // idx_act_latest (exploration_id, created_at DESC) serves the LIMIT 1; ordering
// by the output alias would sort every activity row of the task.
const taskListMetricsQuery = `
SELECT task.exploration_id, SELECT task.exploration_id,
COALESCE(token_metrics.input_tokens,0), COALESCE(token_metrics.input_tokens,0),
COALESCE(token_metrics.output_tokens,0), COALESCE(token_metrics.output_tokens,0),
@@ -1514,10 +1516,10 @@ func (d *DB) TaskListMetricsAll() (map[int64]TaskListMetrics, error) {
WHERE exploration_id=task.exploration_id AND kind='result' WHERE exploration_id=task.exploration_id AND kind='result'
) token_metrics ON true ) token_metrics ON true
LEFT JOIN LATERAL ( LEFT JOIN LATERAL (
SELECT EXTRACT(EPOCH FROM created_at)::bigint AS created_at SELECT EXTRACT(EPOCH FROM activity.created_at)::bigint AS created_at
FROM activity FROM activity
WHERE exploration_id=task.exploration_id WHERE activity.exploration_id=task.exploration_id
ORDER BY created_at DESC ORDER BY activity.created_at DESC
LIMIT 1 LIMIT 1
) latest_activity ON true ) latest_activity ON true
LEFT JOIN LATERAL ( LEFT JOIN LATERAL (
@@ -1539,30 +1541,41 @@ func (d *DB) TaskListMetricsAll() (map[int64]TaskListMetrics, error) {
FROM findings FROM findings
WHERE task_id=task.id WHERE task_id=task.id
) finding_metrics ON true ) finding_metrics ON true
WHERE task.deleted_at IS NULL`) WHERE task.deleted_at IS NULL`
// scanTaskListMetrics reads one row of taskListMetricsQuery.
func scanTaskListMetrics(row interface{ Scan(...any) error }) (int64, TaskListMetrics, error) {
var explorationID int64
var metrics TaskListMetrics
err := row.Scan(
&explorationID,
&metrics.Tokens.InputTokens,
&metrics.Tokens.OutputTokens,
&metrics.Tokens.CacheReadTokens,
&metrics.Tokens.CacheWriteTokens,
&metrics.LastActivity,
&metrics.Goals.Total,
&metrics.Goals.Met,
&metrics.RunningIntents,
&metrics.Findings.Critical,
&metrics.Findings.High,
&metrics.Findings.Medium,
&metrics.Findings.Low,
)
return explorationID, metrics, err
}
// TaskListMetricsAll returns list aggregates for every live task in one query.
func (d *DB) TaskListMetricsAll() (map[int64]TaskListMetrics, error) {
rows, err := d.Query(taskListMetricsQuery)
if err != nil { if err != nil {
return nil, err return nil, err
} }
defer rows.Close() defer rows.Close()
out := map[int64]TaskListMetrics{} out := map[int64]TaskListMetrics{}
for rows.Next() { for rows.Next() {
var explorationID int64 explorationID, metrics, err := scanTaskListMetrics(rows)
var metrics TaskListMetrics if err != nil {
if err := rows.Scan(
&explorationID,
&metrics.Tokens.InputTokens,
&metrics.Tokens.OutputTokens,
&metrics.Tokens.CacheReadTokens,
&metrics.Tokens.CacheWriteTokens,
&metrics.LastActivity,
&metrics.Goals.Total,
&metrics.Goals.Met,
&metrics.RunningIntents,
&metrics.Findings.Critical,
&metrics.Findings.High,
&metrics.Findings.Medium,
&metrics.Findings.Low,
); err != nil {
return nil, err return nil, err
} }
out[explorationID] = metrics out[explorationID] = metrics
@@ -1570,6 +1583,17 @@ func (d *DB) TaskListMetricsAll() (map[int64]TaskListMetrics, error) {
return out, rows.Err() return out, rows.Err()
} }
// TaskListMetrics returns the same aggregates for the one live task of
// exploration expID, with the same laterals as TaskListMetricsAll. A missing or
// deleted task gives zero metrics, as a missing key of TaskListMetricsAll does.
func (d *DB) TaskListMetrics(expID int64) (TaskListMetrics, error) {
_, metrics, err := scanTaskListMetrics(d.QueryRow(taskListMetricsQuery+` AND task.exploration_id=$1`, expID))
if errors.Is(err, sql.ErrNoRows) {
return TaskListMetrics{}, nil
}
return metrics, err
}
// GoalCountsAll returns goal totals (total / met) per exploration id, one query // GoalCountsAll returns goal totals (total / met) per exploration id, one query
// for all tasks — used by listTasks so the task list shows progress for every task. // for all tasks — used by listTasks so the task list shows progress for every task.
func (d *DB) GoalCountsAll() (map[int64]GoalCounts, error) { func (d *DB) GoalCountsAll() (map[int64]GoalCounts, error) {
+130
View File
@@ -117,3 +117,133 @@ VALUES ($1,'goal','{}','met'), ($1,'goal','{}','open')`, task.ExplorationID); er
t.Fatalf("unexpected goal metrics: %+v", metrics.Goals) t.Fatalf("unexpected goal metrics: %+v", metrics.Goals)
} }
} }
// TaskListMetrics(expID) is exactly the row TaskListMetricsAll returns for that
// exploration, on every counter, and TaskListMetricsAll still lists every task.
func TestTaskListMetricsMatchesAll(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) - skipping", err)
}
t.Cleanup(func() { d.Close() })
newest := time.Now().Add(-time.Minute).Truncate(time.Second)
seed := func(name string, n int) *Task {
t.Helper()
task, err := d.CreateTask(name, "goal", nil, 0, 0)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_, err := d.DeleteTaskCascade(task.ID, false, true)
cleanupErr(t, "task "+name, err)
})
exp := task.ExplorationID
// n result rows with tokens; the newest activity first, then an older row
// with a higher id, so ordering by id instead of created_at would fail.
for _, q := range []struct {
sql string
args []any
}{
{`INSERT INTO activity(exploration_id,kind,input_tokens,output_tokens,cache_read_tokens,cache_write_tokens,created_at)
SELECT $1,'result',10,5,3,1,$3::timestamptz - interval '1 hour' FROM generate_series(1,$2::int)`, []any{exp, n, newest}},
{`INSERT INTO activity(exploration_id,kind,created_at) VALUES ($1,'text',$2::timestamptz)`, []any{exp, newest}},
{`INSERT INTO activity(exploration_id,kind,created_at) SELECT $1,'text',$3::timestamptz - interval '2 hours' FROM generate_series(1,$2::int)`, []any{exp, n, newest}},
{`INSERT INTO exploration_nodes(exploration_id,kind,payload,state)
SELECT $1,'goal','{}',CASE WHEN g<=$2::int THEN 'met' ELSE 'open' END FROM generate_series(1,$2::int+1) g`, []any{exp, n}},
{`INSERT INTO exploration_nodes(exploration_id,kind,payload,state)
SELECT $1,'intent','{}',CASE WHEN g<=$2::int THEN 'running' ELSE 'done' END FROM generate_series(1,$2::int+2) g`, []any{exp, n}},
} {
if _, err := d.Exec(q.sql, q.args...); err != nil {
t.Fatal(err)
}
}
if _, err := d.Exec(`INSERT INTO findings(task_id,severity)
SELECT $1, s FROM unnest(ARRAY['critical','high','high','medium','low','info']) s, generate_series(1,$2)`, task.ID, n); err != nil {
t.Fatal(err)
}
return task
}
a := seed("task-list-metrics-a", 2)
b := seed("task-list-metrics-b", 3)
all, err := d.TaskListMetricsAll()
if err != nil {
t.Fatal(err)
}
for _, c := range []struct {
task *Task
n int
}{{a, 2}, {b, 3}} {
row, ok := all[c.task.ExplorationID]
if !ok {
t.Fatalf("TaskListMetricsAll misses task %d", c.task.ID)
}
one, err := d.TaskListMetrics(c.task.ExplorationID)
if err != nil {
t.Fatal(err)
}
if one != row {
t.Fatalf("task %d: TaskListMetrics=%+v, TaskListMetricsAll=%+v", c.task.ID, one, row)
}
n := c.n
want := TaskListMetrics{
Tokens: TokenUsage{InputTokens: 10 * n, OutputTokens: 5 * n, CacheReadTokens: 3 * n, CacheWriteTokens: n},
LastActivity: newest.Unix(),
Goals: GoalCounts{Total: n + 1, Met: n},
RunningIntents: n,
Findings: FindingSeverityCounts{Critical: n, High: 2 * n, Medium: n, Low: n},
}
if one != want {
t.Fatalf("task %d metrics=%+v, want %+v", c.task.ID, one, want)
}
}
if none, err := d.TaskListMetrics(-1); err != nil || none != (TaskListMetrics{}) {
t.Fatalf("missing exploration: %+v, %v; want zero metrics", none, err)
}
}
// BenchmarkTaskListMetrics compares the per-exploration metrics query with the
// all-tasks one on three tasks of 40,000 activity rows each. No time assertion.
func BenchmarkTaskListMetrics(b *testing.B) {
dsn, _, err := DSN()
if err != nil {
b.Skipf("no database config (%v) - skipping", err)
}
d, err := Open(dsn)
if err != nil {
b.Skipf("postgres unavailable (%v) - skipping", err)
}
b.Cleanup(func() { d.Close() })
var exp int64
for i := range 3 {
task, err := d.CreateTask(fmt.Sprintf("bench-task-list-metrics-%d", i), "goal", nil, 0, 0)
if err != nil {
b.Fatal(err)
}
b.Cleanup(func() { _, _ = d.DeleteTaskCascade(task.ID, false, true) })
if _, err := d.Exec(`INSERT INTO activity(exploration_id,kind,input_tokens,created_at)
SELECT $1, CASE WHEN g%20=0 THEN 'result' ELSE 'tool_use' END, 100, now() - (g||' seconds')::interval
FROM generate_series(1,40000) g`, task.ExplorationID); err != nil {
b.Fatal(err)
}
exp = task.ExplorationID
}
if _, err := d.Exec(`ANALYZE activity`); err != nil {
b.Fatal(err)
}
b.Run("one", func(b *testing.B) {
for range b.N {
if _, err := d.TaskListMetrics(exp); err != nil {
b.Fatal(err)
}
}
})
b.Run("all", func(b *testing.B) {
for range b.N {
if _, err := d.TaskListMetricsAll(); err != nil {
b.Fatal(err)
}
}
})
}
+2 -2
View File
@@ -1927,11 +1927,11 @@ func (s *Server) getTask(w http.ResponseWriter, r *http.Request) {
return return
} }
dto := taskDTO(t, s.resolvedTaskStatus(t)) dto := taskDTO(t, s.resolvedTaskStatus(t))
metrics, err := s.m.PG().TaskListMetricsAll() metrics, err := s.m.PG().TaskListMetrics(t.ExpID)
if err != nil { if err != nil {
log.Printf("[tasks] detail metrics query failed for %s — metrics zeroed: %v", t.ID, err) log.Printf("[tasks] detail metrics query failed for %s — metrics zeroed: %v", t.ID, err)
} }
s.applyTaskMetrics(&dto, t, metrics[t.ExpID]) s.applyTaskMetrics(&dto, t, metrics)
archiveBlockers, _ := s.m.PG().TaskArchiveBlockers() archiveBlockers, _ := s.m.PG().TaskArchiveBlockers()
applyTaskArchiveBlocker(&dto, archiveBlockers) applyTaskArchiveBlocker(&dto, archiveBlockers)
writeJSON(w, 200, dto) writeJSON(w, 200, dto)
+47 -10
View File
@@ -38,6 +38,21 @@ VALUES ($1,'goal','{}','met'), ($1,'goal','{}','open')`, task.ExpID); err != nil
t.Fatal(err) t.Fatal(err)
} }
// A second task, so the detail must pick its own row and the list must keep
// serving every task.
other, err := m.CreateTask("detail metrics other", "goal", nil, 0, 0)
if err != nil {
t.Fatal(err)
}
otherID, _ := strconv.ParseInt(other.ID, 10, 64)
defer func() { _, _ = m.pg.DeleteTaskCascade(otherID, false, true) }()
defer func() { _, _ = m.pg.Exec(`DELETE FROM findings WHERE task_id=$1`, taskID) }()
if _, err := m.pg.Exec(`
INSERT INTO findings(task_id, severity) VALUES ($1,'critical'), ($1,'high'), ($1,'high'), ($1,'low'), ($2,'medium')`,
taskID, otherID); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
s := New(ctx, m, t.TempDir(), t.TempDir(), t.TempDir()) s := New(ctx, m, t.TempDir(), t.TempDir(), t.TempDir())
defer stopTestServer(t, s, cancel) defer stopTestServer(t, s, cancel)
@@ -45,7 +60,19 @@ VALUES ($1,'goal','{}','met'), ($1,'goal','{}','open')`, task.ExpID); err != nil
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
// After New: restoring the task resets running intents to open.
if _, err := m.pg.Exec(`
INSERT INTO exploration_nodes(exploration_id, kind, payload, state)
VALUES ($1,'intent','{}','running'), ($1,'intent','{}','done')`, task.ExpID); err != nil {
t.Fatal(err)
}
type findingCounts struct {
Critical int `json:"critical"`
High int `json:"high"`
Medium int `json:"medium"`
Low int `json:"low"`
}
var probe struct { var probe struct {
Tokens struct { Tokens struct {
InputTokens int `json:"input_tokens"` InputTokens int `json:"input_tokens"`
@@ -53,8 +80,10 @@ VALUES ($1,'goal','{}','met'), ($1,'goal','{}','open')`, task.ExpID); err != nil
CacheReadTokens int `json:"cache_read_tokens"` CacheReadTokens int `json:"cache_read_tokens"`
CacheWriteTokens int `json:"cache_write_tokens"` CacheWriteTokens int `json:"cache_write_tokens"`
} `json:"tokens"` } `json:"tokens"`
GoalsMet int `json:"goals_met"` GoalsMet int `json:"goals_met"`
GoalsTotal int `json:"goals_total"` GoalsTotal int `json:"goals_total"`
InFlight int `json:"in_flight"`
Findings findingCounts `json:"findings"`
} }
req := httptest.NewRequest(http.MethodGet, "/api/tasks/"+task.ID, nil) req := httptest.NewRequest(http.MethodGet, "/api/tasks/"+task.ID, nil)
req.Header.Set("Authorization", "Bearer "+token) req.Header.Set("Authorization", "Bearer "+token)
@@ -73,6 +102,9 @@ VALUES ($1,'goal','{}','met'), ($1,'goal','{}','open')`, task.ExpID); err != nil
if probe.GoalsMet != 1 || probe.GoalsTotal != 2 { if probe.GoalsMet != 1 || probe.GoalsTotal != 2 {
t.Fatalf("detail goal progress not filled from metrics: %d/%d", probe.GoalsMet, probe.GoalsTotal) t.Fatalf("detail goal progress not filled from metrics: %d/%d", probe.GoalsMet, probe.GoalsTotal)
} }
if probe.InFlight != 1 || probe.Findings != (findingCounts{Critical: 1, High: 2, Low: 1}) {
t.Fatalf("detail in-flight/findings not filled from this task's metrics: %d %+v", probe.InFlight, probe.Findings)
}
// The list endpoint must report the same numbers for the same task. // The list endpoint must report the same numbers for the same task.
listReq := httptest.NewRequest(http.MethodGet, "/api/tasks", nil) listReq := httptest.NewRequest(http.MethodGet, "/api/tasks", nil)
@@ -89,27 +121,32 @@ VALUES ($1,'goal','{}','met'), ($1,'goal','{}','open')`, task.ExpID); err != nil
InputTokens int `json:"input_tokens"` InputTokens int `json:"input_tokens"`
OutputTokens int `json:"output_tokens"` OutputTokens int `json:"output_tokens"`
} `json:"tokens"` } `json:"tokens"`
GoalsMet int `json:"goals_met"` GoalsMet int `json:"goals_met"`
GoalsTotal int `json:"goals_total"` GoalsTotal int `json:"goals_total"`
InFlight int `json:"in_flight"`
Findings findingCounts `json:"findings"`
} `json:"tasks"` } `json:"tasks"`
} }
if err := json.Unmarshal(listRec.Body.Bytes(), &listed); err != nil { if err := json.Unmarshal(listRec.Body.Bytes(), &listed); err != nil {
t.Fatal(err) t.Fatal(err)
} }
found := false found, foundOther := false, false
for _, lt := range listed.Tasks { for _, lt := range listed.Tasks {
if lt.ID == other.ID {
foundOther = lt.Findings == (findingCounts{Medium: 1})
}
if lt.ID != task.ID { if lt.ID != task.ID {
continue continue
} }
found = true found = true
if lt.Tokens.InputTokens != probe.Tokens.InputTokens || if lt.Tokens.InputTokens != probe.Tokens.InputTokens ||
lt.Tokens.OutputTokens != probe.Tokens.OutputTokens || lt.Tokens.OutputTokens != probe.Tokens.OutputTokens ||
lt.GoalsMet != probe.GoalsMet || lt.GoalsTotal != probe.GoalsTotal { lt.GoalsMet != probe.GoalsMet || lt.GoalsTotal != probe.GoalsTotal ||
t.Fatalf("list and detail metrics disagree: list=%+v/%d detail=%+v/%d", lt.InFlight != probe.InFlight || lt.Findings != probe.Findings {
lt.Tokens, lt.GoalsMet, probe.Tokens, probe.GoalsTotal) t.Fatalf("list and detail metrics disagree: list=%+v detail=%+v", lt, probe)
} }
} }
if !found { if !found || !foundOther {
t.Fatalf("task %s missing from list response", task.ID) t.Fatalf("list misses a task or its metrics: task %s found=%v, task %s with its findings=%v", task.ID, found, other.ID, foundOther)
} }
} }