mirror of
https://github.com/BaaRai/ARTEX.git
synced 2026-11-14 20:20:54 +08:00
Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8fd83ef070 | ||
|
|
bba7989903 | ||
|
|
89ece8eb8d | ||
|
|
623e5e58a7 | ||
|
|
0777edfb8f | ||
|
|
67971ef7da | ||
|
|
57cac93d49 | ||
|
|
b269d3e590 |
@@ -0,0 +1,114 @@
|
||||
# Recursive self-improvement in ARTEX
|
||||
|
||||
Status: measurement and shadow memory only. There is no harness search, no
|
||||
automatic adoption, and no self-modifying code in this repository.
|
||||
|
||||
This file defines the contract any future harness-optimization work must meet,
|
||||
so the first experiment cannot quietly become an unguarded rewrite loop.
|
||||
|
||||
## The three loops
|
||||
|
||||
| loop | object | state |
|
||||
|---|---|---|
|
||||
| A. within a task | planner, worker, retester, adversarial gate | exists |
|
||||
| B. across tasks | evidence-gated lessons from closed tasks | shadow mode, PR #104 |
|
||||
| C. on the harness | prompts, skills, tool guidance | **not built**; this is its contract |
|
||||
|
||||
Loop A produces task gain. Loop B is meant to produce retained gain: a lesson
|
||||
must survive a closed task and change a later one. Loop C is the only loop that
|
||||
can produce improver gain, and it is the only one that can silently damage the
|
||||
product, which is why it goes last.
|
||||
|
||||
## Loop C contract
|
||||
|
||||
Any PR that proposes, probes, or adopts harness edits must satisfy all of:
|
||||
|
||||
1. **Live tasks propose, benchmarks adopt.** A successful bounty task can
|
||||
generate a candidate lesson or edit, but it can never be the evidence for
|
||||
adoption. Adoption requires held-out benchmark runs, because a live task is
|
||||
unreproducible and contaminated by its target.
|
||||
2. **Owner approval per adoption.** The loop may prepare a proposal branch and
|
||||
show the diff. It may not change the active prompt. Adoption is one explicit
|
||||
owner action, like a baseline pin.
|
||||
3. **Immutable surfaces.** The following are never in the search space:
|
||||
the adversarial and quality gates, scope and authorization checks,
|
||||
prompt-injection defenses, tool allowlists, and the code-owned prompt tails
|
||||
(traffic tools, artifact rules, lesson block). Only the editable prompt body,
|
||||
skill markdown, and tool descriptions can be proposed.
|
||||
4. **One-step rollback.** Every adopted prompt change is a new
|
||||
`agent_prompts` version, so rollback is a pointer change. Skills are files
|
||||
under git; no adoption may bypass review.
|
||||
5. **Flag off by default.** The loop runs behind a feature flag. Turning it on
|
||||
requires an owner decision, and the flag is checked per run.
|
||||
6. **Bounded edit budget.** A round may propose a bounded number of edits
|
||||
(annealed over rounds). A critic proposes only edits that are expected to
|
||||
transfer; a pruner rejects edits that only fit the probe task. This follows
|
||||
RRSI (arXiv 2609.24972).
|
||||
7. **Attribution.** Probe results are compared against an unchanged baseline on
|
||||
the same task and seed. Regressions are attributed to a specific module
|
||||
rather than to the loop as a whole; the modular fault-localization idea is
|
||||
from ModularRSI (arXiv 2609.14857). A paired evolving/non-evolving comparison
|
||||
guards against misattributed safety regressions, following SEABench
|
||||
(arXiv 2609.35596).
|
||||
8. **No self-approval and no silent state.** The loop cannot set its own status,
|
||||
edit its own acceptance criteria, or write to the lesson library as
|
||||
`adopted`. Lesson status changes stay owner actions.
|
||||
|
||||
A future Loop C PR is acceptable only with:
|
||||
|
||||
- a benchmark adapter and a held-out task set, independent of the tasks used to
|
||||
propose the edit;
|
||||
- a baseline run and a candidate run with the same model, seed, budget, and
|
||||
judge;
|
||||
- the judge independent of the subject model, with a falsification battery
|
||||
(the existing L3 runner already enforces this);
|
||||
- per-gate results with denominators, not a single aggregate score;
|
||||
- a written rollback path and the flag name.
|
||||
|
||||
## What the current layers do
|
||||
|
||||
- `GET /api/tasks/{id}/outcome` (PR #102) records what a task produced and what
|
||||
it cost: findings by disposition and severity, retest verdicts, tokens, tool
|
||||
calls, skill misses, wall time.
|
||||
- `ARTEX_RSI_LESSONS=off|shadow|inject` (PR #104) extracts evidence-gated
|
||||
lessons from terminal tasks and, in `inject` mode, appends a code-owned
|
||||
data-not-instructions block after the planner and worker system prompts.
|
||||
Status stays `candidate`.
|
||||
- `GET /api/skill-gaps` (PR #106) turns the existing miss ledger into a
|
||||
capability backlog.
|
||||
|
||||
None of these changes an agent's instructions on its own. Shadow mode exists to
|
||||
measure whether retrieval helps before any harness edit is attempted.
|
||||
|
||||
## Credits
|
||||
|
||||
The admission rule (no lesson without cue, action, and evidence) and the
|
||||
propose/probe/compose shape come from:
|
||||
|
||||
> Harness^2: Recursive Agent Harnessing for an Open World.
|
||||
> Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming,
|
||||
> Han Yu, Rujun Han, Chen-Yu Lee, Tomas Pfister. arXiv:2610.14224, 2026.
|
||||
> Google Research. Apache-2.0. https://github.com/google-research/harness2
|
||||
|
||||
The evidence taxonomy used in the design (task gain, retained gain, improver
|
||||
gain) and the implementation pointers come from:
|
||||
|
||||
> awesome-rsi, lobehub. https://github.com/lobehub/awesome-rsi
|
||||
|
||||
Specific ideas referenced above:
|
||||
|
||||
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses,
|
||||
arXiv:2609.24972 (annealed edit budget, critic and pruner).
|
||||
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement,
|
||||
arXiv:2609.14857 (module-level fault localization from success/failure
|
||||
contrast).
|
||||
- SEABench: Benchmarking Endogenous Misalignment in Self-Evolving Agents,
|
||||
arXiv:2609.35596 (paired evolving/non-evolving attribution).
|
||||
- Agentic Harness Engineering: Observability-Driven Automatic Evolution of
|
||||
Coding-Agent Harnesses, arXiv:2604.25850 (observable edits with predictions
|
||||
verified later, the same discipline as the benchmark gate here).
|
||||
- A Safety in Self-Evolving Agents survey, arXiv:2610.00093 (why properties
|
||||
must survive accumulation and cross-context reuse, not just entry screening).
|
||||
|
||||
Only ideas are borrowed. ARTEX code is independent Go and Python; no source
|
||||
from harness2 or the referenced projects is copied.
|
||||
@@ -36,7 +36,6 @@ FLAG_RE = re.compile(r"(?:FLAG\{[0-9a-f]{64}\}|flag\{[0-9a-f-]{36}\})")
|
||||
CHALLENGE_RE = re.compile(r"DBEN-v1-\d{3}")
|
||||
|
||||
APP_SERVICE = "app"
|
||||
APP_CONTAINER_PORT = "8080"
|
||||
|
||||
|
||||
def random_flag() -> str:
|
||||
@@ -80,13 +79,33 @@ def compose_project(challenge: str, run_id: str) -> str:
|
||||
|
||||
|
||||
def host_port(output: str) -> int:
|
||||
"""Parse the host port out of `docker compose port app 8080` output."""
|
||||
"""Parse the host port out of `docker compose port` output."""
|
||||
line = output.strip().splitlines()[-1] if output.strip() else ""
|
||||
if ":" not in line:
|
||||
raise SystemExit(f"could not parse a host port from docker compose output: {output!r}")
|
||||
return int(line.rsplit(":", 1)[1])
|
||||
|
||||
|
||||
def container_host_port(project: str, cdir: pathlib.Path) -> int:
|
||||
"""Host port published by the app service, whatever container port the
|
||||
challenge declares (most challenges use 8080, some use 5000: never assume)."""
|
||||
res = _compose(project, cdir, ["ps", "-q", APP_SERVICE], timeout=60)
|
||||
cid = res.stdout.strip().splitlines()[0] if res.stdout.strip() else ""
|
||||
if not cid:
|
||||
raise SystemExit(f"no running {APP_SERVICE} container for project {project}")
|
||||
insp = subprocess.run(
|
||||
["docker", "inspect", "--format",
|
||||
"{{range $p, $conf := .NetworkSettings.Ports}}{{(index $conf 0).HostPort}}\n{{end}}",
|
||||
cid],
|
||||
capture_output=True, text=True, timeout=60)
|
||||
for token in insp.stdout.split():
|
||||
if token.isdigit():
|
||||
return int(token)
|
||||
raise SystemExit(
|
||||
f"no published host port on container {cid} "
|
||||
f"({insp.stderr.strip()[:200] or 'inspect returned no port mapping'})")
|
||||
|
||||
|
||||
def derive_ground_truth(repo: str, challenge: str) -> dict:
|
||||
"""Ground truth from the challenge's own benchmark.json.
|
||||
|
||||
@@ -168,8 +187,7 @@ def deploy(repo: str, challenge: str, flag: str, run_id: str) -> dict:
|
||||
if up.returncode != 0:
|
||||
print("[drost] up --wait failed; falling back to up -d and polling")
|
||||
_compose(project, cdir, ["up", "-d"], timeout=900)
|
||||
res = _compose(project, cdir, ["port", APP_SERVICE, APP_CONTAINER_PORT], timeout=60)
|
||||
url = f"http://127.0.0.1:{host_port(res.stdout)}"
|
||||
url = f"http://127.0.0.1:{container_host_port(project, cdir)}"
|
||||
for _ in range(60):
|
||||
try:
|
||||
urllib.request.urlopen(url, timeout=5)
|
||||
|
||||
@@ -189,18 +189,21 @@ def fetch_findings(api: Api, tid) -> list[dict]:
|
||||
out = []
|
||||
for f in items:
|
||||
fid = f.get("id") or f.get("node_id")
|
||||
# The detail route keys on finding_id (the findings-table row), not on
|
||||
# the exploration node id the list also carries; asking with the node
|
||||
# id 404s. Fall back to the node id so older payloads still resolve.
|
||||
did = f.get("finding_id") or fid
|
||||
detail = {}
|
||||
try:
|
||||
detail = api.get(f"/api/exploration/findings/{fid}")
|
||||
except SystemExit:
|
||||
raise
|
||||
except Exception:
|
||||
pass
|
||||
detail = api.get(f"/api/exploration/findings/{did}")
|
||||
except (SystemExit, Exception):
|
||||
pass # a missing detail must not kill the run; list fields remain
|
||||
report = detail.get("report") or f.get("report") or ""
|
||||
summary = f.get("summary") or detail.get("summary") or ""
|
||||
name = f.get("name") or f.get("title") or summary[:80]
|
||||
out.append({
|
||||
"id": fid,
|
||||
"finding_id": str(f.get("finding_id") or ""),
|
||||
"name": str(name)[:200],
|
||||
"severity": str(f.get("severity") or detail.get("severity") or "?"),
|
||||
"summary": str(summary)[:600],
|
||||
@@ -217,13 +220,14 @@ def deep_evidence_texts(api: Api, tid, findings: list[dict]) -> list[str]:
|
||||
common path."""
|
||||
texts = []
|
||||
for f in findings:
|
||||
did = f.get("finding_id") or f.get("id")
|
||||
try:
|
||||
texts.append(json.dumps(api.get(f"/api/exploration/findings/{f['id']}")))
|
||||
except Exception:
|
||||
texts.append(json.dumps(api.get(f"/api/exploration/findings/{did}")))
|
||||
except (SystemExit, Exception):
|
||||
continue
|
||||
try:
|
||||
texts.append(json.dumps(api.get(f"/api/exploration/nodes?task={tid}&kind=fact&size=200")))
|
||||
except Exception:
|
||||
except (SystemExit, Exception):
|
||||
pass
|
||||
try:
|
||||
r = urllib.request.Request(f"{api.base}/api/report?task={tid}")
|
||||
@@ -468,72 +472,83 @@ def main() -> None:
|
||||
gt = json.loads(gt_path.read_text())
|
||||
gt_findings = gt["findings"]
|
||||
|
||||
dep = deploy_corpus(args)
|
||||
corpus_url = dep["url"]
|
||||
flag = dep.get("flag")
|
||||
if args.deploy_only:
|
||||
print(f"[drost] deploy-only: up at {corpus_url}, flag {flag}")
|
||||
print(f"[drost] stop it later with: docker compose -p {dep['project']} down --remove-orphans"
|
||||
f" (cwd {dep['challenge_dir']})")
|
||||
return
|
||||
|
||||
task_url = corpus_url
|
||||
if args.target_host != "127.0.0.1":
|
||||
task_url = f"http://{args.target_host}:{corpus_url.rsplit(':', 1)[1]}"
|
||||
print(f"[corpus] task target rewritten for ARTEX reachability: {task_url}")
|
||||
|
||||
api = Api(args.artex_url, token)
|
||||
task_info = create_task(api, args, task_url, gt)
|
||||
status = wait_task(api, task_info["id"], args.task_timeout + 1800)
|
||||
findings = fetch_findings(api, task_info["id"])
|
||||
|
||||
invalid_ids = [f["id"] for f in findings if not (f["summary"].strip() or f["report"].strip())]
|
||||
usable = [f for f in findings if f["id"] not in invalid_ids]
|
||||
|
||||
flag_captured = None
|
||||
if flag:
|
||||
# Objective oracle first pass over the already-fetched fields; the deep
|
||||
# scan (full details, facts, report) runs only before declaring a miss.
|
||||
flag_captured = drost.flag_in_texts(flag, drost.finding_texts(findings))
|
||||
if not flag_captured:
|
||||
flag_captured = drost.flag_in_texts(flag, deep_evidence_texts(api, task_info["id"], findings))
|
||||
print(f"[flag] captured: {flag_captured}")
|
||||
|
||||
if judge_enabled:
|
||||
if not usable:
|
||||
print("[judge] no usable findings; scoring zeros")
|
||||
verdicts = [{"gt_id": g["id"], "detected": False, "matched_finding_id": None,
|
||||
"gates": {"component": False, "class": False, "keywords": False, "proof": False},
|
||||
"reason": "no usable scanner findings"} for g in gt_findings]
|
||||
else:
|
||||
print("[judge] running falsification battery")
|
||||
judge_battery(args)
|
||||
print("[judge] grading")
|
||||
verdicts = run_judge(args, gt_findings, usable)
|
||||
result = tally(gt_findings, verdicts, findings, invalid_ids)
|
||||
else:
|
||||
print("[judge] flag-only run: no judge configured, analysis gates not measured")
|
||||
verdicts = None
|
||||
result = {
|
||||
"mode": "flag-only",
|
||||
"scanner_findings_total": len(findings),
|
||||
"invalid_findings_excluded": invalid_ids,
|
||||
"discovery_candidates": [{"id": f["id"], "name": f["name"]} for f in usable],
|
||||
"note": "analysis gates not measured: no judge configured",
|
||||
}
|
||||
if flag:
|
||||
result["flag_captured"] = flag_captured
|
||||
result["flag_used"] = flag
|
||||
|
||||
out_dir = pathlib.Path("benchmark-results") / args.corpus / args.run_id
|
||||
write_report(out_dir, args.corpus, args, task_info, status, findings, verdicts, result)
|
||||
print(json.dumps(result, indent=2)[:1500])
|
||||
|
||||
# drost project/dir are deterministic, so they are known before deploy and
|
||||
# the finally below can clean up even when the deploy itself fails halfway.
|
||||
drost_project = drost_cdir = None
|
||||
if args.corpus_type == "drost":
|
||||
if not args.keep:
|
||||
drost.teardown(dep["project"], dep["challenge_dir"])
|
||||
elif not (args.keep or args.corpus_url):
|
||||
subprocess.run(["docker", "rm", "-f", f"bench-{args.corpus}-{args.run_id}"], capture_output=True)
|
||||
drost_project = drost.compose_project(args.drost_id, args.run_id)
|
||||
drost_cdir = str(drost.challenge_dir(args.drost_repo, args.drost_id))
|
||||
|
||||
try:
|
||||
dep = deploy_corpus(args)
|
||||
corpus_url = dep["url"]
|
||||
flag = dep.get("flag")
|
||||
if args.deploy_only:
|
||||
print(f"[drost] deploy-only: up at {corpus_url}, flag {flag}")
|
||||
print(f"[drost] stop it later with: docker compose -p {dep['project']} down --remove-orphans"
|
||||
f" (cwd {dep['challenge_dir']})")
|
||||
return
|
||||
|
||||
task_url = corpus_url
|
||||
if args.target_host != "127.0.0.1":
|
||||
task_url = f"http://{args.target_host}:{corpus_url.rsplit(':', 1)[1]}"
|
||||
print(f"[corpus] task target rewritten for ARTEX reachability: {task_url}")
|
||||
|
||||
api = Api(args.artex_url, token)
|
||||
task_info = create_task(api, args, task_url, gt)
|
||||
status = wait_task(api, task_info["id"], args.task_timeout + 1800)
|
||||
findings = fetch_findings(api, task_info["id"])
|
||||
|
||||
invalid_ids = [f["id"] for f in findings if not (f["summary"].strip() or f["report"].strip())]
|
||||
usable = [f for f in findings if f["id"] not in invalid_ids]
|
||||
|
||||
flag_captured = None
|
||||
if flag:
|
||||
# Objective oracle first pass over the already-fetched fields; the deep
|
||||
# scan (full details, facts, report) runs only before declaring a miss.
|
||||
flag_captured = drost.flag_in_texts(flag, drost.finding_texts(findings))
|
||||
if not flag_captured:
|
||||
flag_captured = drost.flag_in_texts(flag, deep_evidence_texts(api, task_info["id"], findings))
|
||||
print(f"[flag] captured: {flag_captured}")
|
||||
|
||||
if judge_enabled:
|
||||
if not usable:
|
||||
print("[judge] no usable findings; scoring zeros")
|
||||
verdicts = [{"gt_id": g["id"], "detected": False, "matched_finding_id": None,
|
||||
"gates": {"component": False, "class": False, "keywords": False, "proof": False},
|
||||
"reason": "no usable scanner findings"} for g in gt_findings]
|
||||
else:
|
||||
print("[judge] running falsification battery")
|
||||
judge_battery(args)
|
||||
print("[judge] grading")
|
||||
verdicts = run_judge(args, gt_findings, usable)
|
||||
result = tally(gt_findings, verdicts, findings, invalid_ids)
|
||||
else:
|
||||
print("[judge] flag-only run: no judge configured, analysis gates not measured")
|
||||
verdicts = None
|
||||
result = {
|
||||
"mode": "flag-only",
|
||||
"scanner_findings_total": len(findings),
|
||||
"invalid_findings_excluded": invalid_ids,
|
||||
"discovery_candidates": [{"id": f["id"], "name": f["name"]} for f in usable],
|
||||
"note": "analysis gates not measured: no judge configured",
|
||||
}
|
||||
if flag:
|
||||
result["flag_captured"] = flag_captured
|
||||
result["flag_used"] = flag
|
||||
|
||||
out_dir = pathlib.Path("benchmark-results") / args.corpus / args.run_id
|
||||
write_report(out_dir, args.corpus, args, task_info, status, findings, verdicts, result)
|
||||
print(json.dumps(result, indent=2)[:1500])
|
||||
finally:
|
||||
# The corpus is torn down even when a step fails halfway: a crash in
|
||||
# deploy, scoring or reporting must not leave the challenge running
|
||||
# (--keep and --deploy-only opt out deliberately).
|
||||
if args.corpus_type == "drost":
|
||||
if not (args.keep or args.deploy_only):
|
||||
drost.teardown(drost_project, drost_cdir)
|
||||
elif not (args.keep or args.corpus_url):
|
||||
subprocess.run(["docker", "rm", "-f", f"bench-{args.corpus}-{args.run_id}"], capture_output=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+33
-18
@@ -1044,34 +1044,44 @@ func (s *Server) listTasks(w http.ResponseWriter, r *http.Request) {
|
||||
active = t.ID
|
||||
}
|
||||
list := s.m.List()
|
||||
metrics, _ := s.m.PG().TaskListMetricsAll()
|
||||
metrics, err := s.m.PG().TaskListMetricsAll()
|
||||
if err != nil {
|
||||
log.Printf("[tasks] list metrics query failed — metrics zeroed: %v", err)
|
||||
}
|
||||
archiveBlockers, _ := s.m.PG().TaskArchiveBlockers()
|
||||
dtos := make([]TaskDTO, 0, len(list))
|
||||
for _, t := range list {
|
||||
dto := taskDTO(t, s.resolvedTaskStatus(t))
|
||||
applyTaskArchiveBlocker(&dto, archiveBlockers)
|
||||
metric := metrics[t.ExpID]
|
||||
dto.Tokens = tokenTotalDTO(metric.Tokens)
|
||||
// prefer the live in-memory heartbeat (fresher) and fall back to the
|
||||
// persisted max activity time (survives restarts) for run-duration display.
|
||||
dto.LastActivity = metric.LastActivity
|
||||
if live := s.engine.LastActivity(t.ID); live > dto.LastActivity {
|
||||
dto.LastActivity = live
|
||||
}
|
||||
dto.GoalsTotal = metric.Goals.Total
|
||||
dto.GoalsMet = metric.Goals.Met
|
||||
dto.InFlight = metric.RunningIntents
|
||||
dto.Findings = FindingSeverityDTO{
|
||||
Critical: metric.Findings.Critical,
|
||||
High: metric.Findings.High,
|
||||
Medium: metric.Findings.Medium,
|
||||
Low: metric.Findings.Low,
|
||||
}
|
||||
s.applyTaskMetrics(&dto, t, metrics[t.ExpID])
|
||||
dtos = append(dtos, dto)
|
||||
}
|
||||
writeJSON(w, 200, map[string]any{"tasks": dtos, "active": active})
|
||||
}
|
||||
|
||||
// applyTaskMetrics fills the aggregate DTO fields (tokens, goal progress,
|
||||
// running intents, findings, last activity) from the list-metrics query so the
|
||||
// detail and list endpoints report identical numbers. A zero metric leaves the
|
||||
// DTO zeroed, which is why callers log the query error instead of assuming it.
|
||||
func (s *Server) applyTaskMetrics(dto *TaskDTO, t *Task, metric db.TaskListMetrics) {
|
||||
dto.Tokens = tokenTotalDTO(metric.Tokens)
|
||||
// prefer the live in-memory heartbeat (fresher) and fall back to the
|
||||
// persisted max activity time (survives restarts) for run-duration display.
|
||||
dto.LastActivity = metric.LastActivity
|
||||
if live := s.engine.LastActivity(t.ID); live > dto.LastActivity {
|
||||
dto.LastActivity = live
|
||||
}
|
||||
dto.GoalsTotal = metric.Goals.Total
|
||||
dto.GoalsMet = metric.Goals.Met
|
||||
dto.InFlight = metric.RunningIntents
|
||||
dto.Findings = FindingSeverityDTO{
|
||||
Critical: metric.Findings.Critical,
|
||||
High: metric.Findings.High,
|
||||
Medium: metric.Findings.Medium,
|
||||
Low: metric.Findings.Low,
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) resolvedTaskStatus(t *Task) string {
|
||||
if t == nil {
|
||||
return "created"
|
||||
@@ -1798,6 +1808,11 @@ func (s *Server) getTask(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
dto := taskDTO(t, s.resolvedTaskStatus(t))
|
||||
metrics, err := s.m.PG().TaskListMetricsAll()
|
||||
if err != nil {
|
||||
log.Printf("[tasks] detail metrics query failed for %s — metrics zeroed: %v", t.ID, err)
|
||||
}
|
||||
s.applyTaskMetrics(&dto, t, metrics[t.ExpID])
|
||||
archiveBlockers, _ := s.m.PG().TaskArchiveBlockers()
|
||||
applyTaskArchiveBlocker(&dto, archiveBlockers)
|
||||
writeJSON(w, 200, dto)
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTaskDetailReturnsListMetrics(t *testing.T) {
|
||||
m, err := NewManager(t.TempDir(), "")
|
||||
if err != nil {
|
||||
t.Skipf("postgres unavailable (%v) — skipping", err)
|
||||
}
|
||||
defer m.Close()
|
||||
task, err := m.CreateTask("detail metrics", "goal", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
taskID, _ := strconv.ParseInt(task.ID, 10, 64)
|
||||
defer func() { _ = m.pg.DeleteTask(taskID) }()
|
||||
|
||||
// Seed the aggregates the metrics query reads: one result-activity with
|
||||
// tokens, one non-result row that must be excluded, and two goals (1 met).
|
||||
if _, err := m.pg.Exec(`
|
||||
INSERT INTO activity(
|
||||
exploration_id, kind, input_tokens, output_tokens,
|
||||
cache_read_tokens, cache_write_tokens, created_at
|
||||
) VALUES ($1,'result',11,7,3,2,now()), ($1,'tool_result',999,999,999,999,now())`,
|
||||
task.ExpID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := m.pg.Exec(`
|
||||
INSERT INTO exploration_nodes(exploration_id, kind, payload, state)
|
||||
VALUES ($1,'goal','{}','met'), ($1,'goal','{}','open')`, task.ExpID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
s := New(context.Background(), m, t.TempDir(), t.TempDir(), t.TempDir())
|
||||
token, err := signJWT(s.jwtKey)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var probe struct {
|
||||
Tokens struct {
|
||||
InputTokens int `json:"input_tokens"`
|
||||
OutputTokens int `json:"output_tokens"`
|
||||
CacheReadTokens int `json:"cache_read_tokens"`
|
||||
CacheWriteTokens int `json:"cache_write_tokens"`
|
||||
} `json:"tokens"`
|
||||
GoalsMet int `json:"goals_met"`
|
||||
GoalsTotal int `json:"goals_total"`
|
||||
}
|
||||
req := httptest.NewRequest(http.MethodGet, "/api/tasks/"+task.ID, nil)
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
rec := httptest.NewRecorder()
|
||||
s.Handler().ServeHTTP(rec, req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("detail status=%d body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &probe); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if probe.Tokens.InputTokens != 11 || probe.Tokens.OutputTokens != 7 ||
|
||||
probe.Tokens.CacheReadTokens != 3 || probe.Tokens.CacheWriteTokens != 2 {
|
||||
t.Fatalf("detail tokens not filled from metrics: %+v", probe.Tokens)
|
||||
}
|
||||
if probe.GoalsMet != 1 || probe.GoalsTotal != 2 {
|
||||
t.Fatalf("detail goal progress not filled from metrics: %d/%d", probe.GoalsMet, probe.GoalsTotal)
|
||||
}
|
||||
|
||||
// The list endpoint must report the same numbers for the same task.
|
||||
listReq := httptest.NewRequest(http.MethodGet, "/api/tasks", nil)
|
||||
listReq.Header.Set("Authorization", "Bearer "+token)
|
||||
listRec := httptest.NewRecorder()
|
||||
s.Handler().ServeHTTP(listRec, listReq)
|
||||
if listRec.Code != http.StatusOK {
|
||||
t.Fatalf("list status=%d body=%s", listRec.Code, listRec.Body.String())
|
||||
}
|
||||
var listed struct {
|
||||
Tasks []struct {
|
||||
ID string `json:"id"`
|
||||
Tokens struct {
|
||||
InputTokens int `json:"input_tokens"`
|
||||
OutputTokens int `json:"output_tokens"`
|
||||
} `json:"tokens"`
|
||||
GoalsMet int `json:"goals_met"`
|
||||
GoalsTotal int `json:"goals_total"`
|
||||
} `json:"tasks"`
|
||||
}
|
||||
if err := json.Unmarshal(listRec.Body.Bytes(), &listed); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
found := false
|
||||
for _, lt := range listed.Tasks {
|
||||
if lt.ID != task.ID {
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
if lt.Tokens.InputTokens != probe.Tokens.InputTokens ||
|
||||
lt.Tokens.OutputTokens != probe.Tokens.OutputTokens ||
|
||||
lt.GoalsMet != probe.GoalsMet || lt.GoalsTotal != probe.GoalsTotal {
|
||||
t.Fatalf("list and detail metrics disagree: list=%+v/%d detail=%+v/%d",
|
||||
lt.Tokens, lt.GoalsMet, probe.Tokens, probe.GoalsTotal)
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("task %s missing from list response", task.ID)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user