--llm-profiles passes llm_profile_ids to the created task (head first)
so a benchmark run exercises the model chain it claims to measure, and
the chain is stamped into results.json and the report.
--target-host rewrites the host in the task goal and company scope when
ARTEX runs in a container and the corpus runs on the host: the task must
reach the challenge via the host gateway (172.17.0.1), the same pattern
as the MCP server URLs. Local reachability checks still use 127.0.0.1.
DrostBench v1 ships 100 docker-compose web challenges, each with one
flag-bearing vulnerability plus decoys that do not reach the flag, so
flag capture is an objective win condition that no judge decides.
scripts/benchmark/drost.py drives one challenge: compose build with an
explicit per-build random flag (the suite's deterministic sha256 default
is never used, so a memorized flag cannot be replayed), up --wait, host
port discovery, down --rmi local. Ground truth is derived from the
challenge benchmark.json; the flag is scanned over the fetched findings
first and over full details, facts and the report only before declaring
a miss.
run_bench.py gains --corpus-type drost (one challenge per invocation),
flag capture in every drost report, a flag-only mode when no judge is
configured (gates reported as not measured, never as zero), and
--deploy-only for install validation. Image mode is unchanged.
test_drost.py covers the pure functions (flag formats, challenge id
validation, port parsing, ground-truth derivation incl. its refusals);
the lifecycle was exercised end to end on DBEN-v1-001: build with a
random flag, reachable app, compiled flag matching the override,
teardown without leftovers.
Closes#94
Model benchmarks measure recognition on snippets. Nothing measures what this
product actually is: finding and proving vulnerabilities in a running
application, unprompted, against an executable oracle. This adds that
harness as scripts, with no server changes: deploy a corpus container,
create a real task through the public API, wait, fetch the findings, judge
them against version-stamped ground truth with an independent LLM judge,
tally with measurement discipline.
scripts/benchmark/run_bench.py plus ground_truth/juice-shop.json (8 stable
entries: SQLi login bypass, admin section access, basket IDOR, /ftp
exposure, forged feedback, error disclosure, token tampering, XSS) and a
README documenting the standing rules:
- multi-gate scoring (component 1 / class 2 / keywords 1 / proof 1),
partial credit, gate pattern over aggregate
- findings beyond ground truth are discovery candidates; precision is
reported as bounds until verified
- the judge must not be the model under test (refuses to start, override
knowingly), and must pass a falsification battery before grading
- invalid findings (errored or empty) are excluded from every rate and
named, never counted as misses
- every rate carries its denominator, the aggregate a Wilson 95% interval;
results are stamped with method + ground-truth versions
The task goal bakes in the standing quality gates, so the benchmark
measures the product as it behaves, prompt rules included.
Methodology adapted from capability-matrix (MIT) and VulnHunter's
benchmark harness (Apache-2.0).
BaaRai