Model benchmarks measure recognition on snippets. Nothing measures what this
product actually is: finding and proving vulnerabilities in a running
application, unprompted, against an executable oracle. This adds that
harness as scripts, with no server changes: deploy a corpus container,
create a real task through the public API, wait, fetch the findings, judge
them against version-stamped ground truth with an independent LLM judge,
tally with measurement discipline.
scripts/benchmark/run_bench.py plus ground_truth/juice-shop.json (8 stable
entries: SQLi login bypass, admin section access, basket IDOR, /ftp
exposure, forged feedback, error disclosure, token tampering, XSS) and a
README documenting the standing rules:
- multi-gate scoring (component 1 / class 2 / keywords 1 / proof 1),
partial credit, gate pattern over aggregate
- findings beyond ground truth are discovery candidates; precision is
reported as bounds until verified
- the judge must not be the model under test (refuses to start, override
knowingly), and must pass a falsification battery before grading
- invalid findings (errored or empty) are excluded from every rate and
named, never counted as misses
- every rate carries its denominator, the aggregate a Wilson 95% interval;
results are stamped with method + ground-truth versions
The task goal bakes in the standing quality gates, so the benchmark
measures the product as it behaves, prompt rules included.
Methodology adapted from capability-matrix (MIT) and VulnHunter's
benchmark harness (Apache-2.0).
