mirror of
https://github.com/BaaRai/ARTEX.git
synced 2026-11-13 11:40:53 +08:00
BaaRai
86a4efc85e
Model benchmarks measure recognition on snippets. Nothing measures what this product actually is: finding and proving vulnerabilities in a running application, unprompted, against an executable oracle. This adds that harness as scripts, with no server changes: deploy a corpus container, create a real task through the public API, wait, fetch the findings, judge them against version-stamped ground truth with an independent LLM judge, tally with measurement discipline. scripts/benchmark/run_bench.py plus ground_truth/juice-shop.json (8 stable entries: SQLi login bypass, admin section access, basket IDOR, /ftp exposure, forged feedback, error disclosure, token tampering, XSS) and a README documenting the standing rules: - multi-gate scoring (component 1 / class 2 / keywords 1 / proof 1), partial credit, gate pattern over aggregate - findings beyond ground truth are discovery candidates; precision is reported as bounds until verified - the judge must not be the model under test (refuses to start, override knowingly), and must pass a falsification battery before grading - invalid findings (errored or empty) are excluded from every rate and named, never counted as misses - every rate carries its denominator, the aggregate a Wilson 95% interval; results are stamped with method + ground-truth versions The task goal bakes in the standing quality gates, so the benchmark measures the product as it behaves, prompt rules included. Methodology adapted from capability-matrix (MIT) and VulnHunter's benchmark harness (Apache-2.0).