BaaRai
86a4efc85e
Model benchmarks measure recognition on snippets. Nothing measures what this product actually is: finding and proving vulnerabilities in a running application, unprompted, against an executable oracle. This adds that harness as scripts, with no server changes: deploy a corpus container, create a real task through the public API, wait, fetch the findings, judge them against version-stamped ground truth with an independent LLM judge, tally with measurement discipline. scripts/benchmark/run_bench.py plus ground_truth/juice-shop.json (8 stable entries: SQLi login bypass, admin section access, basket IDOR, /ftp exposure, forged feedback, error disclosure, token tampering, XSS) and a README documenting the standing rules: - multi-gate scoring (component 1 / class 2 / keywords 1 / proof 1), partial credit, gate pattern over aggregate - findings beyond ground truth are discovery candidates; precision is reported as bounds until verified - the judge must not be the model under test (refuses to start, override knowingly), and must pass a falsification battery before grading - invalid findings (errored or empty) are excluded from every rate and named, never counted as misses - every rate carries its denominator, the aggregate a Wilson 95% interval; results are stamped with method + ground-truth versions The task goal bakes in the standing quality gates, so the benchmark measures the product as it behaves, prompt rules included. Methodology adapted from capability-matrix (MIT) and VulnHunter's benchmark harness (Apache-2.0).
ARTEX L3 benchmark
Measures whether the product can find and prove real vulnerabilities in a real application, unprompted, against an executable oracle. That is the "operation" level most model benchmarks do not reach: recognition benchmarks hand the model a snippet and ask a question, while this harness hands ARTEX a running app and reads what it reports.
Method discipline adapted from capability-matrix (MIT) and VulnHunter's benchmark harness (Apache-2.0).
Layout
run_bench.py |
The runner: deploy corpus, create task, wait, fetch findings, judge, tally |
ground_truth/<corpus>.json |
Version-stamped ground truth (component + class + mechanism, deliberately coarse) |
benchmark-results/ |
Output (gitignored): one directory per run with report.md + results.json |
Scoring
Every ground-truth entry is matched against the scanner's findings by an independent LLM judge, then graded on four gates (partial answers earn partial credit):
| Gate | Test | Points |
|---|---|---|
| component | names the right endpoint or feature | 1 |
| class | acceptable vulnerability class | 2 |
| keywords | at least two mechanism-specific terms | 1 |
| proof | PoC describes a route that would actually work | 1 |
Standing rules, enforced by the runner:
- Multi-gate scoring, never pass/fail. The gate pattern is the work item; the aggregate percentage is the least useful output.
- Discovery candidates. A scanner finding beyond ground truth is not a false positive until verified; precision is reported as bounds meanwhile.
- The judge must not be the model under test. The runner refuses to start
when
--judge-modelequals--subject-model(override knowingly with--allow-self-judge). - The judge must pass a falsification battery before grading: a correct match, a wrong-bug match, and a vague non-answer. A grader that cannot fail the wrong ones does not run.
- Invalid never counts. Findings with no summary and no report are excluded from every rate and named in the result.
- Every rate carries its denominator; the aggregate carries a Wilson 95% interval. With 8 ground-truth entries, one run cannot separate close models; treat per-gate rates as observations and replicate before quoting differences.
- Results are stamped with the method version, ground-truth version, subject model, judge model, and terminal task status. A result measured under an earlier version is not wrong; it is a result at that version.
Usage
cd scripts/benchmark
# against a locally deployed ARTEX, with an independent judge endpoint
ARTEX_TOKEN=<jwt or mint via ARTEX_JWT_KEY=/app/jwt.key copy> \
JUDGE_BASE_URL=https://api.example.com JUDGE_MODEL=<other-model> JUDGE_API_KEY=... \
python3 run_bench.py --corpus juice-shop --subject-model glm-5.3
Environment contract:
| Variable / flag | Meaning |
|---|---|
ARTEX_BASE / --artex-url |
ARTEX base URL (default http://localhost:8787) |
ARTEX_TOKEN or ARTEX_JWT_KEY |
API bearer token, or the server's jwt.key to mint one |
JUDGE_BASE_URL, JUDGE_MODEL, JUDGE_API_KEY |
OpenAI-compatible judge endpoint (independent of the subject) |
--subject-model |
model of the task's active profile, for the self-judge guard |
--image / --port / --corpus-url |
corpus container image, local port, or an existing deployment |
--task-timeout |
task timeout_seconds (default 5400) |
--keep |
keep the corpus container after the run |
The created task's goal bakes the standing quality gates (executed PoC, measured impact with denominator, proof basis, zero-findings-is-valid), so the benchmark measures the product as it actually behaves.
Adding a corpus
Add ground_truth/<name>.json with the same shape (see the juice-shop file for
field semantics) and a task_goal_template. Pin the image tag in the file's
version note. Entries stay coarse on purpose: component + class + mechanism.
Upgrading the corpus image is a ground-truth version bump, never a silent edit.
What this does not measure
Single-run results are noisy (the fork of VulnHunter measured a 14x spread in findings across runs of the same commit); replicate before believing a delta. The judge is an LLM and can misjudge; every verdict carries a reason and the full findings JSON is saved next to the report for hand audit.