BaaRai
7644ae2a2f
Strict vulnerability programs reject, and sometimes ban, reports that miss their published criteria: wrong class, PoC not written from a remote attacker's perspective, no measured impact, same-type findings split across reports, untested claims presented as verified. Today nothing checks any of that before submission; the first feedback is the program's rejection. scripts/submission_review adds the validation side of the loop: - review.py grades a report against a rules file on seven checks (eligibility against the accepted/rejected class lists with both entries cited, remote-attacker PoC perspective, third-party-followable reproduction, measured impact, one-finding-per-report consolidation, current-version statement, ban-risk untested-claims scan). Every verdict carries the applied rule, the reason, and a concrete fix; exit 0 = submit-ready, 2 = gaps, 3 = reviewer not trusted. - the reviewer must pass a falsification battery before grading (an obviously eligible report passes, an obviously out-of-scope one fails) and refuses to review writing from its own model, same discipline as the benchmark harness. - rules/strict-vdp.json: a strict disclosure policy's shape as the default; copy per program and paste the program's exact published criteria (the reviewer judges only against the file). Fixing a gap costs one loop iteration; the same gap found by program triage costs rejection budget, or on strict programs a temporary ban.
Pre-submission report review
The submission-validation loop: review a vulnerability report against a program's eligibility rules BEFORE submitting, fix the gaps, re-review, submit only on SUBMIT-READY. Catches locally what the program's triage would catch later, while fixing it is still free.
Uses the same measurement discipline as the benchmark harness: the reviewer is an independent LLM that must pass a falsification battery before it is allowed to grade (an obviously eligible report must pass, an obviously out-of-scope one must fail), it refuses to review a report written by its own model, and every verdict cites the rule it applied plus a concrete fix.
The loop
finding + report
|
v
review.py --report report.md --rules rules/<program>.json
|
SUBMIT-READY? --no--> fix the cited gaps --> re-run
|
yes
v
submit
Exit codes: 0 submit-ready, 2 gaps found, 3 reviewer not trusted
(battery failed or configuration error).
Checks
| Check | What it enforces |
|---|---|
| eligibility | class is in the rules' accepted list and not in the rejected list (cites both) |
| poc_remote | reproduction is a remote attacker over HTTP, not server-side access |
| reproduction | a third party can follow it; requests complete; expected vs actual stated |
| impact_measured | quantified with denominator or a demonstrated outcome, not adjectives |
| consolidation | one finding per report; same-type findings consolidated |
| version_current | component + exact version + latest-at-time stated |
| ban_risk | no untested claims presented as verified |
Usage
REVIEW_API=https://api.example.com REVIEW_API_KEY=... REVIEW_MODEL=<reviewer> \
python3 review.py --report my-finding.md --rules rules/strict-vdp.json \
--subject-model <model that wrote the report>
--subject-model exists so a model never reviews its own writing (pass
--allow-self-review to override knowingly). Verdicts land next to the report
as <report>.review.json with every check, the applied rule, the reason, and
the suggested fix.
Rule files
rules/strict-vdp.json is the default: a strict disclosure policy's shape
(accepted classes with chaining conditions, rejected classes, format
requirements, risk notes). Copy it per program and paste the program's exact
published criteria — the reviewer judges only against what the file says, so
fidelity to the real policy is the file's job, not the reviewer's.
Limits
The reviewer is an LLM and can misjudge; every verdict carries its reason and the rule text, so audit before trusting a pass. A single pass is one opinion: for high-stakes submissions, re-run after every edit (the loop is cheap; a rejection is not).