mirror of
https://github.com/BaaRai/ARTEX.git
synced 2026-11-13 11:40:53 +08:00
BaaRai
2d8e357961
Confirmed findings today assert impact in prose ("the attacker can dump
the table") with no number behind it, and nothing in the seeds demands
that a PoC demonstrate the attacker's goal end to end rather than the
presence of a vulnerable pattern.
Three changes, all in the default prompt seeds plus a guard test:
- worker and pentest seeds gain a "Finding quality gates" block: the PoC
must show the goal achieved (data returned, command executed, another
user's resource touched); a mechanism blocked downstream is a hardening
note, not a finding; every finding carries a measured impact as a
number with its denominator plus what the defense blocked, a proof
basis line (executed live / executed lab / static unexecuted), bounded
non-destructive testing rules (own objects first, at most three
non-owned objects, smallest sufficient request, reversible values,
touched IDs recorded), and a priority rule against standalone
low-impact oracles.
- the reporter seed gains a mandatory "## Measured impact" report section
(quantified outcome, denominator, defense-blocked facts, proof basis)
and a "measured, not asserted" discipline bullet.
- a seed test guards all markers against silent removal.
Methodology adapted from VulnHunter (Apache-2.0, measured-impact and
attacker-goal doctrine) and bug-bounty-hunting-prompts (MIT, bounded
testing standard and priority discipline).
36 lines
1023 B
Go
36 lines
1023 B
Go
package agent
|
|
|
|
import "strings"
|
|
import "testing"
|
|
|
|
// The finding quality gates (attacker-goal proof standard, measured impact,
|
|
// proof basis, bounded testing) must stay in the executor agents' default
|
|
// seeds. Methodology adapted from VulnHunter (Apache-2.0) and
|
|
// bug-bounty-hunting-prompts (MIT).
|
|
func TestPromptSeedsCarryFindingQualityGates(t *testing.T) {
|
|
for name, tmpl := range BuiltinPromptSeeds() {
|
|
switch name {
|
|
case "worker", "pentest":
|
|
for _, marker := range []string{
|
|
"attacker's goal achieved end to end",
|
|
"a number with its denominator",
|
|
"State the proof basis",
|
|
"at most three non-owned objects",
|
|
} {
|
|
if !strings.Contains(tmpl, marker) {
|
|
t.Errorf("%s seed missing quality-gate marker %q", name, marker)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
for _, marker := range []string{
|
|
"## Measured impact",
|
|
"denominator",
|
|
"claim, not a measurement",
|
|
} {
|
|
if !strings.Contains(ReporterDefaultPrompt, marker) {
|
|
t.Errorf("reporter prompt missing measured-impact marker %q", marker)
|
|
}
|
|
}
|
|
}
|