mirror of
https://github.com/BaaRai/ARTEX.git
synced 2026-11-13 11:40:53 +08:00
BaaRai
bb81979bf6
The worker and pentest seeds already tell the agent to self-check
conclusions, but the check has no concrete axes and no rules for what
an elimination requires. In practice candidates get dropped on vibes
("probably validated downstream") or merged into one narrative finding,
and there is pressure to produce at least one finding per intent.
Changes, all in the default prompt seeds plus a guard test:
- worker and pentest seeds gain an "Adversarial gate before
report_finding" block: actively try to disprove every candidate along
five named axes (reachability, preconditions, downstream controls,
scope, simpler explanation); eliminations need cited evidence at the
same bar as confirmations; distinct entry points, parameters, or
privilege levels are separate findings (anti-merge); zero confirmed
findings is a valid outcome and precise negatives are real work.
- the retester seed now argues the opposite case before every verdict
along the same five axes and must cite the specific requests,
responses, or controls supporting reproduced / fixed / inconclusive
alike; a prose assumption never substitutes for evidence.
- a seed test guards all markers against silent removal.
Methodology adapted from VulnHunter (Apache-2.0, adversarial
verification and anti-merge rules) and bug-bounty-hunting-prompts
(MIT, independent pre-confirmation disprove pass).
35 lines
1.1 KiB
Go
35 lines
1.1 KiB
Go
package agent
|
|
|
|
import "strings"
|
|
import "testing"
|
|
|
|
// The adversarial gate (disprove axes, evidence-cited eliminations, anti-merge,
|
|
// zero-findings-valid) must stay in the executor agents' default seeds, and the
|
|
// retester must stay an evidence-cited adversary. Methodology adapted from
|
|
// VulnHunter (Apache-2.0) and bug-bounty-hunting-prompts (MIT).
|
|
func TestPromptSeedsCarryAdversarialGate(t *testing.T) {
|
|
for name, tmpl := range BuiltinPromptSeeds() {
|
|
switch name {
|
|
case "worker", "pentest":
|
|
for _, marker := range []string{
|
|
"actively tried to disprove it",
|
|
"Eliminations need evidence at the same bar as confirmations",
|
|
"Do not merge distinct candidates",
|
|
"Zero confirmed findings is a valid outcome",
|
|
} {
|
|
if !strings.Contains(tmpl, marker) {
|
|
t.Errorf("%s seed missing adversarial-gate marker %q", name, marker)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
for _, marker := range []string{
|
|
"argue the opposite case",
|
|
"cites the specific requests, responses, or controls",
|
|
} {
|
|
if !strings.Contains(RetesterDefaultPrompt, marker) {
|
|
t.Errorf("retester prompt missing adversarial marker %q", marker)
|
|
}
|
|
}
|
|
}
|