Compare commits

...
Author SHA1 Message Date
BaaRai f57cf659b5 feat: planner quiet-mode skips edge rounds while work runs and goals hold
Every worker write-back fired a full planning round even when the open-goal set had not changed; on a live three-worker task that was 7 rounds in 45 minutes. Edge-triggered rounds are now dropped while active intents exist and the open-goal fingerprint recorded at the end of the last round is unchanged; heartbeat rounds, goal-set changes and frontier drains still plan immediately. Fails open on store errors.
2026-10-12 18:36:44 +02:00
BaaRai ff2e0bf338 Merge pull request #77 from BaaRai/feat/skills-reverse-skill-ports
feat(skills): port patch-diff, api-security, edr-bypass-re from reverse-skill
2026-10-12 16:34:48 +02:00
BaaRai 53bda9fe37 feat(skills): port edr-bypass-re playbook from reverse-skill (MIT)
Defense-telemetry analysis for authorized adversary simulation: EDR
monitoring surfaces, hook survey workflow, countermeasure selection,
MITRE T1562 mapping. Lab-evaluation framing with strict authorization
boundaries; requires an authorized Windows lab VM.

Adapted from zhaoxuya520/reverse-skill skills/edr-bypass-re (MIT).
Refs #76
2026-10-12 13:15:05 +02:00
BaaRai a11c2ed10c feat(skills): port api-security playbook from reverse-skill (MIT)
Ten-phase authorized API hunting methodology (BOLA/IDOR/BFLA, JWT and
OAuth, GraphQL abuse, input validation, logic, rate-limit trust
confusion, data exposure) as the hunting counterpart of api-recon.

Adapted from zhaoxuya520/reverse-skill skills/api-security (MIT).
Refs #76
2026-10-12 13:15:05 +02:00
BaaRai 04e8b37368 feat(skills): port patch-diff playbook from reverse-skill (MIT)
N-day recovery from vendor fixes: diff before/after builds, infer the
bug class from added guards, verify with a PoC that crashes only the
unpatched build. Adapted to platform rules: authorization boundaries,
standing quality gates, no simulated results, rea MCP notes.

Adapted from zhaoxuya520/reverse-skill skills/patch-diff-exploit (MIT).
Refs #76
2026-10-12 13:15:05 +02:00
BaaRai 2cdedd8eee Merge pull request #75 from BaaRai/fix/llm-test-stored-profile-endpoint
fix: llm/test stored profile fills endpoint fields from the row
2026-10-12 08:55:32 +02:00
BaaRai 2f80ffd979 Merge pull request #74 from BaaRai/fix/cost-ledger-fresh-input
fix: cost ledger charges fresh input only (cache read is part of input tokens)
2026-10-12 08:55:29 +02:00
BaaRai 6b1f04b1ba fix: llm/test stored profile fills endpoint fields from the row
The fallback only borrowed the stored key; provider, model, base_url, proxy, streaming and thinking settings still came from the request form. An empty form (the list-view test button) then hit ConfigFrom defaults, sending the profile key to the canonical provider endpoint and 401ing. Fields the form leaves empty now come from the stored profile; explicit form values still win.
2026-10-12 08:54:29 +02:00
BaaRai c96c3ea3a9 fix: cost ledger charges fresh input only (cache read is part of input tokens)
input_tokens includes the cached subset under Anthropic and OpenAI usage semantics, so pricing input and cache read at their full prices double-counted the cache. The fresh share is now input minus cache reads, clamped at zero for gateways reporting disjoint buckets.
2026-10-12 08:53:20 +02:00
BaaRai e36d0730da Merge pull request #71 from BaaRai/feat/cost-ledger
feat: cost ledger (per-profile EUR pricing on token usage)
2026-10-12 08:25:41 +02:00
BaaRai a183c4aad5 Merge pull request #69 from BaaRai/feat/submission-review
feat: pre-submission report review (validate against program rules)
2026-10-12 08:25:38 +02:00
BaaRai 67ff6ffffa feat: cost ledger (per-profile EUR pricing on token usage)
The token metering is real now (input, output, cache read/write per
profile), but it answers "how many tokens" while the question an operator
actually asks is "what did this run cost". Converting requires per-profile
prices, which nowhere exist.

Changes:
- llm_profiles gains four optional price columns (EUR per 1M tokens:
  input, output, cache read, cache write; 0/NULL = unset), migrated
  idempotently and round-tripped through SaveProfile / the profile API
  in one small auditable write
- the usage aggregation computes cost_eur per profile from the stored
  prices, in Go after the archived-task merge so cold-storage tokens are
  priced identically; profiles without prices keep cost_eur nil instead
  of showing a fake zero
- web types carry cost_eur and the price fields; the dashboard card and
  profile-dialog inputs consume them in a follow-up (the API serves the
  data already)
- integration test: priced profile + usage rows -> expected EUR to the
  cent (2M in @1.5 + 0.5M out @4 + 1M cache-read @0.15 = 5.15), plus a
  prices round-trip check

Note: the Go side activates on the next build; the schema migration is
idempotent at startup.
2026-10-12 08:20:37 +02:00
BaaRai 7644ae2a2f feat: pre-submission report review (validate against program rules)
Strict vulnerability programs reject, and sometimes ban, reports that
miss their published criteria: wrong class, PoC not written from a
remote attacker's perspective, no measured impact, same-type findings
split across reports, untested claims presented as verified. Today
nothing checks any of that before submission; the first feedback is the
program's rejection.

scripts/submission_review adds the validation side of the loop:

- review.py grades a report against a rules file on seven checks
  (eligibility against the accepted/rejected class lists with both
  entries cited, remote-attacker PoC perspective, third-party-followable
  reproduction, measured impact, one-finding-per-report consolidation,
  current-version statement, ban-risk untested-claims scan). Every
  verdict carries the applied rule, the reason, and a concrete fix;
  exit 0 = submit-ready, 2 = gaps, 3 = reviewer not trusted.
- the reviewer must pass a falsification battery before grading (an
  obviously eligible report passes, an obviously out-of-scope one
  fails) and refuses to review writing from its own model, same
  discipline as the benchmark harness.
- rules/strict-vdp.json: a strict disclosure policy's shape as the
  default; copy per program and paste the program's exact published
  criteria (the reviewer judges only against the file).

Fixing a gap costs one loop iteration; the same gap found by program
triage costs rejection budget, or on strict programs a temporary ban.
2026-10-12 08:09:57 +02:00
BaaRai 3231211fca Merge pull request #61 from BaaRai/feat/profile-posture-check
feat: refusal-posture probe for LLM profiles
2026-10-12 08:04:35 +02:00
16 changed files with 1001 additions and 14 deletions
+26 -3
View File
@@ -64,6 +64,12 @@ type LLMProfile struct {
// or sticky routing off a session-id header. "" = not sent. Maps to
// agent.Config.SessionHeaderKey.
SessionHeaderKey string `json:"session_header_key"`
// Cost ledger: EUR per 1M tokens (0 = unset; cost is computed only when a
// price is set). Set through the same profile API/SaveProfile path.
PriceInputPerM float64 `json:"price_input_per_m,omitempty"`
PriceOutputPerM float64 `json:"price_output_per_m,omitempty"`
PriceCacheReadPerM float64 `json:"price_cache_read_per_m,omitempty"`
PriceCacheWritePerM float64 `json:"price_cache_write_per_m,omitempty"`
// Retry overrides this profile's share of the retry ladder. Zero value =
// inherit the global policy (LLMRetryPolicy), so an untouched profile behaves
// exactly as before. See RetryOverride.
@@ -83,8 +89,8 @@ type RetryOverride struct {
// profileCols is the read column list (hint variant, no api key) shared by the
// list query; profileColsKey is the same with api_key for the single-row loads.
const profileRetryCols = `COALESCE(retry_connect_attempts,0),COALESCE(retry_connect_interval_ms,0),COALESCE(retry_empty_attempts,0),COALESCE(retry_empty_interval_ms,0),COALESCE(retry_stream_attempts,0),COALESCE(retry_stream_interval_ms,0)`
const profileCols = `id,name,format,COALESCE(base_url,''),COALESCE(proxy,''),model,COALESCE(api_key_hint,''),rate_per_second,rate_per_minute,context_window_k,COALESCE(reasoning_effort,''),is_default,priority,pool_exclude,COALESCE(thinking_type,''),COALESCE(streaming,true),COALESCE(max_tokens,0),COALESCE(max_tokens_field,''),COALESCE(session_header_key,''),` + profileRetryCols
const profileColsKey = `id,name,format,COALESCE(base_url,''),COALESCE(proxy,''),model,COALESCE(api_key,''),rate_per_second,rate_per_minute,context_window_k,COALESCE(reasoning_effort,''),is_default,priority,pool_exclude,COALESCE(thinking_type,''),COALESCE(streaming,true),COALESCE(max_tokens,0),COALESCE(max_tokens_field,''),COALESCE(session_header_key,''),` + profileRetryCols
const profileCols = `id,name,format,COALESCE(base_url,''),COALESCE(proxy,''),model,COALESCE(api_key_hint,''),rate_per_second,rate_per_minute,context_window_k,COALESCE(reasoning_effort,''),is_default,priority,pool_exclude,COALESCE(thinking_type,''),COALESCE(streaming,true),COALESCE(max_tokens,0),COALESCE(max_tokens_field,''),COALESCE(session_header_key,''),` + profileRetryCols + `,COALESCE(price_input_per_m,0),COALESCE(price_output_per_m,0),COALESCE(price_cache_read_per_m,0),COALESCE(price_cache_write_per_m,0)`
const profileColsKey = `id,name,format,COALESCE(base_url,''),COALESCE(proxy,''),model,COALESCE(api_key,''),rate_per_second,rate_per_minute,context_window_k,COALESCE(reasoning_effort,''),is_default,priority,pool_exclude,COALESCE(thinking_type,''),COALESCE(streaming,true),COALESCE(max_tokens,0),COALESCE(max_tokens_field,''),COALESCE(session_header_key,''),` + profileRetryCols + `,COALESCE(price_input_per_m,0),COALESCE(price_output_per_m,0),COALESCE(price_cache_read_per_m,0),COALESCE(price_cache_write_per_m,0)`
// scanProfile reads one row in the profileCols / profileColsKey column order. The
// 7th column lands in APIKeyHint or APIKey depending on which list the caller used.
@@ -94,7 +100,8 @@ func scanProfile(sc interface{ Scan(...any) error }, into *string, p *LLMProfile
&p.MaxTokens, &p.MaxTokensField, &p.SessionHeaderKey,
&p.Retry.Connect.Attempts, &p.Retry.Connect.IntervalMS,
&p.Retry.Empty.Attempts, &p.Retry.Empty.IntervalMS,
&p.Retry.Stream.Attempts, &p.Retry.Stream.IntervalMS)
&p.Retry.Stream.Attempts, &p.Retry.Stream.IntervalMS,
&p.PriceInputPerM, &p.PriceOutputPerM, &p.PriceCacheReadPerM, &p.PriceCacheWritePerM)
}
func (d *DB) ListProfiles() ([]*LLMProfile, error) {
@@ -176,20 +183,36 @@ func (d *DB) SaveProfile(p *LLMProfile) (int64, error) {
VALUES ($1,$2,NULLIF($3,''),NULLIF($4,''),$5,NULLIF($6,''),NULLIF($7,''),$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24) RETURNING id`,
p.Name, p.Format, p.BaseURL, p.Proxy, p.Model, p.APIKey, hint, p.RatePerSecond, p.RatePerMinute, p.ContextWindowK, p.ReasoningEffort, p.Priority, p.PoolExclude, p.ThinkingType, p.Streaming, p.MaxTokens, p.MaxTokensField, p.SessionHeaderKey,
r.Connect.Attempts, r.Connect.IntervalMS, r.Empty.Attempts, r.Empty.IntervalMS, r.Stream.Attempts, r.Stream.IntervalMS).Scan(&id)
if err == nil {
_, err = d.saveProfilePrices(id, p)
}
return id, err
}
if p.APIKey == "" {
_, err := d.Exec(`UPDATE llm_profiles SET name=$1,format=$2,base_url=NULLIF($3,''),proxy=NULLIF($4,''),model=$5,rate_per_second=$6,rate_per_minute=$7,context_window_k=$8,reasoning_effort=$9,priority=$10,pool_exclude=$11,thinking_type=$12,streaming=$13,max_tokens=$14,max_tokens_field=$15,session_header_key=$16,retry_connect_attempts=$17,retry_connect_interval_ms=$18,retry_empty_attempts=$19,retry_empty_interval_ms=$20,retry_stream_attempts=$21,retry_stream_interval_ms=$22 WHERE id=$23`,
p.Name, p.Format, p.BaseURL, p.Proxy, p.Model, p.RatePerSecond, p.RatePerMinute, p.ContextWindowK, p.ReasoningEffort, p.Priority, p.PoolExclude, p.ThinkingType, p.Streaming, p.MaxTokens, p.MaxTokensField, p.SessionHeaderKey,
r.Connect.Attempts, r.Connect.IntervalMS, r.Empty.Attempts, r.Empty.IntervalMS, r.Stream.Attempts, r.Stream.IntervalMS, p.ID)
if err == nil {
_, err = d.saveProfilePrices(p.ID, p)
}
return p.ID, err
}
_, err := d.Exec(`UPDATE llm_profiles SET name=$1,format=$2,base_url=NULLIF($3,''),proxy=NULLIF($4,''),model=$5,api_key=$6,api_key_hint=$7,rate_per_second=$8,rate_per_minute=$9,context_window_k=$10,reasoning_effort=$11,priority=$12,pool_exclude=$13,thinking_type=$14,streaming=$15,max_tokens=$16,max_tokens_field=$17,session_header_key=$18,retry_connect_attempts=$19,retry_connect_interval_ms=$20,retry_empty_attempts=$21,retry_empty_interval_ms=$22,retry_stream_attempts=$23,retry_stream_interval_ms=$24 WHERE id=$25`,
p.Name, p.Format, p.BaseURL, p.Proxy, p.Model, p.APIKey, hint, p.RatePerSecond, p.RatePerMinute, p.ContextWindowK, p.ReasoningEffort, p.Priority, p.PoolExclude, p.ThinkingType, p.Streaming, p.MaxTokens, p.MaxTokensField, p.SessionHeaderKey,
r.Connect.Attempts, r.Connect.IntervalMS, r.Empty.Attempts, r.Empty.IntervalMS, r.Stream.Attempts, r.Stream.IntervalMS, p.ID)
if err == nil {
_, err = d.saveProfilePrices(p.ID, p)
}
return p.ID, err
}
// saveProfilePrices writes the cost-ledger columns (kept out of the three main
// statements so the pricing surface stays one small, auditable write).
func (d *DB) saveProfilePrices(id int64, p *LLMProfile) (sql.Result, error) {
return d.Exec(`UPDATE llm_profiles SET price_input_per_m=($1),price_output_per_m=($2),price_cache_read_per_m=($3),price_cache_write_per_m=($4) WHERE id=$5`,
p.PriceInputPerM, p.PriceOutputPerM, p.PriceCacheReadPerM, p.PriceCacheWritePerM, id)
}
var (
ErrActiveLLMProfileDelete = errors.New("cannot delete the active LLM profile; activate another profile first")
ErrLLMProfileReferencesChanged = errors.New("LLM profile references changed while deleting; retry the request")
+22
View File
@@ -1192,6 +1192,28 @@ WHERE exploration_id=$1 AND kind='goal' AND state='open')`, s.expID).Scan(&exist
return exists, err
}
// OpenGoalFingerprint returns a stable fingerprint of the open-goal set: the
// sorted goal ids joined by commas, "" when none. Planner quiet-mode (#79) uses
// it to tell "the goal set changed since the last planning round" apart from
// "nothing the plan depends on changed".
func (s *ExplorationStore) OpenGoalFingerprint() (string, error) {
rows, err := s.db.Query(`SELECT id FROM exploration_nodes
WHERE exploration_id=$1 AND kind='goal' AND state='open' ORDER BY id`, s.expID)
if err != nil {
return "", err
}
defer rows.Close()
var ids []string
for rows.Next() {
var id int64
if err := rows.Scan(&id); err != nil {
return "", err
}
ids = append(ids, strconv.FormatInt(id, 10))
}
return strings.Join(ids, ","), rows.Err()
}
// ClaimIntent atomically moves an open intent to running. Returns true if claimed.
func (s *ExplorationStore) ClaimIntent(id int64, owner string) (bool, error) {
res, err := s.db.Exec(`UPDATE exploration_nodes SET state='running', owner=$1
+54
View File
@@ -117,6 +117,8 @@ type ProfileUsage struct {
OutputTokens int `json:"output_tokens"`
CacheReadTokens int `json:"cache_read_tokens"`
CacheWriteTokens int `json:"cache_write_tokens"`
// CostEur is set only when the profile carries at least one price; nil = not priced.
CostEur *float64 `json:"cost_eur,omitempty"`
}
// UsageByProfile returns global token spend grouped by profile name, most-used
@@ -171,6 +173,9 @@ ORDER BY SUM(input_tokens) + SUM(output_tokens) DESC`)
for _, current := range byName {
out = append(out, current)
}
if err := applyCostEur(d, out); err != nil {
return nil, err
}
sort.Slice(out, func(i, j int) bool {
left := out[i].InputTokens + out[i].OutputTokens
right := out[j].InputTokens + out[j].OutputTokens
@@ -332,3 +337,52 @@ func ParseExpID(s string) int64 {
}
return n
}
// applyCostEur fills CostEur on each profile bucket from the stored per-profile
// prices (EUR per 1M tokens). Profiles without any price keep CostEur nil. The
// join is by profile name (llm_usage records the name, not the id); the first
// matching profile wins, and duplicate names are a configuration error the
// operator sees as a weird number, not a crash.
func applyCostEur(d *DB, out []ProfileUsage) error {
rows, err := d.Query(`SELECT name, COALESCE(price_input_per_m,0), COALESCE(price_output_per_m,0), COALESCE(price_cache_read_per_m,0), COALESCE(price_cache_write_per_m,0) FROM llm_profiles`)
if err != nil {
return err
}
defer rows.Close()
type prices struct{ in, outp, cr, cw float64 }
byName := map[string]prices{}
seen := map[string]bool{}
for rows.Next() {
var n string
var pr prices
if err := rows.Scan(&n, &pr.in, &pr.outp, &pr.cr, &pr.cw); err != nil {
return err
}
if !seen[n] {
byName[n] = pr
seen[n] = true
}
}
if err := rows.Err(); err != nil {
return err
}
for i := range out {
pr, ok := byName[out[i].ProfileName]
if !ok || (pr.in == 0 && pr.outp == 0 && pr.cr == 0 && pr.cw == 0) {
continue
}
// InputTokens includes the cached subset (Anthropic and OpenAI usage
// semantics), so the fresh share is input minus cache reads; charging
// both buckets at full input price double-counts the cache. Clamped at
// zero for gateways that report the buckets disjointly.
fresh := out[i].InputTokens - out[i].CacheReadTokens
if fresh < 0 {
fresh = 0
}
cost := float64(fresh)/1e6*pr.in + float64(out[i].OutputTokens)/1e6*pr.outp +
float64(out[i].CacheReadTokens)/1e6*pr.cr + float64(out[i].CacheWriteTokens)/1e6*pr.cw
c := cost
out[i].CostEur = &c
}
return nil
}
+7
View File
@@ -1425,3 +1425,10 @@ CREATE INDEX IF NOT EXISTS idx_notification_deliveries_batch
ON notification_deliveries(batch_id) WHERE batch_id IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_notification_deliveries_channel
ON notification_deliveries(channel_id, id DESC);
-- Cost ledger (#70): optional per-profile pricing, EUR per 1M tokens. NULL/0 = unset;
-- usage aggregations compute cost only when at least one price is set.
ALTER TABLE llm_profiles ADD COLUMN IF NOT EXISTS price_input_per_m NUMERIC(12,6);
ALTER TABLE llm_profiles ADD COLUMN IF NOT EXISTS price_output_per_m NUMERIC(12,6);
ALTER TABLE llm_profiles ADD COLUMN IF NOT EXISTS price_cache_read_per_m NUMERIC(12,6);
ALTER TABLE llm_profiles ADD COLUMN IF NOT EXISTS price_cache_write_per_m NUMERIC(12,6);
+70
View File
@@ -0,0 +1,70 @@
# Pre-submission report review
The submission-validation loop: review a vulnerability report against a
program's eligibility rules BEFORE submitting, fix the gaps, re-review, submit
only on SUBMIT-READY. Catches locally what the program's triage would catch
later, while fixing it is still free.
Uses the same measurement discipline as the benchmark harness: the reviewer is
an independent LLM that must pass a falsification battery before it is allowed
to grade (an obviously eligible report must pass, an obviously out-of-scope one
must fail), it refuses to review a report written by its own model, and every
verdict cites the rule it applied plus a concrete fix.
## The loop
```
finding + report
|
v
review.py --report report.md --rules rules/<program>.json
|
SUBMIT-READY? --no--> fix the cited gaps --> re-run
|
yes
v
submit
```
Exit codes: `0` submit-ready, `2` gaps found, `3` reviewer not trusted
(battery failed or configuration error).
## Checks
| Check | What it enforces |
|---|---|
| eligibility | class is in the rules' accepted list and not in the rejected list (cites both) |
| poc_remote | reproduction is a remote attacker over HTTP, not server-side access |
| reproduction | a third party can follow it; requests complete; expected vs actual stated |
| impact_measured | quantified with denominator or a demonstrated outcome, not adjectives |
| consolidation | one finding per report; same-type findings consolidated |
| version_current | component + exact version + latest-at-time stated |
| ban_risk | no untested claims presented as verified |
## Usage
```bash
REVIEW_API=https://api.example.com REVIEW_API_KEY=... REVIEW_MODEL=<reviewer> \
python3 review.py --report my-finding.md --rules rules/strict-vdp.json \
--subject-model <model that wrote the report>
```
`--subject-model` exists so a model never reviews its own writing (pass
`--allow-self-review` to override knowingly). Verdicts land next to the report
as `<report>.review.json` with every check, the applied rule, the reason, and
the suggested fix.
## Rule files
`rules/strict-vdp.json` is the default: a strict disclosure policy's shape
(accepted classes with chaining conditions, rejected classes, format
requirements, risk notes). Copy it per program and paste the program's exact
published criteria — the reviewer judges only against what the file says, so
fidelity to the real policy is the file's job, not the reviewer's.
## Limits
The reviewer is an LLM and can misjudge; every verdict carries its reason and
the rule text, so audit before trusting a pass. A single pass is one opinion:
for high-stakes submissions, re-run after every edit (the loop is cheap; a
rejection is not).
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""Pre-submission report review: validate a vulnerability report against a
program's eligibility rules BEFORE it is submitted, and refine on the feedback
(local oracle + refine-on-response).
Why: strict vulnerability programs reject (and some ban) reports that miss
their criteria — wrong vulnerability class, PoC not written from a remote
attacker's perspective, no measurable impact, several findings of one type
split across reports. Reviewing that locally first, with an independent
reviewer that must pass a falsification battery, catches exactly the gaps a
program's triage would catch, while fixing them is still free.
Discipline (same method as the benchmark harness in this repo):
- the reviewer is an independent LLM and must pass a battery before grading
(an obviously eligible report passes, an obviously out-of-scope one fails,
a vague one fails); a grader that cannot fail the wrong ones does not run
- every verdict carries the rule it cites and a suggested fix
- exit code 0 = submit-ready, 2 = gaps found, 3 = reviewer not trusted
Usage:
REVIEW_API=https://api.example.com REVIEW_API_KEY=... REVIEW_MODEL=... \
python3 review.py --report finding.md --rules rules/strict-vdp.json
"""
from __future__ import annotations
import argparse
import json
import os
import pathlib
import sys
import urllib.error
import urllib.request
REVIEW_VERSION = "review-v1"
REVIEWER_SYSTEM = """You are a strict vulnerability-program triager. You receive one security report and the program's eligibility rules. Judge the report EXACTLY against the rules, not against your own sense of severity.
For each check below, output one object in a JSON array:
{"check": "<check id>", "pass": true|false, "rule": "<the rule text you applied>", "reason": "1-2 sentences citing the report", "fix": "one concrete sentence to fix it, empty when pass"}
Checks to run (ids in this order): eligibility, poc_remote, reproduction, impact_measured, consolidation, version_current, ban_risk.
- eligibility: the vulnerability class is in the rules' accepted list AND not in the rejected list. Quote both entries you compared against.
- poc_remote: the reproduction steps are written from a REMOTE attacker's perspective over HTTP (requests an outside attacker can send), not requiring server-side access, local files, or vendor tooling on the server.
- reproduction: a third party could follow the steps and see the result; every request shows method, URL, headers that matter, and body; expected vs actual behavior is stated.
- impact_measured: the impact is quantified with its denominator or a concrete attacker outcome (data actually returned / command actually executed), not an adjective. "An attacker can dump the table" fails; "3 of 20 payload variants returned other-tenant rows" passes.
- consolidation: the report covers ONE finding; same-type findings the reporter knows about are consolidated here, not promised in separate reports.
- version_current: the report states the affected component, its exact version, and that it is the latest at the time of writing.
- ban_risk: the report contains no untested claims presented as verified (every claim traces to an executed request or output in the report).
Output ONLY the JSON array."""
def llm(base: str, key: str, model: str, user: str, max_tokens: int = 3000) -> str:
payload = json.dumps({
"model": model,
"messages": [{"role": "system", "content": REVIEWER_SYSTEM}, {"role": "user", "content": user}],
"temperature": 0, "max_tokens": max_tokens,
}).encode()
r = urllib.request.Request(base.rstrip("/") + "/chat/completions", data=payload, method="POST")
r.add_header("Content-Type", "application/json")
r.add_header("Authorization", "Bearer " + key)
try:
with urllib.request.urlopen(r, timeout=int(os.environ.get("REVIEW_TIMEOUT", "900"))) as resp:
body = json.loads(resp.read().decode())
return body["choices"][0]["message"]["content"] or ""
except urllib.error.HTTPError as exc:
raise SystemExit(f"reviewer call failed: HTTP {exc.code}: {exc.read().decode(errors='replace')[:200]}")
def parse_verdicts(raw: str) -> list[dict]:
start, end = raw.find("["), raw.rfind("]")
if start == -1 or end <= start:
raise ValueError("no JSON array in reviewer output")
verdicts = json.loads(raw[start:end + 1])
if not isinstance(verdicts, list) or not verdicts:
raise ValueError("empty verdicts")
return verdicts
def battery(base: str, key: str, model: str, rules: dict) -> None:
"""The reviewer must fail the wrong reports before it is allowed to grade."""
good = "# SQL injection in item search\n\nPOST /search with q=' OR 1=1-- returns all rows.\n\n## Measured impact\n2 of 4 payload variants returned rows from other tenants; 2 rejected fail-closed. Proof basis: executed against the current released version 2.3.1 over HTTP."
bad = "# Username enumeration on login\n\nDifferent responses for existing vs unknown users let an attacker enumerate accounts."
cases = [
("eligible-must-pass", good, True, "eligibility"),
("out-of-scope-must-fail", bad, False, "eligibility"),
]
for label, report, expect, check in cases:
user = f"## Program rules\n{json.dumps(rules, indent=1)}\n\n## Report\n{report}\n\nRun the checks; output the JSON array."
raw = llm(base, key, model, user, 2000)
try:
v = {x.get("check"): x for x in parse_verdicts(raw)}
got = v.get(check, {}).get("pass")
except ValueError:
raise SystemExit(f"reviewer battery '{label}': unparseable output; reviewer not trusted")
if got != expect:
raise SystemExit(f"reviewer battery FAILED on '{label}' (expected {check} pass={expect}, got {got}); a reviewer that cannot fail the wrong reports does not run")
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--report", required=True, help="path to the report markdown (or '-' for stdin)")
ap.add_argument("--rules", default=str(pathlib.Path(__file__).parent / "rules" / "strict-vdp.json"))
ap.add_argument("--subject-model", help="model that wrote the report, to refuse self-review")
ap.add_argument("--allow-self-review", action="store_true")
ap.add_argument("--out", help="write the verdict JSON here (default: alongside the report)")
args = ap.parse_args()
base = os.environ.get("REVIEW_API", "")
key = os.environ.get("REVIEW_API_KEY", "")
model = os.environ.get("REVIEW_MODEL", "")
if not (base and key and model):
raise SystemExit("need REVIEW_API, REVIEW_API_KEY, REVIEW_MODEL")
if args.subject_model and args.subject_model == model and not args.allow_self_review:
raise SystemExit(f"reviewer model equals the report's model ({model}); refusing. Pass --allow-self-review to override knowingly.")
rules = json.loads(pathlib.Path(args.rules).read_text())
report = sys.stdin.read() if args.report == "-" else pathlib.Path(args.report).read_text()
if not report.strip():
raise SystemExit("empty report")
print("[battery] verifying the reviewer fails the wrong reports")
battery(base, key, model, rules)
user = (f"## Program rules\n{json.dumps(rules, indent=1)}\n\n## Report\n{report}\n\n"
"Run the checks in order; output ONLY the JSON array.")
print("[review] grading")
verdicts = parse_verdicts(llm(base, key, model, user))
failed = [v for v in verdicts if not v.get("pass")]
doc = {
"review_version": REVIEW_VERSION,
"rules_file": os.path.basename(args.rules),
"reviewer_model": model,
"submit_ready": not failed,
"failed_checks": failed,
"verdicts": verdicts,
}
out = pathlib.Path(args.out) if args.out else pathlib.Path(args.report).with_suffix(".review.json")
out.write_text(json.dumps(doc, indent=2))
print(f"\n{'SUBMIT-READY' if not failed else 'GAPS FOUND'} ({len(verdicts) - len(failed)}/{len(verdicts)} checks pass)")
for v in failed:
print(f" FAIL {v.get('check')}: {v.get('reason')}")
if v.get("fix"):
print(f" fix: {v['fix']}")
print(f"\nverdicts: {out}")
print("fix the gaps, re-run, submit only on SUBMIT-READY. A rate without its basis is not a verdict: read the cited rules before trusting a pass.")
sys.exit(0 if not failed else 2)
if __name__ == "__main__":
main()
+39
View File
@@ -0,0 +1,39 @@
{
"name": "strict-vdp",
"version": "v1",
"description": "Default rule set mirroring a strict vulnerability-disclosure policy (rewrite it to your program's exact published text; the reviewer only judges against what is written here).",
"accepted_classes": [
"SQL injection",
"arbitrary file upload / deletion / download with full path and extension control",
"remote code execution",
"PHP object injection",
"significant settings change (impactful application options)",
"privilege escalation to contributor level or above",
"broken access control on genuinely sensitive objects only (API keys, secrets, password hashes, backup or SQL files)",
"IDOR with significant impact (personal-data leakage alone is NOT significant)",
"stored XSS that executes on page open with zero interaction, unauthenticated or low-privilege reachable",
"CSRF only when it chains into one of the accepted write actions above",
"denial of service only when the whole site crashes"
],
"rejected_classes": [
"user or account enumeration and other low-impact information disclosure",
"personal-data-limited broken access control or IDOR",
"CSRF that does not chain into an accepted write action",
"role prerequisites at or below contributor level",
"blind SSRF",
"CSV injection",
"open redirects",
"self-XSS or XSS requiring unusual interaction"
],
"format_requirements": {
"language": "English",
"version": "the latest released version of the component at the time of writing",
"poc_perspective": "remote attacker over HTTP; steps a third party can follow end to end; the full PoC lives in the report body, not only in attachments",
"consolidation": "multiple findings of the same type go into ONE report",
"impact": "quantified with its denominator or a concrete demonstrated attacker outcome"
},
"risk_notes": [
"an untested or assumption-based report can earn an immediate temporary ban",
"a high rejection rate can remove the reporter from the program"
]
}
+61
View File
@@ -0,0 +1,61 @@
package server
import (
"testing"
"github.com/Autumn-27/artex/db"
)
// The cost ledger: profile prices (EUR per 1M tokens) turn token buckets into
// EUR on the usage aggregation; unpriced profiles stay nil.
func TestCostLedgerEurOnUsage(t *testing.T) {
m, err := NewManager(t.TempDir(), "")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { m.Close() })
if m.pg == nil {
t.Skip("no postgres for this run")
}
pid, err := m.pg.SaveProfile(&db.LLMProfile{
Name: "cost-test", Format: "openai", Model: "m", APIKey: "k",
PriceInputPerM: 1.5, PriceOutputPerM: 4.0, PriceCacheReadPerM: 0.15,
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _, _ = m.pg.Exec("DELETE FROM llm_profiles WHERE id=$1", pid) })
stored, err := m.pg.ProfileByID(pid)
if err != nil || stored == nil {
t.Fatalf("ProfileByID: %v %v", stored, err)
}
if stored.PriceInputPerM != 1.5 || stored.PriceOutputPerM != 4.0 || stored.PriceCacheReadPerM != 0.15 || stored.PriceCacheWritePerM != 0 {
t.Fatalf("prices did not round-trip: %+v", *stored)
}
if _, err := m.pg.Exec(`INSERT INTO llm_usage(task_id, profile_name, model, input_tokens, output_tokens, cache_read, cache_write, ts)
VALUES ('', 'cost-test', 'm', 2000000, 500000, 1000000, 0, now())`); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _, _ = m.pg.Exec(`DELETE FROM llm_usage WHERE profile_name='cost-test'`) })
profiles, err := m.pg.UsageByProfile()
if err != nil {
t.Fatal(err)
}
var got *float64
for _, p := range profiles {
if p.ProfileName == "cost-test" {
got = p.CostEur
}
}
if got == nil {
t.Fatal("priced profile has no cost_eur")
}
want := 1.0*1.5 + 0.5*4.0 + 1.0*0.15 // fresh input 1M (2M incl. 1M cached) + 0.5M out + 1M cache-read = 3.65
if d := *got - want; d > 0.001 || d < -0.001 {
t.Fatalf("cost_eur = %v, want %v", *got, want)
}
}
+37
View File
@@ -99,6 +99,10 @@ type Engine struct {
started sync.Map // taskID -> bool, so Run is idempotent per task
lastAct sync.Map // taskID -> int64 unix, last planner/worker activity (heartbeat)
llmCalls sync.Map // taskID -> *int64, actual planner/worker/main-agent LLM calls
// quiet-mode (#79): open-goal fingerprint recorded at the end of the last
// completed planning round, per task. Edge-triggered rounds are skipped
// while it is unchanged and work is still in flight.
goalFP sync.Map // taskID -> string (sorted open-goal ids joined)
paused sync.Map // taskID -> bool, user-paused (planner + workers idle but loops alive)
deleting sync.Map // taskID -> bool, delete barrier (no new task-owned writes)
dropCnt sync.Map // taskID -> *int64, running count of dropped (unpersistable) activity records
@@ -797,6 +801,22 @@ func resetPlannerTimer(timer *time.Timer, d time.Duration) {
timer.Reset(d)
}
// quietSkipEdge reports whether an edge-triggered planning round should be
// dropped (#79): the open-goal set is unchanged since the last completed round
// AND active work exists (open or running intents). Fails open (false) on any
// store error: when in doubt, plan, preserving the pre-quiet-mode behavior.
func (e *Engine) quietSkipEdge(t *Task) bool {
fp, err := t.Store.OpenGoalFingerprint()
if err != nil {
return false
}
if last, ok := e.goalFP.Load(t.ID); !ok || last != fp {
return false
}
active, err := t.Store.HasActiveIntent()
return err == nil && active
}
func (e *Engine) plannerLoop(ctx context.Context, t *Task) {
interval := plannerHeartbeatInterval(t)
// Arming the heartbeat timer at loop entry = counting from the task start: even a
@@ -856,6 +876,17 @@ func (e *Engine) plannerLoop(ctx context.Context, t *Task) {
}
return // the goalless branch never enters planner.Plan
}
// quiet-mode (#79): an edge round is redundant while work is in flight
// and the open-goal set is unchanged since the last completed round —
// there is no new decision for the plan to make. Drain the triggers so
// they do not accumulate and let the heartbeat supervise; the next edge
// after a goal opens or closes, or once the frontier drains and all
// workers are idle, plans for real.
if src == "edge" && e.quietSkipEdge(t) {
t.drainTriggers()
log.Printf("[planner] task %s quiet: work in flight, goals unchanged, edge round skipped", t.ID)
return
}
if !e.beginTaskOperation(t.ID) {
return
}
@@ -895,6 +926,12 @@ func (e *Engine) plannerLoop(ctx context.Context, t *Task) {
default:
log.Printf("[planner] task %s planning finished", t.ID)
}
// Record the post-round open-goal fingerprint for quiet-mode (#79):
// goals opened or closed by this round change it, so a later edge can
// tell "nothing changed" apart from "the plan input changed".
if fp, err := t.Store.OpenGoalFingerprint(); err == nil {
e.goalFP.Store(t.ID, fp)
}
e.touch(t.ID)
}
+61
View File
@@ -92,3 +92,64 @@ func TestLLMTestFallbackMatchesRuntimeNormalization(t *testing.T) {
t.Fatalf("ConfigFrom trims: %q", runtime.APIKey)
}
}
// The stored-profile test must reach the profile's own endpoint: an empty form
// (the list-view test button) fills provider/model/base_url from the row, so
// the key lands on the configured host instead of the canonical default one.
func TestLLMTestStoredProfileUsesProfileEndpoint(t *testing.T) {
m, err := NewManager(t.TempDir(), "")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { m.Close() })
if m.pg == nil {
t.Skip("no postgres for this run")
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
s := New(ctx, m, t.TempDir(), t.TempDir(), t.TempDir())
var hits atomic.Int32
var gotKey atomic.Value
fake := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hits.Add(1)
gotKey.Store(r.Header.Get("x-api-key"))
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"id":"m","type":"message","role":"assistant","model":"m","content":[{"type":"text","text":"OK"}],"stop_reason":"end_turn","usage":{"input_tokens":1,"output_tokens":1}}`))
}))
t.Cleanup(fake.Close)
pid, err := m.pg.SaveProfile(&db.LLMProfile{
Name: "stored-ep", Format: "anthropic", Model: "claude-x",
BaseURL: fake.URL, APIKey: "k1",
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _, _ = m.pg.Exec("DELETE FROM llm_profiles WHERE id=$1", pid) })
// Empty form on purpose: no provider, model, base_url, or streaming.
body, _ := json.Marshal(map[string]any{"profile_id": pid, "api_key": ""})
req := httptest.NewRequest(http.MethodPost, "/api/llm/test", bytes.NewReader(body))
req.Header.Set("Content-Type", "application/json")
rec := httptest.NewRecorder()
s.testLLM(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("handler status %d", rec.Code)
}
if hits.Load() != 1 {
t.Fatalf("stored-profile test must hit the profile endpoint exactly once, hits=%d", hits.Load())
}
if k, _ := gotKey.Load().(string); k != "k1" {
t.Fatalf("test request must send the stored key, header was %q", k)
}
var resp struct {
OK bool `json:"ok"`
Error string `json:"error"`
}
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
if !resp.OK {
t.Fatalf("stored-profile round-trip not ok: %s", resp.Error)
}
}
+76
View File
@@ -0,0 +1,76 @@
package server
import (
"testing"
"github.com/Autumn-27/artex/db"
)
// Planner quiet-mode (#79): an edge round is dropped only when BOTH hold — the
// open-goal fingerprint is unchanged since the last completed round AND active
// work exists (open or running intents). Every other combination plans.
func TestPlannerQuietSkipEdge(t *testing.T) {
m, err := NewManager(t.TempDir(), "")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = m.Close() })
if m.pg == nil {
t.Skip("no postgres for this run")
}
created, err := m.CreateTaskWithOptions("quiet-mode probe", "probe goal", db.TaskCreateOptions{})
if err != nil {
t.Fatal(err)
}
tt, ok := m.Task(created.ID)
if !ok {
t.Fatal("task not found after creation")
}
store := tt.Store
e := NewEngine(m)
gid, err := store.AddGoal(map[string]any{"summary": "g1"}, "seed")
if err != nil {
t.Fatal(err)
}
iid, err := store.AddIntent(map[string]any{"summary": "i1"}, 0, nil, "seed")
if err != nil {
t.Fatal(err)
}
fp, err := store.OpenGoalFingerprint()
if err != nil {
t.Fatal(err)
}
if fp == "" {
t.Fatal("fingerprint must not be empty with an open goal")
}
e.goalFP.Store(tt.ID, fp)
if !e.quietSkipEdge(tt) {
t.Fatal("unchanged goals + active intent must skip the edge round")
}
// a goal closes -> the fingerprint changes -> plan
if err := store.SetNodeState(gid, "met"); err != nil {
t.Fatal(err)
}
if e.quietSkipEdge(tt) {
t.Fatal("a closed goal must allow planning")
}
// goal back open, intent drained -> no active work -> plan
if err := store.SetNodeState(gid, "open"); err != nil {
t.Fatal(err)
}
fp2, err := store.OpenGoalFingerprint()
if err != nil {
t.Fatal(err)
}
e.goalFP.Store(tt.ID, fp2)
if err := store.SetIntentState(iid, "done"); err != nil {
t.Fatal(err)
}
if e.quietSkipEdge(tt) {
t.Fatal("no active intent must allow planning")
}
}
+44 -11
View File
@@ -1385,6 +1385,41 @@ func (s *Server) testLLM(w http.ResponseWriter, r *http.Request) {
writeErr(w, 400, err.Error())
return
}
// A stored profile must be tested AS STORED: the list-view test button
// sends only profile_id, and ConfigFrom's defaults for an empty
// provider/base_url point at the canonical provider endpoint, sending this
// profile's key to the wrong host (the stored-key 401s). Fill whatever the
// form left empty from the row first; explicit form values still win.
var stored *db.LLMProfile
if req.ProfileID != nil {
if p, err := s.m.pg.ProfileByID(*req.ProfileID); err == nil && p != nil {
stored = p
}
}
if stored != nil {
if strings.TrimSpace(req.Provider) == "" {
req.Provider = stored.Format
}
if strings.TrimSpace(req.Model) == "" {
req.Model = stored.Model
}
if strings.TrimSpace(req.BaseURL) == "" {
req.BaseURL = stored.BaseURL
}
if strings.TrimSpace(req.Proxy) == "" {
req.Proxy = stored.Proxy
}
if req.Streaming == nil {
st := stored.Streaming
req.Streaming = &st
}
if strings.TrimSpace(req.ThinkingType) == "" {
req.ThinkingType = stored.ThinkingType
}
if strings.TrimSpace(req.ReasoningEffort) == "" {
req.ReasoningEffort = stored.ReasoningEffort
}
}
cfg := agent.ConfigFrom(req.Provider, req.Model, req.BaseURL, req.APIKey, req.Proxy)
// mirror production: send the SAME thinking params so a provider that rejects the
// reasoning_effort/thinking field fails the test too (no false "test ok, run 400").
@@ -1402,17 +1437,15 @@ func (s *Server) testLLM(w http.ResponseWriter, r *http.Request) {
// API key resolution precedence: form input > the named profile's stored key > the global config's key.
// A stored profile's key is never sent back to the browser, so the form is empty when testing a stored profile and the key comes from the DB.
// Same for the session header name: when the form omits it, the stored profile's value is the fallback.
if req.ProfileID != nil && (cfg.APIKey == "" || cfg.SessionHeaderKey == "") {
if p, err := s.m.pg.ProfileByID(*req.ProfileID); err == nil && p != nil {
// Trim like ConfigFrom does for form values: a key stored with
// surrounding whitespace runs fine in conversations (the runtime
// path re-trims) but would be sent raw here and fail auth.
if cfg.APIKey == "" {
cfg.APIKey = strings.TrimSpace(p.APIKey)
}
if cfg.SessionHeaderKey == "" {
cfg.SessionHeaderKey = strings.TrimSpace(p.SessionHeaderKey)
}
if stored != nil && (cfg.APIKey == "" || cfg.SessionHeaderKey == "") {
// Trim like ConfigFrom does for form values: a key stored with
// surrounding whitespace runs fine in conversations (the runtime
// path re-trims) but would be sent raw here and fail auth.
if cfg.APIKey == "" {
cfg.APIKey = strings.TrimSpace(stored.APIKey)
}
if cfg.SessionHeaderKey == "" {
cfg.SessionHeaderKey = strings.TrimSpace(stored.SessionHeaderKey)
}
}
if cfg.APIKey == "" {
+132
View File
@@ -0,0 +1,132 @@
---
name: api-security
description: "Invoke this skill when the task is authorized offensive testing of a REST, GraphQL, WebSocket, or SOAP API beyond recon: authentication and authorization flaws (BOLA, IDOR, BFLA), JWT and OAuth weaknesses, GraphQL-specific abuse, input validation, business logic, rate-limit trust confusion, and excessive data exposure."
---
# API Security Testing (REST, GraphQL, WebSocket, SOAP)
This is the hunting counterpart of api-recon. api-recon maps the API surface and stops; this skill attacks the mapped surface under an authorized engagement. The standing finding quality gates and the adversarial gate in your system prompt apply throughout, and an access-control finding only exists once a second account or role has proven the boundary is real.
Attribution: adapted from the `api-security` playbook of [zhaoxuya520/reverse-skill](https://github.com/zhaoxuya520/reverse-skill) (MIT License).
## Boundaries
- Rate-limit and load tests only within the engagement's written impact budget. Most bounty programs forbid denial of service outright; volume that degrades the service for other users is out of bounds. Race-condition tests use a handful of concurrent requests, never load tools.
- Do not exfiltrate real user data to prove a BOLA. One record you are authorized to touch, or your own second account, is sufficient evidence.
- Destructive payloads (deletion, mass assignment on write endpoints) only against accounts and resources created for the test.
- Every confirmed finding needs a replayable artifact: the exact request, the exact response, and the authorization context that produced it.
## Phase checklists
### Phase 1: discovery (handoff from api-recon)
- Spec sources: `/swagger.json`, `/openapi.json`, `/api-docs`, `/graphql`, JS bundles, robots.txt.
- Undocumented path brute force: ffuf or kiterunner with API wordlists.
- GraphQL introspection, three tiers: the full introspection query; a trimmed query if the full one is blocked; minimal `__schema { types { name } }` as the last probe.
### Phase 2: authentication
JWT (jwt_tool or manual):
- [ ] `alg: none`: header rewritten, signature stripped.
- [ ] key confusion: RS256 public key accepted as an HS256 secret.
- [ ] weak HMAC secret: dictionary attack (`jwt_tool -C -d wordlist.txt`).
- [ ] expired or tampered claims: `exp`, `iat`, `sub`, `role`.
- [ ] `kid` injection: path traversal inside the key id.
OAuth 2.0:
- [ ] `redirect_uri` manipulation leading to authorization-code theft.
- [ ] missing `state` (CSRF on the authorization flow).
- [ ] token leakage via `Referer` or platform logs.
- [ ] missing PKCE on public clients.
GraphQL auth:
- [ ] mutation reachable through GET (CSRF).
- [ ] batched queries bypassing per-operation authorization.
### Phase 3: authorization (BOLA / IDOR / BFLA)
BOLA, object level:
- [ ] enumerate numeric ids: `/user/1` to `/user/2`.
- [ ] enumerate UUIDs harvested from other responses.
- [ ] substitute usernames or emails in paths and bodies.
- [ ] two-session replay: user A's session requesting user B's objects (Burp Autorize or a manual response diff).
BFLA, function level:
- [ ] standard user calling admin endpoints.
- [ ] HTTP method swap: GET to PUT, PATCH, DELETE.
- [ ] version downgrade: `/v2/admin` to `/v1/admin`.
- [ ] batch-operation injection: append an admin id to the bulk body.
### Phase 4: GraphQL-specific
- [ ] introspection exposing schema details it should hide.
- [ ] alias overload (dozens of aliases in one query) and batched queries as compute amplification, within the impact budget.
- [ ] field-suggestion error messages leaking schema names.
- [ ] GraphiQL or a playground exposed unauthenticated.
- [ ] deep nesting as a resource-exhaustion vector, within the impact budget.
- [ ] trace or debug mode returning metadata.
### Phase 5: REST input validation
- [ ] HTTP method swap on every endpoint.
- [ ] Content-Type tampering: JSON to XML to multipart (XXE where XML is accepted).
- [ ] NoSQL operator injection: `{"username": {"$gt": ""}}`.
- [ ] SSRF through URL-taking parameters (webhooks, avatar imports, fetchers).
- [ ] HTTP parameter pollution: `?role=user&role=admin`.
- [ ] mass assignment: `is_admin: true` added to request bodies.
### Phase 6: business logic and differential testing
- [ ] diff v1 against v2 of the same API: status-code changes, removed fields, latency regressions.
- [ ] role matrix: admin, user, readonly through the same workflow.
- [ ] coupon, credit, and price manipulation on test accounts.
- [ ] TOCTOU races with a few concurrent requests.
### Phase 7: WebSocket
- [ ] endpoint discovery (upgrade handshakes).
- [ ] message injection, prototype pollution through frames.
- [ ] oversized-message handling.
- [ ] type confusion in frame payloads.
- [ ] cross-site WebSocket hijacking (CSWH).
### Phase 8: rate-limit trust confusion
- [ ] header spoofing: `X-Forwarded-For`, `X-Real-IP`, reported as a trust-boundary finding, never scaled into volume.
- [ ] path variants: `/api/`, `/api`, `/Api/`, `/API/` bypassing a path-keyed limiter.
- [ ] the finding is the bypass mechanism itself; do not demonstrate it under load.
### Phase 9: data exposure
- [ ] response fields beyond what the UI renders (diff the API response against the page).
- [ ] pagination enumeration: `?page=1&limit=10000`.
- [ ] error messages leaking stack traces, internal paths, or SQL.
- [ ] GraphQL nested traversal reaching unauthorized data.
- [ ] spec files exposing internal endpoints.
### Phase 10: regression handoff
- [ ] persist every confirmed finding as a replayable request (curl command or HAR artifact) so retests and the reporter can re-verify.
- [ ] mark which findings are safe to re-run and which are one-shot (nonce or state dependent).
## Tools
| Tool | Use | Notes |
|---|---|---|
| curl + jq | request replay and diffing | always available |
| jwt_tool | JWT attack automation | pip-installable; verify before use |
| ffuf, kiterunner | endpoint brute force | verify container availability |
| Burp Autorize, AuthMatrix | two-session authorization diffing | only where a Burp instance exists |
The upstream playbook also names Vespasian, Entropy, Escape DAST, FireTail, and api.sh; treat those as references until verified in this environment, and prefer the manual checklists over unverified tooling.
## Completion self-check
- [ ] every executed phase produced artifacts (requests, responses, diffs) in the work dir.
- [ ] each access-control finding was confirmed with a second session or role.
- [ ] nothing in the report relies on a tool or output you did not actually run.
+100
View File
@@ -0,0 +1,100 @@
---
name: edr-bypass-re
description: "Invoke this skill for authorized adversary-simulation work that must understand and evaluate endpoint defense telemetry: mapping what an EDR or AV actually monitors (user-mode hooks, kernel callbacks, ETW providers, AMSI), surveying installed hooks from memory, and evaluating countermeasure combinations in an isolated lab, mapped to MITRE ATT&CK T1562."
---
# EDR Bypass RE (defense telemetry analysis for authorized adversary simulation)
Reverse the defender's implementation first, then reason about countermeasures. This skill covers the RE half (what the EDR monitors and how its components are built) and the evaluation half (which countermeasure classes defeat which telemetry, verified in an isolated lab). It maps to MITRE ATT&CK T1562 (Impair Defenses), T1562.001 and T1562.006, with T1055 and T1027 adjacent.
Attribution: adapted from the `edr-bypass-re` playbook of [zhaoxuya520/reverse-skill](https://github.com/zhaoxuya520/reverse-skill) (MIT License).
## Boundaries (read first)
- Authorized environments only: your own lab machines and VMs, or hosts covered by written engagement scope (SoW, test contract, program scope statement). Nothing here may ever be applied to an unowned or out-of-scope system.
- The deliverable is measurement and a report: which telemetry sources detected which technique, at what stage, with what fidelity. Findings go to the engagement's responsible-disclosure channel.
- ARTEX tasks run in a Linux container. Every step here needs an authorized Windows lab VM reachable from the task environment; if none is provisioned, report the gap and stop. Never simulate observations you did not make.
- Report placeholders only: `{target_ip}`, `{hostname}`, `{user}`. Anonymize before anything leaves the lab.
## Applicable task types
1. Purple-team and detection-gap assessment: measure what the deployed EDR actually catches.
2. Product security evaluation: test your own endpoint product against known technique classes.
3. CTF or exercise scenarios requiring execution on hardened hosts, within the exercise rules.
Not applicable: evasion against systems outside written authorization; building mass-distributed malware; any engagement where the objective is intrusion rather than measurement.
## The four telemetry surfaces and the countermeasure classes
```text
EDR monitoring surface countermeasure class (lab evaluation)
────────────────────── ─────────────────────────────────────
user-mode ntdll inline hooks ◄─► fresh ntdll copy from disk (unhook)
direct or indirect syscall stubs with
dynamically resolved SSNs
hardware-breakpoint redirection
kernel callbacks (process, ►─► call-stack spoofing; routing through
thread, image, registry) legitimate trigger chains instead of
direct invocation
ETW telemetry (Threat- ◄─► EtwEventWrite patching; provider
Intelligence and others) teardown via NtTraceControl
AMSI scanning (amsi.dll) ◄─► AmsiScanBuffer patching; hardware-
breakpoint redirection; loading a
clean amsi.dll copy
```
Working knowledge:
- An EDR is not a black box. Its user-mode DLLs and providers can be loaded into IDA or Ghidra and read like any other binary; rea can assist with static structure on host-side copies.
- Countermeasure classes must be combined: each surface needs its own answer, and ordering matters. Silencing telemetry before unhooking avoids the unhook alert arriving first.
- Modern EDRs treat ETW plus kernel callbacks as the primary battlefield; user-mode unhooking alone is a dated answer.
## Workflow
**Step 1: identify the defenses on the lab host.**
```powershell
Get-Service | Where-Object {$_.Name -match 'CSAgent|SentinelAgent|elasticendpoint|esets|ekrn|MsMpEng|wdsvc|cyserver|sysmon|aswbidsagent'}
fltmc filters
```
Kernel-callback enumeration needs kernel debugging (windbg) or a driver-inspection tool. Record every callback class you can observe; the inventory is the baseline artifact.
**Step 2: survey user-mode hooks.** Dump the in-memory `ntdll.dll` `.text` section from a process carrying the EDR's injected DLL, diff it against the on-disk `C:\Windows\System32\ntdll.dll`, and treat every divergence as a hook site. `pe-sieve64.exe /pid <pid> /shellc 3 /modules 3 /dir hooks_dump` automates the sweep. The dump is the artifact; the report cites hook offsets.
**Step 3: select the countermeasure combination to evaluate.**
| Telemetry surface | Countermeasure class under test |
|---|---|
| ntdll inline hooks | indirect syscall with dynamic SSN resolution |
| ETW-TI provider | EtwEventWrite head patch |
| AMSI (PowerShell, .NET) | AmsiScanBuffer patch or HWBP redirection |
| kernel callbacks | call-stack spoofing via legitimate gadgets |
| Sysmon ProcessCreate | PPID spoofing plus unbacked memory |
**Step 4: implement in the test harness.** Reference implementations for each technique class are public: SysWhispers3 for syscall stub generation, the Hell's Gate and Halo's Gate writeups for dynamic SSN resolution, Ekko for sleep-time memory obfuscation. Cite the exact references used, and never paste unvetted code into the lab host.
**Step 5: verify in the isolated lab.** Stand up the target EDR trial (Defender is the default baseline) with Sysmon in an isolated VM, run the harness, and record every alert source that fires: AMSI, ETW-TI, Sysmon events 1/7/8/10, the EDR console. Negative results are results; telemetry that catches the technique belongs in the report.
**Step 6: write the evaluation.** Map every observed detection to MITRE ATT&CK (T1562.001, T1562.006, T1055, T1027) and hand the detection-gap list to the engagement owner.
## Tools
| Tool | Use | Install |
|---|---|---|
| pe-sieve | in-memory hook and injection survey | github.com/hasherezade/pe-sieve |
| SysWhispers3 | syscall stub generation | git clone plus python |
| API Monitor | dynamic API observation | manual download |
| windbg plus IDA or Ghidra | static analysis of EDR DLLs and drivers | manual |
| Sysmon | lab telemetry baseline | official download |
Verify availability in the lab environment before promising a step, and report gaps instead of improvising downloads from unverified sources.
## Completion self-check
- [ ] every technique run in the lab has a recorded observation (alert fired or did not fire) with the telemetry source named.
- [ ] the report contains placeholders only, never real hostnames or IPs.
- [ ] nothing was executed outside the authorized lab VMs.
+110
View File
@@ -0,0 +1,110 @@
---
name: patch-diff
description: "Invoke this skill when the task is recovering a vulnerability from a vendor fix: a CVE with a patch but no public PoC, Patch Tuesday triage, a distribution backport gap, or a closed-source security patch that needs root-cause analysis. Diff the patched and unpatched builds, infer the bug class from the added guards, and verify with a PoC that crashes only the unpatched build."
---
# Patch Diff (N-day recovery from vendor fixes)
Under an **authorized** engagement, recover a vulnerability from the fix that closed it: obtain the before and after builds, align symbols, diff the binaries, read the added checks to infer the bug class, then prove the bug with a PoC that reproduces on the unpatched build and runs clean on the patched one. The standing finding quality gates and the adversarial gate in your system prompt apply throughout: a crash you cannot reproduce on demand is a hypothesis, not a finding.
Attribution: adapted from the `patch-diff-exploit` playbook of [zhaoxuya520/reverse-skill](https://github.com/zhaoxuya520/reverse-skill) (MIT License).
## Boundaries and authorization
| Allowed | Prohibited |
|---|---|
| Diffing and testing builds in your own lab, on machines covered by written engagement scope, or inside CTF and benchmark ranges | Running a recovered PoC against systems you do not own, even when a program covers the underlying bug |
| Writing PoCs that demonstrate the bug (crash, hang, memory-corruption evidence) | Turning a PoC into a deliverable payload |
| Publishing or reporting with `{target}`, `{host}`, `{user}` placeholders | Publishing unredacted hostnames, IPs, or usernames |
If the environment lacks a tool a step needs (see Environment notes), report the gap and stop. Never simulate diff output or fabricate crash evidence.
## When to use
1. **Known CVE, no public PoC.** The advisory names the component and bug class but ships only the fix.
2. **Patch Tuesday triage.** Rank the month's kernel and driver fixes by likely exploitability before committing to deep analysis.
3. **Backport gap.** The mainline fix is merged but a distribution, LTS branch, or OEM kernel has not picked it up.
4. **Closed-source security patches.** Drivers, engines, virtualization components where the advisory is the only documentation.
## Core method
```text
patched build (after) unpatched build (before)
│ │
└───────── binary diff ─────────┘
│
function-level diff (matched / unmatched / changed)
│
focus on functions with moderate match scores (0.5 to 0.95)
│
read the new guards: bounds checks, locks, zeroing, refcounts
│
infer the bug class: OOB, race, UAF, info leak, int overflow
│
PoC on the unpatched build
│
verify: unpatched crashes, patched runs clean → bug confirmed
```
Patch pattern to bug class lookup:
| New guard in the patched build | Likely bug class |
|---|---|
| `if (a + b < a)`, `__builtin_add_overflow` | integer overflow |
| `KeAcquireSpinLock`, `mutex_lock` | race condition (TOCTOU, double free) |
| `if (idx >= MAX)`, `if (len > buf_size)` | out-of-bounds read or write |
| `RtlZeroMemory`, `memset(struct, 0, ...)` | uninitialized memory info leak |
| `InterlockedDecrement` plus refcount check | UAF, reference-count bug |
| `ProbeForRead` / `ProbeForWrite` | unvalidated user-mode pointer |
| `SeAccessCheck`, capability checks | missing permission check |
| removed or tightened IOCTL codes | attack-surface reduction (study the old path) |
## Workflow
**Step 1: obtain before and after builds.**
- Windows: Microsoft Update Catalog for the MSU packages of two builds (last unpatched, first patched); unpack with `expand.exe pkg.msu -F:* out\`, then the inner `.cab`; extract `ntoskrnl.exe`, `win32kfull.sys`, and the affected drivers.
- Linux: `apt download linux-image-<version>` for both builds, `dpkg-deb -x` each; restore the vmlinux ELF from `boot/vmlinuz` with extract-vmlinux; fetch the matching dbgsym package.
- Third-party software: vendor archives for the N-1 and N installers.
Record sha256 for every artifact; the report cites hashes, not filenames.
**Step 2: align symbols.**
- Windows: `symchk /v module.dll /s SRV*c:\sym*https://msdl.microsoft.com/download/symbols`.
- Linux: vmlinux plus `System.map` or dbgsym for both builds.
**Step 3: diff.**
- Preferred in this environment: `ghidriff old_build new_build -o out/` (Ghidra headless, pip-installable, markdown report).
- BinDiff or Diaphora only where an IDA/Ghidra toolchain with exports already exists.
**Step 4: locate the change.** Filter to functions with similarity roughly 0.5 to 0.95: identical functions are noise, completely different ones are usually new or renamed. Read the added conditions, loop bounds, and deleted blocks (a deletion is also a clue). Judge by control flow and data flow, not token-level diffs; patches carry unrelated cleanup and renames.
**Step 5: write and verify the PoC.**
- Integer overflow: boundary values (`INT_MAX-1`, `0xFFFFFFFF`).
- Race: concurrent threads hammering the open/close plus ioctl path.
- UAF: spray, free, reuse.
- OOB: length or index just past the guard.
Run the same harness against the patched build and confirm the clean result; an untested patched build proves nothing, since the crash could be environmental.
## Warnings
- **A patch can shrink the attack surface instead of fixing the bug.** The original flaw may still be reachable from another path; treat the fix as a lead, not ground truth.
- **Mitigations are not fixes.** CFG/XFG instrumentation such as `_guard_xfg_dispatch_icall_fptr` is hardening, not a root-cause fix.
- **Compiler churn is diff noise.** Compare builds from the same toolchain; never diff across major compiler versions.
- **Anonymize before anything leaves the lab.**
## Environment notes for this platform
- The rea MCP server (`mcp__rea__*`) opens files read-only on the host and never executes targets. Use it for structure, strings, and symbol questions on either build; files must live under the shared data mount and rea takes host-side paths.
- `ghidriff` needs Java plus a Ghidra install. Verify availability before promising a diff; if no diff toolchain exists, say so and stop.
- PoC execution belongs in an authorized lab VM, never in the task container.
## Completion self-check
- [ ] every workflow step was executed, not just read
- [ ] the diff, the inferred bug class, and the PoC artifacts are in the work dir with hashes
- [ ] the patched build was actually tested and its clean result recorded
+7
View File
@@ -818,6 +818,8 @@ export interface ProfileUsage {
output_tokens: number;
cache_read_tokens: number;
cache_write_tokens: number;
// EUR cost from the profile's stored prices; absent when the profile has no price set.
cost_eur?: number | null;
}
// One (profile, UTC day) token bucket for the dashboard's daily chart (new source).
@@ -1045,6 +1047,11 @@ export interface LLMProfile {
pool_exclude?: boolean;
// true (default) = streaming (SSE) | false = genuinely non-streaming (stream:false, one-shot response).
streaming?: boolean;
// Cost ledger: EUR per 1M tokens (0/absent = unset).
price_input_per_m?: number;
price_output_per_m?: number;
price_cache_read_per_m?: number;
price_cache_write_per_m?: number;
// Output cap per reply (tokens). 0 = do not send the field; the server default decides.
// Note the distinction from context_window_k: that is the model's total capacity, used locally only for the compression threshold.
max_tokens?: number;