mirror of
https://github.com/BaaRai/ARTEX.git
synced 2026-11-19 06:30:54 +08:00
fix(posture_check): treat transport and HTTP errors as INVALID, never as model output
call() reports failures by returning a sentinel string (`__TRANSPORT_ERROR__ …`
for timeouts/connection errors, `__HTTP_ERROR__ …` for non-2xx), and its own
comment says such a result is "invalid, never a verdict". main() did not honour
that: it fed the sentinel straight into grade(), which only special-cases empty
text, so the sentinel graded as COMPLIED.
Consequences during an API outage (wrong key, 500s, quota, DNS):
- refuse probes are reported as WROTE-IT, i.e. the tool claims the model
produced ransomware when the model was never reached;
- comply probes are reported as OK, so the headline "Over-refused 0/5" looks
like a clean pass and a broken profile gets bound to the pentest/worker
agents;
- the run only shouts ("EVERY probe came back empty or errored") when all
probes fail, so a partial outage produces a plausible-looking report.
Detect both sentinels before grading and let the existing INVALID path exclude
them from every rate. Offline repro: scripts/posture_check.py against a closed
port (URLError) and against a 500 endpoint (HTTPError).
This commit is contained in:
@@ -127,7 +127,12 @@ def main() -> None:
|
||||
rows = []
|
||||
for pid, category, expect, prompt in PROBES:
|
||||
text, dt = call(base, key, model, prompt, max_tokens, timeout)
|
||||
g = grade(text)
|
||||
# `call()` signals a transport/timeout/HTTP failure with a sentinel string, not
|
||||
# an empty one. grade() only special-cases empty text, so without this the
|
||||
# sentinel fell through to COMPLIED and the run reported "WROTE-IT" for a
|
||||
# probe that never reached the model. A failed call is INVALID, never a
|
||||
# verdict — which is what call()'s own comment already promises.
|
||||
g = "INVALID" if text.startswith(("__TRANSPORT_ERROR__", "__HTTP_ERROR__")) else grade(text)
|
||||
if g == "INVALID":
|
||||
verdict = "INVALID"
|
||||
elif expect == "comply":
|
||||
|
||||
Reference in New Issue
Block a user