The server wrote jwt.key to the executable's directory, /app in the
image, so it lived in the container's writable layer and every container
rebuild generated a new key and signed all users out.
- ARTEX_JWT_KEY_DIR selects the key directory (default: config.BaseDir(),
unchanged for non-Docker installs). docker-compose.yml sets it to
/app/keys and bind-mounts ./keys there; the Dockerfile declares the
volume. The key stays out of data/, which the file manager browses.
- loadOrCreateJWTKey creates the directory (0700). When keyDir/jwt.key is
missing it migrates dataDir/jwt.key, then BaseDir()/jwt.key (mode 0600,
old file removed), otherwise generates a new key. An existing key is
never overwritten; a short one is now a startup error.
- .gitignore and .dockerignore exclude keys/.
- README and CHANGELOG updated; the 0.5.0 known issue is removed.
The suite does not use one container port: most challenges publish 8080
but some (051, 091, 100) publish 5000, and the adapter had 8080
hardcoded, so deploy failed on those and the failure happened before
the teardown block, leaving the challenge running.
- the host port now comes from the app container port mapping (any
container port, via docker inspect), never from an assumed 8080
- the teardown finally now wraps the deploy as well: a failure halfway
through deploy no longer leaves containers behind (--keep and
--deploy-only still opt out)
The first live drost run died right after the task completed: the
findings detail route keys on finding_id (the findings-table row), not
on the exploration node id the list also carries, so every detail fetch
404ed; fetch_findings then re-raised the SystemExit from the API
wrapper, aborting scoring, the flag scan, the report and the teardown.
- details are fetched with finding_id (node id as fallback) and a
missing detail no longer kills the run: the list fields remain and the
finding stays gradeable
- the stored finding carries finding_id so the deep flag scan can
re-fetch untruncated text
- the deep scan tolerates the same failures
- teardown runs in a finally around the whole task phase, so a crash in
scoring or reporting can no longer leave the challenge running
(--keep still opts out deliberately)
The drost task goal template never included the target URL, so the
created task named no target: the first live run's planner matched the
challenge description against the global asset store and sent every
worker to an unrelated WordPress lab from past tasks. The template now
carries the {url} placeholder that create_task replaces with the task
target, and says the URL is the only target. The unit test locks the
placeholder in.
--llm-profiles passes llm_profile_ids to the created task (head first)
so a benchmark run exercises the model chain it claims to measure, and
the chain is stamped into results.json and the report.
--target-host rewrites the host in the task goal and company scope when
ARTEX runs in a container and the corpus runs on the host: the task must
reach the challenge via the host gateway (172.17.0.1), the same pattern
as the MCP server URLs. Local reachability checks still use 127.0.0.1.
DrostBench v1 ships 100 docker-compose web challenges, each with one
flag-bearing vulnerability plus decoys that do not reach the flag, so
flag capture is an objective win condition that no judge decides.
scripts/benchmark/drost.py drives one challenge: compose build with an
explicit per-build random flag (the suite's deterministic sha256 default
is never used, so a memorized flag cannot be replayed), up --wait, host
port discovery, down --rmi local. Ground truth is derived from the
challenge benchmark.json; the flag is scanned over the fetched findings
first and over full details, facts and the report only before declaring
a miss.
run_bench.py gains --corpus-type drost (one challenge per invocation),
flag capture in every drost report, a flag-only mode when no judge is
configured (gates reported as not measured, never as zero), and
--deploy-only for install validation. Image mode is unchanged.
test_drost.py covers the pure functions (flag formats, challenge id
validation, port parsing, ground-truth derivation incl. its refusals);
the lifecycle was exercised end to end on DBEN-v1-001: build with a
random flag, reachable app, compiled flag matching the override,
teardown without leftovers.
Closes#94
Model benchmarks measure recognition on snippets. Nothing measures what this
product actually is: finding and proving vulnerabilities in a running
application, unprompted, against an executable oracle. This adds that
harness as scripts, with no server changes: deploy a corpus container,
create a real task through the public API, wait, fetch the findings, judge
them against version-stamped ground truth with an independent LLM judge,
tally with measurement discipline.
scripts/benchmark/run_bench.py plus ground_truth/juice-shop.json (8 stable
entries: SQLi login bypass, admin section access, basket IDOR, /ftp
exposure, forged feedback, error disclosure, token tampering, XSS) and a
README documenting the standing rules:
- multi-gate scoring (component 1 / class 2 / keywords 1 / proof 1),
partial credit, gate pattern over aggregate
- findings beyond ground truth are discovery candidates; precision is
reported as bounds until verified
- the judge must not be the model under test (refuses to start, override
knowingly), and must pass a falsification battery before grading
- invalid findings (errored or empty) are excluded from every rate and
named, never counted as misses
- every rate carries its denominator, the aggregate a Wilson 95% interval;
results are stamped with method + ground-truth versions
The task goal bakes in the standing quality gates, so the benchmark
measures the product as it behaves, prompt rules included.
Methodology adapted from capability-matrix (MIT) and VulnHunter's
benchmark harness (Apache-2.0).
BaaRai