Compare commits

..
Author SHA1 Message Date
BaaRai b3717f7a5f release: v0.4.0
First tagged release of this fork. Adds the 0.4.0 changelog section covering the work since upstream 0.3.15, guards the Docker Hub publishing job behind a repository variable (no credentials are configured for that namespace), and stops auto-generated release notes in favour of the changelog body.
2026-10-23 14:40:39 +02:00
BaaRai 5d3b1d142d Merge pull request #125 from stevenAthompson/chore/remove-self-update
chore: remove automatic update check and in-app self-update
2026-10-23 14:28:51 +02:00
stevenAthompson 2d5f9677f4 chore: remove automatic update check and in-app self-update
The update source is hard-coded to a repository that no longer exists and the release checksums are unsigned. Removes the selfupdate package and the /api/update/* endpoints, the top-bar check that queried GitHub on every page load, the Version & Updates card, and the exit-75 swap protocol in the start scripts.
2026-10-22 23:58:37 -04:00
BaaRai faf76d5f10 Merge pull request #123 from BaaRai/docs/ai-pr-review-technology
docs: AI PR review REA analysis and ARTEX integration plan
2026-10-22 14:51:28 +02:00
BaaRai 27d2241cda docs: cover the platform's continuous red-teaming offering 2026-10-22 14:49:52 +02:00
BaaRai 70f9ea3f01 docs: add AI PR review REA analysis and ARTEX integration plan 2026-10-22 14:42:56 +02:00
BaaRai 271deabeb0 Merge pull request #121 from BaaRai/chore/norma-v0.4.4
chore: bump norma to v0.4.4 (fork synced to upstream tag)
2026-10-22 13:07:26 +02:00
BaaRai e6b4f6c5f4 chore: bump norma to v0.4.4 (fork synced to upstream tag)
Syncs BaaRai/Norma to upstream Autumn-27/Norma v0.4.4 (fast-forward, our
earlier changes were already merged upstream) and repins the replace from
the pre-merge pseudo-version to the tag.

What v0.4.4 brings for us: per-task process-tree RSS watchdog (kills the
process group over a memory cap, matters now that workers spawn nuclei,
ffuf, sqlmap and bbot), UTF-8-aligned output truncation, non-interactive
default env for subprocesses (no more tools hanging on hidden stdin reads),
input usage recovery from message_delta for accurate token accounting, and
streaming / thought_signature fixes. Full test suite green against a
scratch postgres.
2026-10-22 13:04:12 +02:00
BaaRai c7bef98689 Merge pull request #118 from BaaRai/feat/bbot-orchestrator
feat: bbot recursive recon orchestrator (toolbox layer + policy-gated skill)
2026-10-22 12:07:53 +02:00
BaaRai 4c62b57bcc feat(skills): bbot recursive recon orchestrator with policy gates
Toolbox layer pins bbot v3.0.2 (AGPL-3.0, same license as ARTEX; standalone
CLI dependency, not linked code) in its own venv with module dependencies
preinstalled via --install-all-deps, so scans never need pip under a program
egress allowlist. Build-time gate dry-runs a dep-heavy module sample twice:
the first primes the dependency check, the second must be free of Installing
and ERRR lines or the build fails.

The bbot-recon skill gates usage on program policy: passive zero-touch mode
via seeds-only scans (-s, no -t) when scanners are banned, active-safe and
loud module sets when the program allows them, scope discipline with
blacklist and strict-scope, --no-deps mandatory on every run, JSONL ingestion
into facts and finding candidates with bbot findings treated as leads that
still need independent verification.

Closes #117
2026-10-22 12:03:24 +02:00
BaaRai b5bee2b8d2 Merge pull request #116 from BaaRai/fix/toolbox-tar-paths
fix(image): toolbox tar extraction paths + dalfox version flag
2026-10-21 15:55:20 +02:00
25 changed files with 588 additions and 2513 deletions
+7 -2
View File
@@ -95,13 +95,18 @@ jobs:
files: |
dist/*.zip
dist/SHA256SUMS
generate_release_notes: true
generate_release_notes: false
body: "See CHANGELOG.md for the list of changes in this release."
# ④ Multi-arch image → push to Docker Hub (reuses the Linux binaries built by the
# binaries job; the image only installs tools + ships the binary; arm64 only needs
# to emulate the apt layer, so builds are much faster)
# Publishing the image needs Docker Hub credentials and a namespace we own, so the
# job stays skipped unless the repository defines DOCKERHUB_ENABLED (set it to true
# and provide DOCKERHUB_USERNAME / DOCKERHUB_TOKEN secrets plus DOCKERHUB_IMAGE).
docker:
needs: binaries
if: ${{ vars.DOCKERHUB_ENABLED == 'true' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4 # fetch the Dockerfile and skills/
@@ -122,7 +127,7 @@ jobs:
- uses: docker/metadata-action@v5
id: meta
with:
images: autumn27/artex
images: ${{ vars.DOCKERHUB_IMAGE }}
tags: |
type=ref,event=tag
type=raw,value=latest
+36 -1
View File
@@ -4,6 +4,40 @@ All notable changes to this project are documented in this file. The format is b
## [Unreleased]
## [0.4.0] - 2026-10-23
First release of this fork. The engine underneath is still upstream's, but this tree carries a long stretch of operational work on top of it: a recon toolbox and skills, findings discipline, cost accounting, and container packaging.
### Added
- **Recon toolbox in the image**: pinned nuclei, httpx, subfinder, katana, ffuf, dalfox and a sqlmap checkout, verified at build time, so hunt workers can call them directly (#112).
- **Recursive recon skill** (`bbot-recon`): bbot v3.0.2 in its own virtualenv with module dependencies preinstalled. Policy-gated modes (passive seeds-only when scanners are banned, active-safe, loud only on explicit permission) and JSONL ingestion into the facts graph (#118).
- **php2shell skill**: turns a file-inclusion finding into verified RCE through the session.upload_progress chain, lab first and gated on program policy (#113).
- **Code audit and reverse-engineering skills** (#55, #77).
- **Goal ambiguity gate**: task creation rejects goals that name no target and goals under 80 characters; `[targetless]` opts out for tasks that legitimately have none (#111).
- **LLM chain hardening**: consecutive-failure escalation across the ordered profile chain, plus per-intent profile rotation with a strict round-robin (`llm_rotate`) (#111, #114).
- **Stagnation warning**: the graph overview tells a worker when it has repeated the same direction without producing findings (#111).
- **Per-task credentials vault**: accounts stored per task, agent-generated passwords allowed, injected into the worker context and managed from the task page.
- **Cost ledger**: token spend metered per call, with usage surfaced in the UI and judge/fallback calls counted separately (#71, #74).
- **Backup script and MCP health reporting** (#43, #44).
- **Ask-operator tool with environment handoff** (#45, #57).
- **DrostBench corpus and benchmark harness** (#59, #95, #96, #97, #98).
- **One database per test package**, removing cross-package test races (#82).
### Changed
- **Norma upgraded to v0.4.4**: per-task process-tree memory watchdog, UTF-8 aligned output truncation, non-interactive subprocess environments, message_delta usage recovery, and streaming fixes (#121).
- **Planner quiet mode** for less narration during long runs (#81, #88).
- **Findings discipline**: proof-basis gates, measured impact, adversarial verification and dedup rules decide what a finding must contain before it is reported (#51, #53, #63, #69).
- **Prompt-injection hardening** on seed material and MCP tool descriptions (#49).
- **Pause preserves the task timeout budget**, and paused time is frozen in the task clock (#2, #65, #110).
- **Task detail metrics and outcome cards** summarise a finished run (#92, #102).
- **Skill-gap, loop-contract and shadow-lesson reporting** for the planner (#104, #106, #108).
### Removed
- **In-app self-update and the automatic version check**: the update source was hard-coded to a repository that no longer exists, and its checksums came from the same release. The `selfupdate` package, the `/api/update/*` endpoints, the top-bar update badge and the "Version & Updates" settings card are gone; `start.sh` / `start.bat` now only restart the process after a crash. Upgrade with `update.sh` or a release archive. (#125, contributed by @stevenAthompson)
## [0.3.15] - 2026-10-07
### Interception
@@ -886,7 +920,8 @@ All notable changes to this project are documented in this file. The format is b
- [@Autumn-27](https://github.com/Autumn-27)
- [@neouks](https://github.com/neouks)
[Unreleased]: https://github.com/Autumn-27/ARTEX/compare/v0.3.10...HEAD
[Unreleased]: https://github.com/BaaRai/ARTEX/compare/v0.4.0...HEAD
[0.4.0]: https://github.com/BaaRai/ARTEX/compare/v0.3.15...v0.4.0
[0.3.10]: https://github.com/Autumn-27/ARTEX/compare/v0.3.9...v0.3.10
[0.3.9]: https://github.com/Autumn-27/ARTEX/compare/v0.3.8...v0.3.9
[0.3.8]: https://github.com/Autumn-27/ARTEX/compare/v0.3.7...v0.3.8
+18 -2
View File
@@ -55,6 +55,23 @@ RUN set -eux; \
nuclei -version >/dev/null; httpx -version; subfinder -version; katana -version; \
ffuf -V >/dev/null; dalfox --version >/dev/null 2>&1 || true; sqlmap --version >/dev/null
# bbot recursive recon orchestrator (#117): blacklanternsecurity/bbot, AGPL-3.0
# (same license as ARTEX; standalone CLI dependency, not linked code). Pinned
# version, own venv to isolate from the image python. Module dependencies are
# preinstalled (--install-all-deps) so scans never need pip at runtime, which
# would fail under a program's egress allowlist.
ARG BBOT_VERSION=3.0.2
RUN set -eux; \
python -m venv /opt/bbot; \
/opt/bbot/bin/pip install --no-cache-dir "bbot==${BBOT_VERSION}"; \
ln -s /opt/bbot/bin/bbot /usr/local/bin/bbot; \
bbot -y --install-all-deps; \
bbot --version >/dev/null; \
bbot --dry-run -t example.com -m wafw00f,paramminer_getparams,lightfuzz,sslcert,gowitness >/dev/null 2>&1 || true; \
bbot --dry-run -t example.com -m wafw00f,paramminer_getparams,lightfuzz,sslcert,gowitness --no-color >/tmp/bbot-dry.log 2>&1 || true; \
! grep -qE "Installing|ERRR|CRITICAL" /tmp/bbot-dry.log || { cat /tmp/bbot-dry.log; exit 1; }
# Preinstall Playwright MCP and CLI (globally) so no npx network download is needed at runtime.
# @playwright/mcp: browser MCP runs directly via `npx @playwright/mcp` (installed globally, no -y/@latest needed).
# @playwright/cli: provides playwright-cli, verified with --help right after install.
@@ -68,8 +85,7 @@ RUN npm install -g @playwright/mcp@latest @playwright/cli@latest playwright@late
WORKDIR /app
# pre-built binary for the matching architecture (dist/amd64/artex or dist/arm64/artex)
COPY dist/${TARGETARCH}/artex /app/artex
# Daemon startup script: decides whether to relaunch the process based on its exit code;
# the in-page one-click update relies on it to swap the binary. It also forwards SIGTERM
# Daemon startup script: relaunches the process after a crash and forwards SIGTERM
# to artex — docker stop only signals PID 1; without forwarding, artex never receives it,
# cannot shut down gracefully, and is hard-killed by SIGKILL after 10 seconds.
COPY start.sh /app/start.sh
+5 -18
View File
@@ -115,7 +115,7 @@ cp config.example.json config.json # fill in the database connection
./start.sh # → http://localhost:8787
```
> Please start with `start.sh` / `start.bat` instead of running `./artex` directly. It is a daemon script: after the program exits it decides whether to relaunch based on the exit code, and **the in-page [one-click update](#option-1-one-click-update-from-the-page-recommended) relies on it to swap the binary**. If you run `./artex` directly, it will not be relaunched after an update.
> Please start with `start.sh` / `start.bat` instead of running `./artex` directly. It is a daemon script: if the process crashes, it restarts it after a backoff.
> To keep it running in the background: `nohup ./start.sh >artex.log 2>&1 &`.
### Option 4: build the single binary from source
@@ -175,20 +175,7 @@ Daily cron example (03:20, operator crontab):
> Upgrading only swaps the program and never touches data: the Postgres volume `pgdata`, `./data` (jwt.key / SQLite, etc.), and `./skills` are all preserved. **No manual database migration is needed**: `artex` idempotently re-runs `schema.sql` on every start (with `ADD COLUMN` / `CREATE INDEX IF NOT EXISTS`), i.e. "restart equals migrate". It is still recommended to back up `./data` and the database before upgrading.
### Option 1: one-click update from the page (recommended)
On the **System Settings** page (sidebar "System Settings" → `/system/settings`), the **Version & Updates** card lets you check for and install new versions directly, without logging into the server.
After clicking "Update": it downloads the release package for the current platform → verifies it against the release's `SHA256SUMS` → smoke-tests the new binary with `-h` → stages it as `artex.new` → the program exits and is relaunched by `start.sh` / `start.bat`, which completes the swap. The page automatically waits for the new version to come online, then refreshes.
- **A failure never leaves a broken program**: if verification or the smoke test fails, the staged file is discarded and the current version keeps running; if the swapped-in new version fails to start 3 times in a row, it automatically rolls back to `artex.old` (the failed one is kept as `artex.failed` for troubleshooting).
- **Rollback anytime**: the previous version is kept as `artex.old`, and the card has "Roll back to previous version". Note that the database schema is not rolled back.
- **Updating interrupts running tasks**: updating means restarting, so do it when the system is idle.
- **Dev builds can't update**: updating is disabled when the version is `dev` or a `git describe` output with a suffix, preventing a release from overwriting your local debug binary.
- **Under Docker, only the program is swapped, not the image**: toolchains inside the image like playwright / nmap are not upgraded along with it, and recreating the container with `docker compose up -d` reverts to the version bundled in the image. To upgrade the image as well, still use `docker compose pull artex && docker compose up -d artex`.
- If accessing GitHub requires a proxy, configure the **global proxy** on the same page and the update chain will use it. Updates only download from GitHub domains and enforce HTTPS.
### Option 2: one-click update script
### Option 1: one-click update script
```bash
cd ARTEX
@@ -200,7 +187,7 @@ The script first optionally `git pull`s the latest code, then lets you choose **
- **① Docker**: you can specify a target image tag (press Enter to keep `.env`'s `ARTEX_TAG`, default `latest`) → `docker compose pull` → `docker compose up -d` (restarting with the new image migrates automatically).
- **② Local**: rebuild the frontend static assets → recompile `./artex` (restart the process for it to take effect).
### Option 3: Docker Compose (manual)
### Option 2: Docker Compose (manual)
```bash
cd ARTEX
@@ -211,7 +198,7 @@ docker compose up -d artex # restart with the new image → schema migrates
docker image prune -f # clean up old images (optional)
```
### Option 4: prebuilt binaries (Releases)
### Option 3: prebuilt binaries (Releases)
Download the new version's zip from [Releases](https://github.com/BaaRai/ARTEX/releases), stop the old process, then overwrite `artex` and `skills/` (keep your `config.json` and `data/`), and restart:
@@ -220,7 +207,7 @@ cp -r <extracted dir>/skills ./ && cp <extracted dir>/artex ./
./start.sh
```
### Option 5: build from source
### Option 4: build from source
```bash
git pull
+5 -32
View File
@@ -18,7 +18,6 @@ import (
"github.com/Autumn-27/artex/agent"
"github.com/Autumn-27/artex/config"
"github.com/Autumn-27/artex/selfupdate"
"github.com/Autumn-27/artex/server"
)
@@ -42,10 +41,8 @@ func printBanner(addr string) {
version, runtime.GOOS, runtime.GOARCH, runtime.Version(), addr)
}
// main only maps run's result onto the process exit code. The exit code is part
// of the update protocol — the supervising start script reads it to decide
// whether to relaunch us (see selfupdate.ExitRestart) — so the body has to live
// in a function that can *return* rather than os.Exit past its own defers.
// main maps run's result onto the process exit code. The body lives in a
// function that can *return* rather than os.Exit, so its own defers still run.
func main() {
os.Exit(run())
}
@@ -68,16 +65,6 @@ func run() int {
// page can show a live log stream. Do this first, to catch startup logs too.
server.StartLogCapture()
// Self-update bootstrap: swap in a staged binary, or count a post-swap boot
// attempt and roll back if the new build keeps dying. Must run before we open
// the stores or bind a port — this may end with "exit and let the start script
// relaunch me", and there is no point paying for either first.
action, upState := selfupdate.Bootstrap()
server.SetBootUpdateState(upState)
if action == selfupdate.Restart {
return selfupdate.ExitRestart
}
// surface which config file the binary reads (absolute, so `go run`'s relative
// "config.json" — resolved against the CWD — is unambiguous).
cfgPath := config.Path()
@@ -101,12 +88,6 @@ func run() int {
}
defer mgr.Close()
// Surviving this long means a freshly swapped-in build actually works, so drop
// the upgrade marker and stop counting attempts. Until it fires, every boot
// increments the count and a build that keeps dying gets rolled back.
settle := time.AfterFunc(selfupdate.SettleDelay, selfupdate.Settle)
defer settle.Stop()
skillDir := config.SkillDir()
if abs, err := filepath.Abs(skillDir); err == nil {
skillDir = abs
@@ -126,22 +107,14 @@ func run() int {
}
}()
// Two ways out: a signal (normal stop → exit 0, the start script stops looping)
// or a staged update / rollback (→ exit 75, the script relaunches us and the
// bootstrap above installs the new build).
code := 0
select {
case <-ctx.Done():
case <-server.RestartRequested():
code = selfupdate.ExitRestart
shutdown(agent.AbortShutdown)
}
// Wait for a stop signal.
<-ctx.Done()
log.Println("shutting down...")
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_ = httpSrv.Shutdown(shutdownCtx)
return code
return 0
}
// shutdownContext deliberately does not derive from signalCtx. If it did, the
+402
View File
@@ -0,0 +1,402 @@
# AI PR Review: REA Analysis and ARTEX Integration Plan
Status: research + design. No source code, prompts, or proprietary assets from the reference product are reproduced here. The analysis is based on a public client bundle observed with REA, the authenticated UI of a trial organization, and the vendor's public documentation and marketing pages. Field and enum names below come from generated client schemas and are evidence of the wire/data model, not a copy of server code.
Date: 2026-10-22
Target: the reference web application
Tooling: REA 6.7.0 (`analyze-web-bundle`, `analyze-javascript-application`, `capture-web-screenshot`, `list-browser-targets`), Chrome 155 over CDP, the vendor's public documentation.
## 1. TL;DR
The reference PR check is not a single "LLM reviews the diff" prompt. The client data model exposes a pipeline with these distinct stages:
1. **Webhook-triggered, policy-gated PR scan** — runs on PR/MR open or update, skips drafts/forks/outside contributors, caps changed files at 500, and honors repo config and exclusions.
2. **Repository context assembly** — repo indexing (symbols, call graph, token counts), an auto-generated threat model, uploaded context documents, and repository rules.
3. **Candidate discovery** — static data-flow/taint analysis plus LLM review agents; findings carry `taint_path`, `affected_code`, `affected_file`, `cwe_id`, and `owasp_category`.
4. **Validation** — a validation agent produces `verified_claims` / `unverified_claims` and may perform optional dynamic verification in a sandbox, yielding a `dynamic_verdict` and `dynamic_evidence`.
5. **Dedup + triage** — fingerprint/embedding-based duplicate handling, `detection_count`, and states such as `true_positive`, `false_positive`, `accepted_risk`, `resolved`.
6. **GitHub output** — inline PR comments plus a check run with either annotations or status-only output; the check fails only when `fail_on.severity` is configured and an open/valid finding meets it.
7. **Remediation** — an agent validates exploitability, writes a minimal fix, runs tests, adds a regression test, and opens a PR. A retest later marks fixes `verified_fixed` or `not_fixed`.
For ARTEX, the highest-value additions are the **PR eligibility/diff layer**, **code-context index + taint candidates**, **validator pass**, **dedup/triage lifecycle**, and **check-run publisher**. The autonomous pentest graph and existing finding/retest machinery can be reused rather than duplicated.
### 1.1 Platform context
The vendor's public homepage markets three offerings. The PR check is only the first; the other two matter because they share the finding, triage, and verification model.
| Area | Offering | What is public |
| --- | --- | --- |
| Code security | PR review | Reviews every PR/MR, validates vulnerabilities before merge, inline comments and checks. This document reconstructs it. |
| Product security | Whitebox pentest | Combines repository context with active testing against target URLs and auth instructions; one free retest; heavier use of dynamic verification. |
| Organization security | Continuous red teaming | "Simulate a persistent adversary." Advertised as a product area, but no public technical documentation describes scheduling, scope, agents, or output. |
The homepage also markets two cross-product loops: auto-triage/autofix to verify findings and prepare tested fixes, and self-improving threat models based on triage feedback. Both are consistent with the client schemas and the remediation workflow below, but the red-teaming internals are not observable from public evidence.
## 2. Evidence and confidence
| Evidence | What it establishes |
| --- | --- |
| REA `analyze_web_bundle` on the authenticated app (135 scripts, 64 routes), Evidence `ev_455acd3a8b10796167f2499ac63b1e131a63e48e9696065a8b6633c6f9a47b3b` | Route map, script inventory, authenticated app shell, settings and copy |
| REA `analyze_javascript_application` (127 files, 6,877 findings), Evidence `ev_f0d14c017237436a907624c1d8f4db8b7a81ee9309cfa2eb9cead52f6e9dcf9b` | Static module graph and endpoint/storage observations |
| Generated protobuf-es parsers/serializers in `_nuxt/ba334sRx.js` | Full client-side wire model for findings, scans, PR checks, fixes, validation, and repo indexing |
| Public docs: `code-review/overview`, `code-review/config`, `code-review/rules`, `code-review/findings-feedback`, `platform/automations/remediation`, `white-box-pentest/overview`, `api-reference/scans/create-scan` | Trigger behavior, gating semantics, review limits, remediation phases, scan request fields |
| Authenticated UI copy | Empty-state wording, settings, triage/fix/retest actions, auth-instruction handling |
| Vendor marketing homepage (EN/FR) | Three product areas, including continuous red teaming; cross-product auto-triage/autofix and self-improving threat-model claims; no red-team internals |
**Proven:** route and field existence; enums and state names; PR check/comment/gating behavior as documented; remediation phases; 500-file limit; config/rules behavior; validation/dynamic-verification fields.
**Inferred:** the exact ordering and implementation of static taint analysis, LLM prompting, and validation; whether every PR scan performs dynamic testing; which parsing/static-analysis engines are used server-side.
**Unknown:** server implementation, model providers, prompt text, rule engine internals, the internal scoring/ranking formula, and the continuous red-teaming pipeline (scheduling, scope, agent topology, output). The client exposes only `Default` / `Legacy` model sets and cost/token accounting, not provider names.
## 3. What the PR check does
### 3.1 Trigger and eligibility
- On GitHub, the integration is a GitHub App; GitLab/Bitbucket/Azure DevOps have equivalent flows. Webhooks deliver on PR/MR open or update. A delivery-status model (`Webhooks delivering` / `failing` / `Webhook status unavailable`) is surfaced in the UI.
- PR Review is enabled per repository. Drafts and PRs from forks or outside contributors can be skipped without a comment; other skips post the reason on the PR while comments are enabled.
- Each PR/MR scan supports **up to 500 changed files**; larger PRs are skipped (public docs).
- A force-review mention command (for example, a bot mention in a PR comment) forces a review when config would otherwise exclude the PR.
- The check record carries `checkRunId`, `headSha`, `installationId`, `prNumber`, repository owner/name, PR author identity, a `prReviewReactionId`, and a status enum of `pending / in_progress / completed / failed / cancelled / skipped`.
### 3.2 Configuration and gating
The repository config file is read from the **default branch** so a PR cannot change its own rules. It supports:
- `exclude.{branches,labels,keywords,paths,authors}` and `include.{branches,labels,...}`. Path exclusion applies only when **every** changed file matches.
- `fail_on.severity`: `critical | high | medium | low | info`. Without it, the check stays green when the scan completes; findings are comments only. With it, the check fails when an **open or confirmed-valid** finding is at or above the threshold.
- Repo config wins per category; unset categories inherit organization defaults; `[]` clears an org default.
- Invalid config fails open (missing/malformed ignored; unknown keys skipped; a type mismatch discards the file).
The repository rules file is separate from scope: it supplies app-specific context (trusted services, authz patterns, risk areas, scoped exclusions) to reduce false positives.
Triage updates the check immediately: marking a gated finding `false_positive` or `accepted_risk` can make the check pass; reopening it makes it fail again. Feedback commands on GitHub/GitLab include `!fp`, `!accepted_risk`, `!valid`, `!fixed`.
### 3.3 Review output
- Inline comments are posted on affected code by default.
- GitHub check output mode is `annotations` or `status-only`.
- Findings posted as PR comments; when source-control comments are off, findings stay in the platform while scans and check statuses still update.
- For public repositories, outside-diff findings hide sensitive details (title, description, PoC, file locations); inline findings on changed code remain visible.
- The scan record tracks `pr_number`, `pr_url`, `pr_lines_added`, `pr_lines_removed`, `scan_type` (`pr` or `full`).
### 3.4 Relation to the other product areas
- **Whitebox pentest** uses the same finding and verification vocabulary (`scan_type: full`, target URLs, auth instructions, `dynamic_verdict`, `dynamic_evidence`, retest). The PR check can be treated as the continuous, diff-scoped path through the same platform.
- **Continuous red teaming** is marketed as "simulate a persistent adversary" but has no public technical documentation. It may share the finding, triage, and threat-model model, but that is unverified; do not assume its scanner or scheduling model in an implementation.
- The shared loop is finding -> triage feedback -> threat-model update -> better review/pentest signal. The client exposes threat-model timestamps and triage activity, not the update algorithm.
## 4. Reconstructed vulnerability-detection pipeline
```mermaid
flowchart TB
W["GitHub/GitLab webhook<br/>PR opened / updated"] --> E{"Eligibility<br/>draft? fork? excluded? >500 files?"}
E -->|skip| SK["Check: skipped<br/>(reason in status)"]
E -->|run| CTX["Assemble context<br/>diff @ head SHA + repo index<br/>+ threat model + rules.md + prior triage"]
CTX --> SA["Static analysis<br/>symbols, call graph,<br/>taint/data-flow rules"]
CTX --> LLM["LLM review agents<br/>changed hunks + repository context"]
CTX --> DEP["Dependency verdicts<br/>known-malicious / OSV feeds"]
SA --> CAND["Candidate findings<br/>file, code, taint path, CWE/OWASP"]
LLM --> CAND
DEP --> CAND
CAND --> VAL["Validator agent<br/>verified/unverified claims"]
VAL --> DYN{"requires verification<br/>and auth/target available?"}
DYN -->|yes| DVER["Sandbox/browser verification<br/>confirmed / partial / not reproduced / blocked"]
DYN -->|no| DEDUP
DVER --> DEDUP["Dedup + fingerprint<br/>detection_count, duplicate_of, similarity"]
DEDUP --> PUB["Publish<br/>inline comments + check run<br/>(annotations or status-only)"]
PUB --> GATE{"fail_on severity<br/>and open/valid finding?"}
GATE -->|yes| RED["Check: failure"]
GATE -->|no| GREEN["Check: success"]
PUB --> TRIAGE["Triage / feedback<br/>!valid !fp !accepted_risk !fixed"]
TRIAGE --> PUB
TRIAGE --> FIX["Remediation agent<br/>validate → minimal fix → tests → PR"]
FIX --> RETEST["Retest<br/>verified_fixed / not_fixed"]
```
### 4.1 Context assembly
The scan record and client model show repository indexing with `symbol_count`, `total_repo_tokens`, `unsupported_languages`, `commit_sha`, `from_cache`, predicted cost, and a provider/status field. A separate call-graph download exists for scans. A threat model is generated automatically the first time a PR is reviewed in a repo (or when context documents are uploaded), and can be synthesized at the application level across repos. Reviews also consume the repository rules file and prior triage feedback.
Likely purpose: give the reviewer a compact, semantically indexed view of the changed code plus reachability context (call graph, threat model) instead of raw repository text.
### 4.2 Static candidate discovery
The finding schema is the strongest evidence of data-flow analysis:
- `affected_file`, `affected_code`
- `taint_path` (ordered collection)
- `cwe_id`, `owasp_category`
- `category`, `title`, `description`, `recommendation`
- `proof_of_concept`
- `scan_pr_number`, `scan_pr_url`, `scan_type`
`taint_path` plus the downloadable call graph implies source-to-sink data-flow tracking, not just pattern matching. The exact rules engine is not exposed; candidates may come from built-in rules, repository rules, and LLM review.
### 4.3 LLM review and validation
Client strings and schemas expose:
- "Review agent", "Agents", "Delegating to subagent", "Delegated to subagent".
- "Starting validation agent...", "View validation activity", "Validated during triage".
- A validation object: `verified_claims`, `unverified_claims`, `evidence_summary`, `validation_output`, `blocker_detail`.
- Review summaries with `conclusion`, `riskRating`, `riskRationale`, `narrativeMarkdown`, `nextSteps`, `severityCounts`, and `model`.
The safest reconstruction is a two-pass design: a **reviewer** proposes findings from the diff plus repository context, and a **validator** attempts to falsify or confirm each claim against code and evidence. Only validated claims are eligible for gating.
### 4.4 Optional dynamic verification
The scan model contains `requires_verification`, `has_auth_instructions`, `auth_verification_status`, and `auth_verification_failure_reason`. Findings carry `dynamic_verdict` (`confirmed`, `partially_confirmed`, `not_reproduced`, `blocked`) and `dynamic_evidence`. UI copy warns that auth instructions are shared with AI agents for testing and recommends a test environment.
This is a sandbox/browser verification stage. It is optional and more prominent in whitebox pentests, but the PR-scan schema supports it.
### 4.5 Dedup and triage
Finding records include `detection_count`, `duplicate_of`, `first_seen_at`, `last_seen_at`, `first_seen_scan_id`, `state`, `verification_status`, and `latest_state_change`. One variant includes an `embedding` and `similarity`, indicating semantic duplicate matching. States:
- Finding state: `open`, `true_positive`, `false_positive`, `accepted_risk`, `resolved`, `closed`.
- Triage verification: `pending`, `approved`, `rejected`.
- Dynamic verdict: `confirmed`, `partially_confirmed`, `not_reproduced`, `blocked`.
### 4.6 Remediation
The remediation docs and fix schema agree:
1. Validate the finding: test exploitability; if unreachable/sound, mark `false_positive` instead of patching.
2. Implement the minimal correct fix.
3. Run the repo's test suite, add relevant regression tests, and verify the fix.
4. Commit to a fix branch and open a PR/MR.
Fix record: `base_commit_sha`, `unified_diff`, `commit_message`, `pr_title`, `pr_body`, `reason`, `summary`, `cost_usd`, `task_id_cli`, and verification `{notes, reproduced, regression_test_added, test_command, test_output}`. Fix status: `dispatched`, `pr_created`, `no_fix`, `false_positive`, `failed`, `stopped_cap_exceeded`, `pr_closed`. Verification: `not_requested`, `pending`, `verified_fixed`, `not_fixed`.
## 5. Data model worth mirroring in ARTEX
These are client field names, not a server schema. They are still a useful target model.
| Concept | Important fields |
| --- | --- |
| PR check task | `checkRunId`, `headSha`, `installationId`, `prNumber`, `repositoryOwner`, `repositoryName`, `pr_author_login`, `prReviewReactionId`, `status` |
| Scan | `scan_type` (`pr`/`full`), `pr_number`, `pr_url`, `branch`, `commit`, `current_step`, `display_status`, `model` (`default`/`legacy`), `requires_verification`, `has_auth_instructions`, `triage_enabled`, `findings_summary`, `credits_used`, `pr_lines_added/removed` |
| Finding | `affected_file`, `affected_code`, `taint_path[]`, `cwe_id`, `owasp_category`, `severity`, `category`, `title`, `description`, `recommendation`, `proof_of_concept`, `detection_count`, `duplicate_of`, `state`, `verification_status`, `dynamic_verdict`, `dynamic_evidence`, `scan_pr_number/url`, `first_seen_at`, `last_seen_at`, `resolved_at` |
| Validation | `verified_claims[]`, `unverified_claims[]`, `evidence_summary`, `validation_output`, `blocker_detail` |
| Review summary | `conclusion`, `riskRating`, `riskRationale`, `narrativeMarkdown`, `nextSteps`, `severityCounts`, `model` |
| Repo indexing | `symbol_count`, `total_repo_tokens`, `unsupported_languages[]`, `provider`, `from_cache`, `commit_sha`, predicted cost bounds |
| Fix | `fix{base_commit_sha, unified_diff}`, `pr_title`, `pr_body`, `commit_message`, `status`, `verification{reproduced, regression_test_added, test_command, test_output, notes}` |
Scan status values observed: `pending`, `running`, `completed`, `failed`, `stopped`, `cancelled`, `skipped`, `pending_verification`, `pending_triage`, `draft`, `estimating`, `estimated`.
## 6. How the check decides pass/fail
The check is deliberately **advisory by default**. A completed scan is green; findings do not block. `fail_on.severity` opts into gating. Only open or confirmed-valid findings count. Triage immediately recomputes the check, and the config is read from the default branch.
For ARTEX this means the check runner must separate:
- **Scan outcome** (completed/failed/cancelled/skipped) from
- **Gate outcome** (success/failure based on policy + current finding disposition).
Keeping those independent avoids the common mistake of failing a check because the scanner crashed, or passing it because a finding was dismissed in the platform but not recomputed.
## 7. ARTEX integration blueprint
ARTEX today is an autonomous pentest system: Go backend, Next.js UI, PostgreSQL, dual graph (assets + exploration), planner/worker agents, traffic proxy, findings, and retests. It does **not** have a GitHub PR review subsystem. The following design adds one without turning ARTEX into a full static-analysis platform on day one.
### 7.1 Design principles
- Reuse findings/retests/notifications; don't create a parallel vulnerability model.
- Treat repository content and PR text as untrusted input (prompt injection, secrets).
- Separate deterministic gates (eligibility, config, fail_on) from probabilistic detection (LLM/static candidates).
- Make every scan idempotent by `(repository, head_sha)`.
- Keep the check advisory by default; gating is explicit policy.
- Never execute untrusted repository code on the host. Dynamic verification runs in the existing sandbox/proxy boundary.
### 7.2 Proposed package layout
```
codereview/
webhook.go GitHub/GitLab webhook verification and event normalization
model.go Scan, CheckRun, PullRequest, Finding, FixRun, enums
diff.go unified-diff parsing, hunk→file mapping, 500-file cap
config.go .artex/config.yaml (repository policy)
index.go repository snapshot + symbol/call-graph index
rules.go .artex/rules.md, org/repo rules, path/author/label filters
analyze.go deterministic rules + taint candidate extraction
review.go LLM reviewer agent (diff + context)
validate.go validator agent (claims, reachability, false-positive checks)
dynamic.go optional verification through ARTEX target/traffic tooling
dedup.go fingerprints + embeddings + duplicate_of
publish.go GitHub check run + inline annotations/comments
fix.go remediation run and fix-PR creation
retest.go post-fix verification and state transitions
```
Each package should expose interfaces, not global state, so the pipeline can run in the existing engine or as a separate worker.
### 7.3 Storage
Add a small code-review schema rather than overloading `findings`:
- `cr_repositories`: provider, installation id, owner/name, enabled, default branch, config hash, index state.
- `cr_pull_requests`: repo id, number, head SHA, base SHA, author, draft/fork flags, state, `check_run_id`, `check_conclusion`, `check_output_mode`, `last_event_at`.
- `cr_scans`: PR id, head SHA, status, current step, model set, token/cost counters, started/finished, error.
- `cr_findings`: scan id, fingerprint, file, line range, taint path JSONB, CWE, OWASP, severity, title, description, recommendation, PoC, detection count, duplicate_of, state, verification status, dynamic verdict/evidence, resolved_at.
- `cr_fix_runs`: finding id, status, base commit, unified diff, test command/output, reproduced/regression flags, PR URL, cost.
- `cr_feedback`: finding id, actor, command/state, reason, created_at, check recomputed_at.
Alternatively, extend `findings` with nullable `source`, `file`, `line`, `fingerprint`, `cwe`, `owasp`, `taint_path`; but a separate table keeps pentest findings and code findings cleanly separable and avoids changing the existing API contract.
### 7.4 API surface
- `POST /api/webhooks/github` — verify `X-Hub-Signature-256`, normalize `pull_request` events, enqueue.
- `POST /api/codereview/pull-requests/{id}/review` — force review (equivalent to a bot mention in a PR comment).
- `GET /api/codereview/pull-requests/{id}` — status, check, findings.
- `GET /api/codereview/findings` — filters: repo, PR, severity, state, CWE/OWASP.
- `PATCH /api/codereview/findings/{id}` — triage state, severity, reason; recompute check.
- `POST /api/codereview/findings/{id}/fix` — start remediation (idempotent).
- `POST /api/codereview/findings/{id}/retest` — after a fix.
- `GET /api/codereview/repos/{id}/index` — index stats and unsupported languages.
Server routes in ARTEX use Go 1.22+ `mux.HandleFunc("METHOD /path/{id}", ...)`; follow that style.
### 7.5 PR lifecycle
1. Verify webhook signature; reject replayed deliveries using the delivery id.
2. Upsert `cr_pull_requests` by `(repo, number)`; ignore events for an already-scanned `head_sha`.
3. Load config from the **base/default branch**; evaluate include/exclude; skip drafts/forks/outside contributors; enforce the changed-file cap.
4. Create `cr_scan` with status `pending`; set check to `in_progress` with a stable `details_url`.
5. Fetch diff and base/head metadata; parse hunks; map findings to changed line ranges.
6. Build/refresh the repo index (incremental by commit; cache by `(repo, commit)`).
7. Run deterministic rules and taint extraction; run the LLM reviewer; merge candidates by fingerprint.
8. Validate each candidate; drop unreachable/false candidates; optionally run dynamic verification.
9. Dedup, assign display IDs, persist findings.
10. Publish inline comments (respecting public-repo redaction) and a check run (`annotations` or `status-only`).
11. Compute the gate from policy + open/valid findings; transition check to `completed`.
12. On triage, recompute the gate and optionally delete/update the stale comment.
13. On `!fixed` or a new commit, schedule retest; on `verified_fixed`, mark `resolved`.
### 7.6 Agent roles and prompts
- **Reviewer** (`codereview/review.go`): input = changed hunks + file summaries + call-graph neighborhood + threat model + rules. Output = candidate findings with file/line, category, severity, CWE/OWASP guess, evidence, and a proposed taint path.
- **Validator** (`codereview/validate.go`): input = candidate + surrounding code + reachability facts + prior feedback. Output = `verified_claims`, `unverified_claims`, `evidence_summary`, `validation_output`, `blocker_detail`, and a final disposition suggestion. The validator must be able to say "not exploitable" and suppress the finding.
- **Fixer** (`codereview/fix.go`): input = validated finding + repo snapshot. Output = minimal unified diff, commit message, PR title/body, test command, and verification notes.
- **Retester** (`codereview/retest.go`): re-run the reproduction or the relevant tests against the fixed commit; produce `verified_fixed` or `not_fixed`.
Use ARTEX's existing `llmpool` for provider selection, per-run token budgets, retries, and usage recording. Store model/provider metadata and cost per scan, matching the reference product's `LLM calls` / `Tokens (in/out)` visibility.
### 7.7 Static analysis for a Go codebase
A pragmatic first pass:
- **Tree-sitter** parsers for the languages you support; persist symbols, imports, and call edges.
- **Call graph** from definitions + references (start intra-repo; mark unresolved dynamic dispatch as unknown).
- **Taint rules** for common sources/sinks (HTTP handlers, SQL, template rendering, command execution, file paths, deserialization) with explicit sanitizers.
- **Custom rules** from `.artex/rules.md` and repo config; allow path/label/author exclusions.
- **Dependency verdicts**: optional OSV/advisory lookup plus a known-malicious package feed, surfaced as a separate finding category.
Do not claim full-program reachability when the call graph is partial. Record `unknown` for unresolved edges; this mirrors the reference product's `unsupported_languages` and call-graph limitations.
### 7.8 Check publisher
- Use GitHub Checks API with `name`, `head_sha`, `status`, `conclusion`, `output.title`, `output.summary`, and annotations (file, start_line, end_line, annotation_level, message).
- `status-only` mode omits annotations and only reports the summary/check state.
- `fail_on.severity` maps to the gate; severity order: `critical > high > medium > low > info`.
- On skip, post `conclusion: skipped` with a reason when comments are enabled.
- Public repositories: redact title/description/PoC/file locations for findings outside the diff; keep inline findings on changed lines visible.
- Recompute the check on triage, not only on scan completion.
### 7.9 Remediation and retest
- Only validated findings are eligible for remediation.
- Create a dedicated branch; never push to the PR head by default.
- Run the repository's existing test command in the sandbox; add a regression test only when the repository has a clear test convention.
- Store the unified diff, test command/output, and reproduction result. Open the PR/MR with a body that links the finding and the evidence.
- If the source PR merges mid-run, retarget the fix to the default branch (the vendor's documented behavior).
- A retest re-runs the reproduction/tests; `verified_fixed` marks the finding resolved, `not_fixed` keeps it open with evidence.
### 7.10 Security, cost, and limits
- **Prompt injection**: repository files, PR descriptions, and rules are untrusted; never let them override system instructions. Strip secrets before sending to a model.
- **Code execution**: verification runs in ARTEX's existing sandbox/proxy boundary; do not execute repo code on the server host.
- **Public repos**: apply redaction before publishing comments; a public comment can leak a vulnerability.
- **Idempotency**: `(repo, head_sha)` is the scan key; triage commands are idempotent by finding/state.
- **Cost control**: per-PR token and credit budget; fail closed to `skipped` when the budget is exhausted rather than scanning partially and publishing a misleading check.
- **Config trust**: read `.artex/config.yaml` from the default branch, not the PR head; invalid config fails open.
- **Audit**: store the check conclusion, policy threshold, finding dispositions, and recomputation timestamps.
### 7.11 Reuse map for ARTEX
| ARTEX component | Reuse |
| --- | --- |
| `db/findings.go`, `findings` table | Status vocabulary, lists, grouping, notifications; either extend or mirror in `cr_findings` |
| `db/finding_retests.go`, `agent/retester.go` | Retest lifecycle and `reproduced`/`fixed`/`inconclusive` verdicts |
| `server/finding_workflow.go` | Tool wiring for agent-driven finding updates |
| `llmpool`, `agent/provider.go` | Model selection, retries, quota, token accounting |
| `traffic/`, `intercept/` | Dynamic verification evidence and HTTP replay |
| `notify/` | Slack/email alerts for new gated findings |
| `guard/` | Tool approval and execution boundary |
| `web/` | New Code Review pages and PR detail views |
### 7.12 Phased delivery
**Phase 0 — Ingest and publish (no detection).**
GitHub App/webhook, repository config, eligibility, diff parse, check run with `skipped`/`in_progress`/`completed`, idempotency by head SHA. Acceptance: a PR with no findings gets a green check; an excluded PR gets `skipped` with the reason; a duplicate webhook does not create a second scan.
**Phase 1 — Deterministic rules and comments.**
Tree-sitter index, a small taint ruleset, inline annotations, `fail_on` gate, triage recompute. Acceptance: a seeded vulnerable handler is flagged at the right file/line; a `false_positive` triage turns the check green; a `fail_on: critical` policy fails only on critical/high.
**Phase 2 — LLM reviewer + validator.**
Reviewer proposes candidates; validator produces verified/unverified claims; merge by fingerprint; cost/token accounting. Acceptance: a false candidate is suppressed by the validator; the scan stays green; token usage is recorded per scan.
**Phase 3 — Dedup, feedback, and operations.**
Embedding/fingerprint dedup, `detection_count`, duplicate links, feedback commands (`!fp`, `!valid`, `!accepted_risk`, `!fixed`), Slack notifications, dashboard. Acceptance: the same finding across commits is deduped; feedback survives rescans; the dashboard shows fix rate and suppression rate.
**Phase 4 — Remediation and retest.**
Fix agent, unified diff, test execution, fix PR, retest. Acceptance: a validated finding produces a PR with a regression test; retest marks `verified_fixed`; a failed fix stays open with test output.
**Phase 5 — Dynamic verification.**
Target URLs + auth instructions, browser verification, dynamic verdict/evidence, public-repo redaction. Acceptance: a flagged SSRF/SQLi candidate is confirmed or refuted against a sandbox target; evidence is attached; no secrets leak into comments.
### 7.13 Acceptance criteria for "PR check works"
- Webhook → check in under a bounded time; idempotent retries.
- Check is advisory by default; `fail_on` is the only blocker.
- Findings map to real changed lines and include file, severity, category, evidence.
- False positives can be dismissed from GitHub without visiting the platform, and the check updates.
- Public comments never disclose PoC/secrets for outside-diff findings.
- Every scan records model, token/cost, commit, and policy used.
- No repository code executes outside the sandbox.
- All state transitions are visible and auditable.
## 8. Mapping: Reference behavior → ARTEX implementation
| Reference behavior | ARTEX implementation |
| --- | --- |
| GitHub App webhook + check run | New `codereview/webhook.go` + `publish.go`; GitHub App installation stored in `cr_repositories` |
| Repo indexing + call graph | `codereview/index.go`; tree-sitter symbols + call edges; persist in `cr_symbols`/`cr_call_edges` |
| Threat model + rules | Reuse ARTEX context documents/task context; add `.artex/rules.md` loader |
| Taint findings | `codereview/analyze.go` taint rules; store `taint_path` JSONB |
| Reviewer/validator agents | `codereview/review.go` + `validate.go`; reuse `llmpool` |
| Dynamic verification | `codereview/dynamic.go`; reuse `traffic`/intercept + sandbox |
| Triage state machine | Reuse `findings` status vocabulary or `cr_findings` with the same states |
| Remediation | `codereview/fix.go`; branch/PR creation; run tests in sandbox |
| Retest | Reuse `finding_retests` pattern; add `verified_fixed`/`not_fixed` |
| Slack notifications | Reuse `notify` |
## 9. Open questions
- Which static-analysis engine does the reference product use server-side? The client proves taint/call-graph data but not the engine.
- Are PR scans always LLM-reviewed, or only when static candidates exist? The model and agent fields suggest LLM review is normal.
- How are repository embeddings stored and refreshed? `embedding`/`similarity` exist in one finding variant, but the index internals are unknown.
- What is the exact ranking/scoring formula for PR risk? Only `riskRating`, `severityCounts`, and review narrative are exposed.
- How does dynamic verification authenticate to customer apps? `auth_instructions` (max 2,000 chars) and `auth_verification_status` are exposed; the browser automation is not.
- What are the model providers behind `Default`/`Legacy`? Not present in the client bundle.
- How does continuous red teaming schedule runs, choose scope, and hand findings into triage? The homepage advertises it, but no public documentation or client surface explains it.
## 10. Sources
- REA Evidence: `ev_455acd3a8b10796167f2499ac63b1e131a63e48e9696065a8b6633c6f9a47b3b` (authenticated bundle), `ev_f0d14c017237436a907624c1d8f4db8b7a81ee9309cfa2eb9cead52f6e9dcf9b` (static app graph), `ev_bb2c67fec41a43f211b557d91c9aff64e11b349cf641ca34ebbcc624a81d2d99` (dashboard screenshot).
- Vendor docs: `code-review/overview`, `/code-review/config`, `/code-review/rules`, `/code-review/findings-feedback`, `/platform/automations/remediation`, `/white-box-pentest/overview`, `/api-reference/scans/create-scan`, `/api-reference/hfw/get-package-verdict`.
- Client bundle: the app's `_nuxt/ba334sRx.js` bundle and related chunks; generated message fields/enums extracted statically. Minified function names are not original type names.
## 11. License and use
This document is original analysis for interoperability and research. The reference product is proprietary; do not copy its code, prompts, or assets. The reconstructed field names are factual observations of a public client schema. Any ARTEX implementation should be written independently against this behavioral specification and reviewed for license/legal compatibility before redistribution. Continuous red teaming is advertised as a third offering but has no public technical documentation; this document treats its internals as unknown and out of scope.
+1 -1
View File
@@ -61,4 +61,4 @@ require (
modernc.org/memory v1.11.0 // indirect
)
replace github.com/Autumn-27/norma => github.com/BaaRai/Norma v0.4.4-0.20261011224847-63b38dabc837
replace github.com/Autumn-27/norma => github.com/BaaRai/Norma v0.4.4
+2 -2
View File
@@ -1,5 +1,5 @@
github.com/BaaRai/Norma v0.4.4-0.20261011224847-63b38dabc837 h1:slZeNksaiCKS4XZSNnNIlmyqnfgDT/PGoPjOPrijnoE=
github.com/BaaRai/Norma v0.4.4-0.20261011224847-63b38dabc837/go.mod h1:BudWrf7iHl96FgHqFr/HwN3mKYmGjJDt+xL2ZHXfFZw=
github.com/BaaRai/Norma v0.4.4 h1:x5UAeaTuhREI1GiIYuwIEybRxr/pLCeCUvXmUbwv5Is=
github.com/BaaRai/Norma v0.4.4/go.mod h1:BudWrf7iHl96FgHqFr/HwN3mKYmGjJDt+xL2ZHXfFZw=
github.com/andybalholm/brotli v1.2.1 h1:R+f5xP285VArJDRgowrfb9DqL18yVK0gKAW/F+eTWro=
github.com/andybalholm/brotli v1.2.1/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3dyBCFEj5IhUbnKptjxatkF07cF2ak3yi77so=
-313
View File
@@ -1,313 +0,0 @@
package selfupdate
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"log"
"os"
"os/exec"
"strings"
"time"
)
// smokeEnv lets the child process launched by the smoke test skip Bootstrap entirely.
//
// Strictly speaking it would be fine without it: the child's os.Executable() is
// artex.new, so every derived path carries the .new infix and can never touch the
// real upgrade files. But relying on that coincidence is too fragile — an explicit
// short-circuit is obvious at a glance and saves the child a pointless disk probe.
const smokeEnv = "ARTEX_SELFUPDATE_SMOKE"
// Action is Bootstrap's instruction to main.
type Action int
const (
// Continue: start the server as usual.
Continue Action = iota
// Restart: exit immediately with ExitRestart so the supervisor script relaunches us.
Restart
)
// State describes the upgrade state at this startup, so /api/update/check can
// faithfully tell the frontend "did the last upgrade succeed or was it rolled back".
type State struct {
Pending bool // swapped but not yet confirmed stable
RolledBack bool // this startup just performed an automatic rollback
FailedStage bool // staged file failed verification/smoke test and was discarded
Detail string // one-sentence user-facing explanation
}
// Bootstrap runs at the very beginning of main, and must be called before any port
// is listened on or any database is opened.
//
// Three situations:
//
// ① staged file artex.new exists → verify + smoke test; on success swap and ask
// for a restart; on failure discard it and keep
// running the old version
// ② only the marker file remains → the swap just happened; count one attempt;
// roll back after enough consecutive failures
// ③ nothing → normal startup
func Bootstrap() (Action, State) {
if os.Getenv(smokeEnv) != "" {
return Continue, State{}
}
p, err := ResolvePaths()
if err != nil {
log.Printf("[update] skipping bootstrap: %v", err)
return Continue, State{}
}
if _, err := os.Stat(p.New); err == nil {
return applyStaged(p)
}
m, ok := readMarker(p.Marker)
if !ok {
return Continue, State{}
}
return confirmOrRollback(p, m)
}
// applyStaged handles the "staged file exists" situation: swap if it verifies,
// discard it if not.
//
// This is the only place in the whole upgrade chain that overwrites the executable,
// and the final gate — the smoke test stops problems like a corrupted download,
// a wrong architecture choice, or missing dynamic libraries. Once a binary that
// cannot run is let through, the supervisor script will tirelessly keep relaunching
// it while the Go code never gets a chance to run, making automatic rollback impossible.
func applyStaged(p Paths) (Action, State) {
m, _ := readMarker(p.Marker)
if err := verifyStaged(p); err != nil {
log.Printf("[update] staged new version failed verification, discarded, continuing with the current version: %v", err)
cleanStaged(p)
_ = os.Remove(p.Marker)
return Continue, State{FailedStage: true, Detail: "new version failed verification and was discarded: " + err.Error()}
}
if err := swap(p); err != nil {
log.Printf("[update] swap failed, continuing with the current version: %v", err)
cleanStaged(p)
_ = os.Remove(p.Marker)
return Continue, State{FailedStage: true, Detail: "swap failed: " + err.Error()}
}
// Swap succeeded. Keep the marker; the next startup (running the new version) confirms whether it is stable.
m.Attempts = 0
if m.StagedAt == 0 {
m.StagedAt = time.Now().Unix()
}
if err := writeMarker(p.Marker, m); err != nil {
log.Printf("[update] failed to write upgrade marker (automatic rollback capability lost): %v", err)
}
log.Printf("[update] swapped to %s, exiting to restart (exit %d)", orUnknown(m.To), ExitRestart)
return Restart, State{Pending: true}
}
// confirmOrRollback handles "the startup after a swap": counts attempts, and swaps
// the old version back once the limit is exceeded.
//
// The counter only increments after the Go code is running, so it covers failures
// of the "executable but crashes during initialization" kind (incompatible config,
// port occupied, DB migration blew up); "cannot exec at all" is stopped by the
// smoke test before the swap — together they are complete.
func confirmOrRollback(p Paths, m marker) (Action, State) {
m.Attempts++
if m.Attempts > maxAttempts {
if err := rollback(p); err != nil {
// If even the rollback fails, do not restart again, or we fall into an
// infinite restart loop. Remove the marker and let the process start
// as-is — if it cannot come up, the user can at least see why in the logs.
log.Printf("[update] new version failed to start %d times in a row, and rollback failed too: %v", maxAttempts, err)
_ = os.Remove(p.Marker)
return Continue, State{Detail: "new version failed to start and rollback failed: " + err.Error()}
}
log.Printf("[update] new version failed to start %d times in a row, rolled back to %s, exiting to restart (exit %d)",
maxAttempts, orUnknown(m.From), ExitRestart)
_ = os.Remove(p.Marker)
return Restart, State{RolledBack: true, Detail: fmt.Sprintf("new version failed to start, rolled back to %s", orUnknown(m.From))}
}
if err := writeMarker(p.Marker, m); err != nil {
log.Printf("[update] failed to update upgrade marker: %v", err)
}
log.Printf("[update] new version starting (attempt %d/%d); the upgrade will be confirmed once it runs stably",
m.Attempts, maxAttempts)
return Continue, State{Pending: true}
}
// Settle confirms the new version has been running stably and clears the upgrade marker.
//
// Called by main with a delay after the HTTP listener is up: only surviving that
// period counts; otherwise the marker stays in place and the next startup keeps
// counting attempts until a rollback is triggered.
func Settle() {
p, err := ResolvePaths()
if err != nil {
return
}
settle(p)
}
func settle(p Paths) {
if _, ok := readMarker(p.Marker); !ok {
return // not a post-upgrade startup, nothing to do
}
if err := os.Remove(p.Marker); err != nil && !errors.Is(err, os.ErrNotExist) {
log.Printf("[update] failed to clear upgrade marker: %v", err)
return
}
log.Printf("[update] new version is running stably, upgrade complete (previous version kept as %s)", p.Old)
}
// SettleDelay is the runtime duration required to decide "the new version survived".
const SettleDelay = 30 * time.Second
// verifyStaged verifies the staged file: compare SHA256 first, then actually launch it once.
func verifyStaged(p Paths) error {
want, err := os.ReadFile(p.Sum)
if err != nil {
return fmt.Errorf("failed to read checksum: %w", err)
}
got, err := fileSHA256(p.New)
if err != nil {
return fmt.Errorf("failed to compute checksum: %w", err)
}
if !strings.EqualFold(strings.TrimSpace(string(want)), got) {
return errors.New("SHA256 mismatch (download corrupted or tampered with)")
}
return smokeTest(p.New)
}
// smokeTest launches the new binary with -h to confirm it can really execute on
// the current system. This stops a whole class of problems: truncated downloads,
// wrong architecture (exec format error), missing dependencies, and so on.
func smokeTest(bin string) error {
if err := os.Chmod(bin, 0o755); err != nil {
return fmt.Errorf("failed to grant execute permission: %w", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, bin, "-h")
cmd.Env = append(os.Environ(), smokeEnv+"=1")
out, err := cmd.CombinedOutput()
if ctx.Err() != nil {
return errors.New("smoke test timed out (new binary unresponsive)")
}
if err != nil {
snippet := strings.TrimSpace(string(out))
if len(snippet) > 300 {
snippet = snippet[:300] + "…"
}
return fmt.Errorf("smoke test failed: %v: %s", err, snippet)
}
return nil
}
// swap replaces the current binary with the staged new version.
//
// Both Unix and Windows allow renaming a running executable (what Windows forbids
// is deleting and overwriting; rename is not on that list), so there is no need to
// split by platform here, nor to stop ourselves first.
func swap(p Paths) error {
// Windows rename does not overwrite an existing target, so the .old left by the previous upgrade must be removed first.
if err := os.Remove(p.Old); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("failed to clean up old backup %s: %w", p.Old, err)
}
if err := os.Rename(p.Current, p.Old); err != nil {
return fmt.Errorf("failed to back up the current version: %w", err)
}
if err := os.Rename(p.New, p.Current); err != nil {
// The swap failed but the current version has already been moved away; it must be put back as-is, or the next startup has no executable.
if rerr := os.Rename(p.Old, p.Current); rerr != nil {
return fmt.Errorf("failed to install the new version (%v), and restoring the current version failed too: %w", err, rerr)
}
return fmt.Errorf("failed to install the new version: %w", err)
}
_ = os.Remove(p.Sum)
return nil
}
// rollback swaps back the old version that swap backed up.
func rollback(p Paths) error {
if _, err := os.Stat(p.Old); err != nil {
return fmt.Errorf("no backup to roll back to %s: %w", p.Old, err)
}
// Move the unstartable new version to .failed for troubleshooting instead of deleting it outright.
failed := p.Current + ".failed"
_ = os.Remove(failed)
if err := os.Rename(p.Current, failed); err != nil {
return fmt.Errorf("failed to move the failing version away: %w", err)
}
if err := os.Rename(p.Old, p.Current); err != nil {
return fmt.Errorf("failed to restore the old version: %w", err)
}
return nil
}
// Rollback implements /api/update/rollback: actively return to the previous version.
// It only does the swap; the restart is likewise left to the supervisor script (the caller then exits with ExitRestart).
func Rollback() error {
p, err := ResolvePaths()
if err != nil {
return err
}
if _, err := os.Stat(p.Old); err != nil {
return errors.New("no previous version to roll back to (" + p.Old + " does not exist)")
}
cleanStaged(p)
if err := smokeTest(p.Old); err != nil {
return fmt.Errorf("the previous version cannot execute, refusing to roll back: %w", err)
}
// Swap current and backup: after the rollback you can still roll forward again.
tmp := p.Current + ".swap"
_ = os.Remove(tmp)
if err := os.Rename(p.Current, tmp); err != nil {
return fmt.Errorf("failed to move the current version away: %w", err)
}
if err := os.Rename(p.Old, p.Current); err != nil {
_ = os.Rename(tmp, p.Current)
return fmt.Errorf("failed to install the previous version: %w", err)
}
if err := os.Rename(tmp, p.Old); err != nil {
log.Printf("[update] failed to tidy up the backup after rollback (does not affect operation): %v", err)
}
_ = os.Remove(p.Marker)
return nil
}
// HasBackup reports whether a previous version exists to roll back to, so the frontend can decide whether to show the rollback button.
func HasBackup() bool {
p, err := ResolvePaths()
if err != nil {
return false
}
_, err = os.Stat(p.Old)
return err == nil
}
func fileSHA256(path string) (string, error) {
f, err := os.Open(path)
if err != nil {
return "", err
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
return "", err
}
return hex.EncodeToString(h.Sum(nil)), nil
}
func orUnknown(s string) string {
if strings.TrimSpace(s) == "" {
return "unknown version"
}
return s
}
-143
View File
@@ -1,143 +0,0 @@
package selfupdate
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/url"
"strings"
"time"
)
// Repo is the release source. Hardcoded rather than configurable: a configurable
// update source hands anyone who can change the config a remote code execution
// channel — a door a penetration testing platform must never open.
const Repo = "Autumn-27/artex"
// latestURL is GitHub's "latest stable release" endpoint. It automatically skips prereleases and drafts.
const latestURL = "https://api.github.com/repos/" + Repo + "/releases/latest"
// allowedHosts restricts the domains the upgrade chain may visit. Together with
// checkRedirect below, any hop redirected to a host outside this list fails
// immediately — this is the first gate against DNS poisoning / MITM swapping out
// the binary; the second is the SHA256SUMS comparison.
var allowedHosts = map[string]bool{
"api.github.com": true,
"github.com": true,
"objects.githubusercontent.com": true, // the object storage where release assets actually land
"release-assets.githubusercontent.com": true,
"raw.githubusercontent.com": true,
}
// Release holds the fields we care about in a GitHub Release.
type Release struct {
TagName string `json:"tag_name"`
Name string `json:"name"`
Body string `json:"body"`
Draft bool `json:"draft"`
Prerelease bool `json:"prerelease"`
PublishedAt time.Time `json:"published_at"`
HTMLURL string `json:"html_url"`
Assets []Asset `json:"assets"`
}
// Asset is a file attached to a Release.
type Asset struct {
Name string `json:"name"`
URL string `json:"browser_download_url"`
Size int64 `json:"size"`
}
// NewClient builds an HTTP client that only trusts GitHub domains. Direct
// connection when proxy is empty.
//
// Deliberately does not reuse the default Transport: the upgrade chain must force
// TLS with certificate verification, and must not be affected by things like
// InsecureSkipVerify set elsewhere.
func NewClient(proxy string) *http.Client {
tr := &http.Transport{
ForceAttemptHTTP2: true,
TLSHandshakeTimeout: 15 * time.Second,
}
if p := strings.TrimSpace(proxy); p != "" {
if pu, err := url.Parse(p); err == nil {
tr.Proxy = http.ProxyURL(pu)
}
}
return &http.Client{
Transport: tr,
Timeout: 30 * time.Minute, // downloading a full package must not be killed by a request-level timeout
CheckRedirect: func(req *http.Request, via []*http.Request) error {
if len(via) >= 10 {
return fmt.Errorf("too many redirects")
}
return checkURL(req.URL)
},
}
}
// checkURL enforces https + a domain whitelist.
func checkURL(u *url.URL) error {
if u.Scheme != "https" {
return fmt.Errorf("refusing non-HTTPS address: %s", u.Scheme+"://"+u.Host)
}
if !allowedHosts[strings.ToLower(u.Hostname())] {
return fmt.Errorf("refusing non-GitHub domain: %s", u.Hostname())
}
return nil
}
// FetchLatest queries the latest stable release.
func FetchLatest(ctx context.Context, c *http.Client) (*Release, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, latestURL, nil)
if err != nil {
return nil, err
}
if err := checkURL(req.URL); err != nil {
return nil, err
}
req.Header.Set("Accept", "application/vnd.github+json")
req.Header.Set("User-Agent", "artex-selfupdate")
resp, err := c.Do(req)
if err != nil {
return nil, fmt.Errorf("failed to reach GitHub (a global proxy can be configured in system settings): %w", err)
}
defer resp.Body.Close()
switch {
case resp.StatusCode == http.StatusForbidden, resp.StatusCode == http.StatusTooManyRequests:
// Unauthenticated GitHub API allows 60 requests per IP per hour; easy to hit when sharing an egress IP.
return nil, fmt.Errorf("GitHub API rate limit (60 per hour), please try again later")
case resp.StatusCode == http.StatusNotFound:
return nil, fmt.Errorf("repository %s has no stable release yet", Repo)
case resp.StatusCode != http.StatusOK:
return nil, fmt.Errorf("GitHub returned %d", resp.StatusCode)
}
var rel Release
if err := json.NewDecoder(resp.Body).Decode(&rel); err != nil {
return nil, fmt.Errorf("failed to parse Release: %w", err)
}
if strings.TrimSpace(rel.TagName) == "" {
return nil, fmt.Errorf("Release is missing a tag")
}
return &rel, nil
}
// AssetName returns the release package name for the current platform, matching
// build.sh's package_binary: artex-<version>-<os>-<arch>.zip (version without the v prefix).
func AssetName(tag, goos, goarch string) string {
return fmt.Sprintf("artex-%s-%s-%s.zip", strings.TrimPrefix(tag, "v"), goos, goarch)
}
// FindAsset finds an asset by name in the Release.
func (r *Release) FindAsset(name string) (Asset, bool) {
for _, a := range r.Assets {
if strings.EqualFold(a.Name, name) {
return a, true
}
}
return Asset{}, false
}
-175
View File
@@ -1,175 +0,0 @@
// Package selfupdate implements ARTEX's one-click update from the web UI: it pulls
// the new binary from a GitHub Release, verifies and stages it, and swaps it in atomically on the next start.
//
// Overall division of labor (see start.sh / start.bat):
//
// Start script = a dumb supervision loop, responsible only for "deciding whether to relaunch the process after it exits, based on its exit code"
// This package = all the error-prone logic (download / SHA256 verify / smoke test / swap / rollback on failure)
//
// The swap lives in Go rather than the scripts because sha256 verification and the smoke test would need two implementations in sh and bat,
// (sha256sum / shasum / certutil), and this is precisely the step that must never fail — swap in a binary that cannot run,
// and the supervisor will faithfully relaunch it forever, leaving the user to rescue things on the machine by hand.
//
// A full upgrade goes through three process starts:
//
// (1) the old server receives /api/update/apply → download and verify → stage artex.new → exit 75
// (2) the script relaunches the old version → Bootstrap spots artex.new → verify + smoke test → swap → exit 75
// (3) the script relaunches again, now the new version → Bootstrap records one attempt → clears the marker after a successful start
//
// Any failed step falls back to the old version: in (2) a failed verification deletes the staged file and keeps the old version running; in (3) three
// consecutive crashes before the marker is cleared mean the new version cannot start, so artex.old is swapped back automatically.
package selfupdate
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
)
// ExitRestart is the "please relaunch me" exit code (EX_TEMPFAIL). When the start script sees it,
// it reruns immediately without counting toward crash backoff. 0 means the user stopped it normally (the script exits the loop); anything else is treated as a crash.
const ExitRestart = 75
// maxAttempts is the number of start attempts allowed after the swap. Every start of the new version increments the counter; surviving
// settleDelay clears the marker; maxAttempts consecutive crashes mean the new version simply cannot start, so it rolls back automatically.
const maxAttempts = 3
// Paths lists every file an upgrade touches, all anchored under **the executable's directory**.
// CWD is deliberately avoided: when running as a service the working directory may be / or any path, and using CWD would drop the staged files
// somewhere else, breaking the swap logic outright.
type Paths struct {
Dir string // directory of the executable
Current string // currently running binary artex / artex.exe
New string // staged new version artex.new / artex.new.exe
Sum string // new version sha256 (hex) artex.new.sha256 / artex.new.exe.sha256
Old string // pre-swap backup of old version artex.old / artex.old.exe
Marker string // upgrade state marker artex.upgrade.json
}
// ResolvePaths derives all upgrade paths from the current executable.
//
// On Windows .new/.old must also carry the .exe suffix, otherwise both the smoke test and post-swap execution fail,
// so the suffix is stripped off first and re-appended, keeping the naming symmetric across the two platforms.
func ResolvePaths() (Paths, error) {
exe, err := os.Executable()
if err != nil {
return Paths{}, fmt.Errorf("locate executable: %w", err)
}
if resolved, err := filepath.EvalSymlinks(exe); err == nil {
exe = resolved
}
dir := filepath.Dir(exe)
name := filepath.Base(exe)
ext := filepath.Ext(name) // ".exe" on Windows, usually empty on Unix
stem := strings.TrimSuffix(name, ext)
join := func(suffix string) string { return filepath.Join(dir, stem+suffix+ext) }
return Paths{
Dir: dir,
Current: exe,
New: join(".new"),
Sum: join(".new") + ".sha256",
Old: join(".old"),
Marker: filepath.Join(dir, stem+".upgrade.json"),
}, nil
}
// marker records the progress of one swap and triggers automatic rollback when the new version cannot start.
type marker struct {
From string `json:"from"` // version before the upgrade
To string `json:"to"` // target version
Attempts int `json:"attempts"` // start attempts made after the swap
StagedAt int64 `json:"staged_at"`
}
func readMarker(path string) (marker, bool) {
b, err := os.ReadFile(path)
if err != nil {
return marker{}, false
}
var m marker
if json.Unmarshal(b, &m) != nil {
return marker{}, false
}
return m, true
}
func writeMarker(path string, m marker) error {
b, err := json.MarshalIndent(m, "", " ")
if err != nil {
return err
}
return os.WriteFile(path, b, 0o644)
}
// cleanStaged removes the staged files. Successful swap, failed verification, and user cancellation all go through it, so a leftover
// artex.new is not retried on the next start.
func cleanStaged(p Paths) {
_ = os.Remove(p.New)
_ = os.Remove(p.Sum)
}
// CompareVersions compares two version numbers and returns -1/0/1 (a<b / a==b / a>b).
// ok=false means at least one side is not a comparable version number (such as a local dev build's "dev" or
// a git describe output like "0.3.7-2-gabc1234-dirty"); the caller must disable one-click update then,
// otherwise a dev build would be "upgraded" to a release, overwriting uncommitted changes.
func CompareVersions(a, b string) (int, bool) {
av, aok := parseVersion(a)
bv, bok := parseVersion(b)
if !aok || !bok {
return 0, false
}
for i := range 3 {
if av[i] != bv[i] {
if av[i] < bv[i] {
return -1, true
}
return 1, true
}
}
return 0, true
}
// parseVersion parses a "v0.3.7" / "0.3.7" style version number into [3]int.
//
// Only a clean three-part form is accepted: for non-tag builds build.sh produces suffixed
// versions via git describe like "0.3.7-2-gabc1234"; those must be judged non-comparable rather than treated as
// 0.3.7 — otherwise a dev build gets misjudged as "already latest" or overwritten by a release.
func parseVersion(s string) ([3]int, bool) {
s = strings.TrimSpace(s)
s = strings.TrimPrefix(s, "v")
if s == "" {
return [3]int{}, false
}
parts := strings.Split(s, ".")
if len(parts) != 3 {
return [3]int{}, false
}
var out [3]int
for i, p := range parts {
n, err := strconv.Atoi(p)
if err != nil || n < 0 {
return [3]int{}, false
}
out[i] = n
}
return out, true
}
// InDocker reports whether the process runs inside a container. Under Docker the swap writes to the container's writable layer,
// and `docker compose up -d` recreating the container reverts to the version bundled in the image — that is expected behavior
// (the user is pulling a new image at that point anyway), but the frontend needs to be able to explain it clearly.
func InDocker() bool {
if _, err := os.Stat("/.dockerenv"); err == nil {
return true
}
b, err := os.ReadFile("/proc/1/cgroup")
if err != nil {
return false
}
s := string(b)
return strings.Contains(s, "docker") || strings.Contains(s, "containerd")
}
-459
View File
@@ -1,459 +0,0 @@
package selfupdate
import (
"archive/zip"
"net/url"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
)
// testPaths creates an isolated upgrade directory. ResolvePaths() cannot be used directly — it would point at the test binary itself,
// and running it once would rename the go test executable.
func testPaths(t *testing.T) Paths {
t.Helper()
dir := t.TempDir()
return Paths{
Dir: dir,
Current: filepath.Join(dir, "artex"),
New: filepath.Join(dir, "artex.new"),
Sum: filepath.Join(dir, "artex.new.sha256"),
Old: filepath.Join(dir, "artex.old"),
Marker: filepath.Join(dir, "artex.upgrade.json"),
}
}
// fakeBin writes an executable shell script impersonating artex. smokeTest merely launches it with -h and checks the exit code,
// so a script is entirely sufficient and much faster than compiling a real binary.
func fakeBin(t *testing.T, path, marker string, exitCode int) {
t.Helper()
script := "#!/bin/sh\necho " + marker + "\nexit " + itoa(exitCode) + "\n"
if err := os.WriteFile(path, []byte(script), 0o755); err != nil {
t.Fatalf("write fake binary %s: %v", path, err)
}
}
func itoa(n int) string {
if n == 0 {
return "0"
}
return string(rune('0' + n))
}
// stage arranges bin to look "staged and ready to swap": it writes artex.new and its checksum.
func stage(t *testing.T, p Paths, marker string, exitCode int) {
t.Helper()
fakeBin(t, p.New, marker, exitCode)
sum, err := fileSHA256(p.New)
if err != nil {
t.Fatalf("compute checksum: %v", err)
}
if err := os.WriteFile(p.Sum, []byte(sum), 0o644); err != nil {
t.Fatalf("write checksum: %v", err)
}
}
func readAll(t *testing.T, path string) string {
t.Helper()
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read %s: %v", path, err)
}
return string(b)
}
func requireUnix(t *testing.T) {
t.Helper()
if runtime.GOOS == "windows" {
t.Skip("the fake binary is an sh script and cannot run on Windows")
}
}
func TestCompareVersions(t *testing.T) {
cases := []struct {
a, b string
want int
comparable bool
}{
{"0.3.7", "0.3.8", -1, true},
{"0.3.8", "0.3.7", 1, true},
{"0.3.7", "0.3.7", 0, true},
{"v0.3.7", "0.3.8", -1, true}, // build.sh strips the v while tags carry it; both sides must be accepted
{"0.3.7", "v0.3.7", 0, true},
{"0.9.0", "0.10.0", -1, true}, // compare numerically, not lexicographically
{"1.0.0", "0.99.99", 1, true},
// Dev builds must be judged non-comparable, otherwise a release overwrites uncommitted changes.
{"dev", "0.3.8", 0, false},
{"0.3.7-2-gabc1234", "0.3.8", 0, false},
{"0.3.7-dirty", "0.3.8", 0, false},
{"0.3", "0.3.8", 0, false},
{"", "0.3.8", 0, false},
}
for _, c := range cases {
got, ok := CompareVersions(c.a, c.b)
if ok != c.comparable {
t.Errorf("CompareVersions(%q,%q) comparable=%v, want %v", c.a, c.b, ok, c.comparable)
continue
}
if ok && got != c.want {
t.Errorf("CompareVersions(%q,%q)=%d, want %d", c.a, c.b, got, c.want)
}
}
}
func TestResolvePathsNaming(t *testing.T) {
p, err := ResolvePaths()
if err != nil {
t.Fatalf("ResolvePaths: %v", err)
}
// Key invariant: every upgrade file sits next to the executable. Landing in CWD would break the swap completely when running as a service
// (the working directory may be /).
for name, path := range map[string]string{"New": p.New, "Sum": p.Sum, "Old": p.Old, "Marker": p.Marker} {
if filepath.Dir(path) != p.Dir {
t.Errorf("%s is not in the executable's directory: %s (want %s)", name, path, p.Dir)
}
}
// On Windows .new/.old must keep .exe, otherwise the smoke test and post-swap execution both fail.
if runtime.GOOS == "windows" {
if !strings.HasSuffix(p.New, ".exe") || !strings.HasSuffix(p.Old, ".exe") {
t.Errorf("on Windows .new/.old must end with .exe: new=%s old=%s", p.New, p.Old)
}
}
}
func TestVerifyStagedRejectsTamperedBinary(t *testing.T) {
requireUnix(t)
p := testPaths(t)
stage(t, p, "new", 0)
// Modify the file after its checksum is written, simulating a corrupted / tampered download.
fakeBin(t, p.New, "tampered", 0)
if err := verifyStaged(p); err == nil {
t.Fatal("expected the SHA256 mismatch to be rejected, but it passed")
}
}
func TestVerifyStagedRejectsUnrunnableBinary(t *testing.T) {
requireUnix(t)
p := testPaths(t)
stage(t, p, "broken", 1) // runs but exits non-zero
if err := verifyStaged(p); err == nil {
t.Fatal("expected the smoke-test failure to be rejected, but it passed")
}
}
func TestApplyStagedHappyPath(t *testing.T) {
requireUnix(t)
p := testPaths(t)
fakeBin(t, p.Current, "old", 0)
stage(t, p, "new", 0)
if err := writeMarker(p.Marker, marker{From: "0.3.7", To: "0.3.8"}); err != nil {
t.Fatalf("write marker: %v", err)
}
action, st := applyStaged(p)
if action != Restart {
t.Fatalf("want Restart, got %v", action)
}
if !st.Pending {
t.Error("status should be Pending after the swap")
}
if !strings.Contains(readAll(t, p.Current), "new") {
t.Error("artex should have been replaced with the new version")
}
if !strings.Contains(readAll(t, p.Old), "old") {
t.Error("the old version should be backed up to artex.old")
}
if _, err := os.Stat(p.New); !os.IsNotExist(err) {
t.Error("artex.new should be gone after the swap")
}
if _, err := os.Stat(p.Sum); !os.IsNotExist(err) {
t.Error("the checksum file should be cleaned up after the swap")
}
// The marker must stay: the next start (running the new version) counts on it and rolls back if necessary.
if _, ok := readMarker(p.Marker); !ok {
t.Error("the upgrade marker should be kept after the swap")
}
}
func TestApplyStagedKeepsCurrentWhenVerifyFails(t *testing.T) {
requireUnix(t)
p := testPaths(t)
fakeBin(t, p.Current, "old", 0)
stage(t, p, "new", 0)
fakeBin(t, p.New, "tampered", 0) // break the checksum
action, st := applyStaged(p)
if action != Continue {
t.Fatalf("want Continue on verification failure, got %v", action)
}
if !st.FailedStage {
t.Error("status should be marked FailedStage")
}
if !strings.Contains(readAll(t, p.Current), "old") {
t.Fatal("the current version must never be touched on verification failure")
}
if _, err := os.Stat(p.New); !os.IsNotExist(err) {
t.Error("the staged file of a failed verification should be cleaned up, otherwise the next start retries it")
}
}
func TestSwapOverwritesPreviousBackup(t *testing.T) {
requireUnix(t)
p := testPaths(t)
fakeBin(t, p.Current, "v2", 0)
fakeBin(t, p.Old, "v1", 0) // backup left by a previous upgrade
stage(t, p, "v3", 0)
if err := swap(p); err != nil {
t.Fatalf("swap: %v", err)
}
if !strings.Contains(readAll(t, p.Current), "v3") {
t.Error("should have swapped to v3")
}
if !strings.Contains(readAll(t, p.Old), "v2") {
t.Error("the backup should be updated to the just-swapped-out v2")
}
}
func TestConfirmCountsAttemptsThenRollsBack(t *testing.T) {
requireUnix(t)
p := testPaths(t)
fakeBin(t, p.Current, "broken-new", 0)
fakeBin(t, p.Old, "good-old", 0)
m := marker{From: "0.3.7", To: "0.3.8"}
// The first maxAttempts starts only increment the counter, giving the new version a chance to settle.
for i := 1; i <= maxAttempts; i++ {
action, st := confirmOrRollback(p, m)
if action != Continue {
t.Fatalf("attempt %d: want Continue, got %v", i, action)
}
if !st.Pending {
t.Errorf("attempt %d: status should be Pending", i)
}
got, ok := readMarker(p.Marker)
if !ok || got.Attempts != i {
t.Fatalf("after attempt %d, attempts=%d (ok=%v), want %d", i, got.Attempts, ok, i)
}
m = got
}
// One more crash exceeds the limit; the old version is swapped back automatically.
action, st := confirmOrRollback(p, m)
if action != Restart {
t.Fatalf("want Restart once the attempt limit is exceeded, got %v", action)
}
if !st.RolledBack {
t.Error("status should be marked RolledBack")
}
if !strings.Contains(readAll(t, p.Current), "good-old") {
t.Fatal("should have rolled back to the old version")
}
if _, err := os.Stat(p.Marker); !os.IsNotExist(err) {
t.Error("the marker should be cleared after rollback, otherwise rollback loops forever")
}
// The version that cannot start is kept for diagnosis, not deleted outright.
if _, err := os.Stat(p.Current + ".failed"); err != nil {
t.Error("the failed version should be kept as .failed for diagnosis")
}
}
func TestManualRollbackIsReversible(t *testing.T) {
requireUnix(t)
p := testPaths(t)
fakeBin(t, p.Current, "v2", 0)
fakeBin(t, p.Old, "v1", 0)
// Rollback() goes through ResolvePaths(); this tests the underlying swap semantics directly.
tmp := p.Current + ".swap"
if err := os.Rename(p.Current, tmp); err != nil {
t.Fatal(err)
}
if err := os.Rename(p.Old, p.Current); err != nil {
t.Fatal(err)
}
if err := os.Rename(tmp, p.Old); err != nil {
t.Fatal(err)
}
if !strings.Contains(readAll(t, p.Current), "v1") {
t.Error("after rollback the current version should be v1")
}
if !strings.Contains(readAll(t, p.Old), "v2") {
t.Error("after rollback the backup should become v2, so it can roll back again")
}
}
func TestParseSums(t *testing.T) {
const (
linuxSum = "1111111111111111111111111111111111111111111111111111111111111111"
winSum = "ABCDEF0000000000000000000000000000000000000000000000000000000000"
)
// sha256sum output is double-space separated; shasum -a 256 in binary mode prefixes the filename with *.
raw := linuxSum + " artex-0.3.8-linux-amd64.zip\n" +
winSum + " *artex-0.3.8-windows-amd64.zip\n" +
"\n" +
"garbage line\n" + // exactly two fields, but the first is not a digest
"deadbeef artex-0.3.8-darwin-arm64.zip\n" // wrong digest length
out := parseSums(raw)
if out["artex-0.3.8-linux-amd64.zip"] != linuxSum {
t.Errorf("linux entry parsed wrong: %v", out)
}
// Digests are normalized to lowercase so case differences cannot be misjudged as mismatches.
if got := out["artex-0.3.8-windows-amd64.zip"]; got != strings.ToLower(winSum) {
t.Errorf("windows entry wrong (* prefix should be stripped, digest lowercased): %q", got)
}
if len(out) != 2 {
t.Errorf("blank lines, non-digest lines, and wrong-length lines should be ignored, got %v", out)
}
}
func TestExtractBinaryFindsNestedEntry(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("the in-archive basename is artex.exe on Windows; this case is built with Unix naming")
}
dir := t.TempDir()
zipPath := filepath.Join(dir, "release.zip")
f, err := os.Create(zipPath)
if err != nil {
t.Fatal(err)
}
zw := zip.NewWriter(f)
// Real release archive layout: artex-<version>-<os>-<arch>/artex, plus a few distractor files.
for name, body := range map[string]string{
"artex-0.3.8-linux-amd64/README.md": "readme",
"artex-0.3.8-linux-amd64/skills/a.md": "skill",
"artex-0.3.8-linux-amd64/artex": "#!/bin/sh\nexit 0\n",
"artex-0.3.8-linux-amd64/config.example.json": "{}",
} {
w, err := zw.Create(name)
if err != nil {
t.Fatal(err)
}
if _, err := w.Write([]byte(body)); err != nil {
t.Fatal(err)
}
}
if err := zw.Close(); err != nil {
t.Fatal(err)
}
f.Close()
dst := filepath.Join(dir, "out")
if err := extractBinary(zipPath, dst); err != nil {
t.Fatalf("extractBinary: %v", err)
}
if got := readAll(t, dst); !strings.Contains(got, "exit 0") {
t.Errorf("what was extracted is not the artex executable: %q", got)
}
info, err := os.Stat(dst)
if err != nil {
t.Fatal(err)
}
if info.Mode().Perm()&0o111 == 0 {
t.Error("the extracted binary must carry the executable bit")
}
}
func TestExtractBinaryMissingEntry(t *testing.T) {
dir := t.TempDir()
zipPath := filepath.Join(dir, "release.zip")
f, err := os.Create(zipPath)
if err != nil {
t.Fatal(err)
}
zw := zip.NewWriter(f)
w, _ := zw.Create("artex-0.3.8-linux-amd64/README.md")
_, _ = w.Write([]byte("readme"))
_ = zw.Close()
f.Close()
if err := extractBinary(zipPath, filepath.Join(dir, "out")); err == nil {
t.Fatal("should error when the archive has no executable")
}
}
func TestCheckURLRejectsNonGitHub(t *testing.T) {
bad := []string{
"http://github.com/x", // not HTTPS
"https://evil.com/artex.zip", // domain not on the allowlist
"https://github.com.evil.com/x", // suffix spoofing
"https://raw.githubusercontent.com.evil.com/x",
}
for _, raw := range bad {
u := mustParse(t, raw)
if err := checkURL(u); err == nil {
t.Errorf("checkURL(%q) should reject", raw)
}
}
good := []string{
"https://api.github.com/repos/x/releases/latest",
"https://objects.githubusercontent.com/blah",
"https://GitHub.com/x", // domain matching is case-insensitive
}
for _, raw := range good {
u := mustParse(t, raw)
if err := checkURL(u); err != nil {
t.Errorf("checkURL(%q) should allow, but errored: %v", raw, err)
}
}
}
func TestAssetNameMatchesBuildScript(t *testing.T) {
// build.sh's package_binary uses artex-<version>-<os>-<arch>.zip with the v prefix
// stripped from the version. One wrong character here and one-click update finds no asset on any platform.
if got := AssetName("v0.3.8", "linux", "amd64"); got != "artex-0.3.8-linux-amd64.zip" {
t.Errorf("AssetName = %q", got)
}
if got := AssetName("0.3.8", "windows", "amd64"); got != "artex-0.3.8-windows-amd64.zip" {
t.Errorf("AssetName = %q", got)
}
}
func mustParse(t *testing.T, raw string) *url.URL {
t.Helper()
u, err := url.Parse(raw)
if err != nil {
t.Fatalf("parse %q: %v", raw, err)
}
return u
}
func TestSettleClearsMarkerAndStopsRollback(t *testing.T) {
requireUnix(t)
p := testPaths(t)
fakeBin(t, p.Current, "new", 0)
fakeBin(t, p.Old, "old", 0)
if err := writeMarker(p.Marker, marker{From: "0.3.7", To: "0.3.8", Attempts: 2}); err != nil {
t.Fatal(err)
}
settle(p)
if _, err := os.Stat(p.Marker); !os.IsNotExist(err) {
t.Fatal("the upgrade marker must be cleared once stability is confirmed")
}
// With the marker gone, later normal restarts neither count attempts nor trigger a spurious rollback.
if _, ok := readMarker(p.Marker); ok {
t.Error("reading the marker should fail")
}
// The backup stays so the user can still roll back manually.
if _, err := os.Stat(p.Old); err != nil {
t.Error("the previous version's backup should still be kept once stability is confirmed")
}
}
func TestSettleIsNoopWithoutMarker(t *testing.T) {
requireUnix(t)
p := testPaths(t)
fakeBin(t, p.Current, "cur", 0)
settle(p) // the plain start path: must not panic or touch any file
if _, err := os.Stat(p.Current); err != nil {
t.Error("settle must not affect any file when there is no marker")
}
}
-339
View File
@@ -1,339 +0,0 @@
package selfupdate
import (
"archive/zip"
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"net/http"
"os"
"path"
"runtime"
"strings"
"time"
)
// sumsAsset is the checksum manifest generated by release.yml, covering every zip in the Release.
const sumsAsset = "SHA256SUMS"
// maxBinarySize caps the size of the extracted binary so a malformed zip cannot fill the disk.
const maxBinarySize = 512 << 20 // 512 MiB
// Phase is a stage of the upgrade process, used directly as the phase field of SSE events.
type Phase string
const (
PhaseIdle Phase = "idle"
PhaseDownload Phase = "downloading"
PhaseVerify Phase = "verifying"
PhaseExtract Phase = "extracting"
PhaseStaged Phase = "staged"
PhaseFailed Phase = "failed"
)
// Progress is supplied by the caller to push progress to the frontend. pct is only meaningful during the download phase (0-100);
// other phases pass -1.
type Progress func(ph Phase, pct int, msg string)
// Stage downloads the current platform's release archive from the given Release, verifies it, and stages the new binary as artex.new.
//
// It uses the full zip rather than a bare binary for two reasons: existing Releases' SHA256SUMS only cover zips anyway,
// so going through the zip needs no CI change and stays compatible with already-published versions; and the zip also carries
// skills/, leaving the door open for syncing built-in skills later. The only cost is those few hundred KB of skills.
//
// Returning means staging is complete; the caller then shuts down gracefully and exits with ExitRestart.
func Stage(ctx context.Context, c *http.Client, rel *Release, currentVersion string, prog Progress) error {
if prog == nil {
prog = func(Phase, int, string) {}
}
p, err := ResolvePaths()
if err != nil {
return err
}
if err := checkWritable(p.Dir); err != nil {
return err
}
name := AssetName(rel.TagName, runtime.GOOS, runtime.GOARCH)
asset, ok := rel.FindAsset(name)
if !ok {
return fmt.Errorf("this release ships no package for %s/%s (missing %s)", runtime.GOOS, runtime.GOARCH, name)
}
prog(PhaseDownload, 0, "Fetching checksum manifest…")
sums, err := fetchSums(ctx, c, rel)
if err != nil {
return err
}
want, ok := sums[name]
if !ok {
return fmt.Errorf("%s does not list %s; refusing to install an unverified binary", sumsAsset, name)
}
// All temp files land in the target directory so the final rename stays an atomic operation within one filesystem
// (a cross-device rename fails, and /tmp is often a separate mount point).
zipPath := p.New + ".zip.part"
binPath := p.New + ".part"
defer func() {
_ = os.Remove(zipPath)
_ = os.Remove(binPath)
}()
prog(PhaseDownload, 0, fmt.Sprintf("Downloading %s (%s)…", name, humanSize(asset.Size)))
got, err := download(ctx, c, asset, zipPath, prog)
if err != nil {
return err
}
prog(PhaseVerify, -1, "Verifying SHA256…")
if !strings.EqualFold(got, want) {
return fmt.Errorf("SHA256 mismatch: want %s, got %s (corrupted download or tampering)", short(want), short(got))
}
prog(PhaseExtract, -1, "Extracting and smoke testing…")
if err := extractBinary(zipPath, binPath); err != nil {
return err
}
if err := smokeTest(binPath); err != nil {
return fmt.Errorf("the new version cannot run on this system: %w", err)
}
// The staged file's own sha256 is stored separately: it gets verified once more before the swap on the next start,
// guarding against the file being altered or corrupted between staging and restart.
binSum, err := fileSHA256(binPath)
if err != nil {
return fmt.Errorf("compute new binary checksum: %w", err)
}
if err := os.WriteFile(p.Sum, []byte(binSum), 0o644); err != nil {
return fmt.Errorf("write checksum: %w", err)
}
if err := os.Rename(binPath, p.New); err != nil {
_ = os.Remove(p.Sum)
return fmt.Errorf("stage new version: %w", err)
}
if err := writeMarker(p.Marker, marker{
From: currentVersion,
To: strings.TrimPrefix(rel.TagName, "v"),
StagedAt: time.Now().Unix(),
}); err != nil {
// The marker only affects automatic rollback; the staged file itself is already in place, so the upgrade is not interrupted over it.
prog(PhaseStaged, -1, "Warning: writing the upgrade marker failed; this upgrade will have no automatic rollback protection")
}
prog(PhaseStaged, 100, "New version ready, restarting…")
return nil
}
// fetchSums downloads and parses SHA256SUMS, returning filename → hex digest.
func fetchSums(ctx context.Context, c *http.Client, rel *Release) (map[string]string, error) {
asset, ok := rel.FindAsset(sumsAsset)
if !ok {
return nil, fmt.Errorf("this Release has no %s; integrity cannot be verified, refusing to upgrade", sumsAsset)
}
body, err := get(ctx, c, asset.URL)
if err != nil {
return nil, fmt.Errorf("download %s: %w", sumsAsset, err)
}
defer body.Close()
raw, err := io.ReadAll(io.LimitReader(body, 1<<20))
if err != nil {
return nil, fmt.Errorf("read %s: %w", sumsAsset, err)
}
out := parseSums(string(raw))
if len(out) == 0 {
return nil, fmt.Errorf("%s is empty or in an unrecognized format", sumsAsset)
}
return out, nil
}
// parseSums parses a sha256sum-style manifest, returning filename → hex digest.
//
// The first field must be 64 hex characters to be included. Checking only for "exactly two fields" is not enough —
// any line of two-word prose would pass as a valid entry and stuff garbage into the digest table,
// letting a real asset match a wrong digest.
func parseSums(raw string) map[string]string {
out := map[string]string{}
for line := range strings.Lines(raw) {
// The format is "<sha256> <filename>" (sha256sum uses a double space; shasum's binary
// mode prefixes the filename with *).
fields := strings.Fields(strings.TrimSpace(line))
if len(fields) != 2 || !isHexSHA256(fields[0]) {
continue
}
name := strings.TrimPrefix(fields[1], "*")
if name == "" {
continue
}
out[name] = strings.ToLower(fields[0])
}
return out
}
func isHexSHA256(s string) bool {
if len(s) != 64 {
return false
}
for _, c := range s {
switch {
case c >= '0' && c <= '9', c >= 'a' && c <= 'f', c >= 'A' && c <= 'F':
default:
return false
}
}
return true
}
// download writes the asset to dst, computing SHA256 along the way and reporting progress against Content-Length.
func download(ctx context.Context, c *http.Client, a Asset, dst string, prog Progress) (string, error) {
body, err := get(ctx, c, a.URL)
if err != nil {
return "", fmt.Errorf("download %s: %w", a.Name, err)
}
defer body.Close()
f, err := os.Create(dst)
if err != nil {
return "", fmt.Errorf("create temp file: %w", err)
}
defer f.Close()
h := sha256.New()
pw := &progressWriter{total: a.Size, prog: prog, name: a.Name, last: time.Now()}
if _, err := io.Copy(io.MultiWriter(f, h, pw), body); err != nil {
return "", fmt.Errorf("download interrupted: %w", err)
}
if err := f.Sync(); err != nil {
return "", fmt.Errorf("flush to disk failed: %w", err)
}
if a.Size > 0 && pw.written != a.Size {
return "", fmt.Errorf("incomplete download: want %d bytes, got %d bytes", a.Size, pw.written)
}
return hex.EncodeToString(h.Sum(nil)), nil
}
// get issues an allowlist-constrained GET and returns the response body.
func get(ctx context.Context, c *http.Client, rawURL string) (io.ReadCloser, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, rawURL, nil)
if err != nil {
return nil, err
}
if err := checkURL(req.URL); err != nil {
return nil, err
}
req.Header.Set("User-Agent", "artex-selfupdate")
resp, err := c.Do(req)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
resp.Body.Close()
return nil, fmt.Errorf("HTTP %d", resp.StatusCode)
}
return resp.Body, nil
}
// extractBinary pulls the artex executable out of the release archive.
//
// The archive layout is artex-<version>-<os>-<arch>/artex, but this matches by **basename** instead of building the full path:
// the version already appears once in the archive name, and one wrong character would fail the whole upgrade; matching by basename is more robust to changes.
func extractBinary(zipPath, dst string) error {
want := "artex"
if runtime.GOOS == "windows" {
want = "artex.exe"
}
zr, err := zip.OpenReader(zipPath)
if err != nil {
return fmt.Errorf("open release archive: %w", err)
}
defer zr.Close()
for _, entry := range zr.File {
if entry.FileInfo().IsDir() || !strings.EqualFold(path.Base(entry.Name), want) {
continue
}
rc, err := entry.Open()
if err != nil {
return fmt.Errorf("read %s: %w", entry.Name, err)
}
defer rc.Close()
f, err := os.OpenFile(dst, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o755)
if err != nil {
return fmt.Errorf("write out new binary: %w", err)
}
defer f.Close()
n, err := io.Copy(f, io.LimitReader(rc, maxBinarySize+1))
if err != nil {
return fmt.Errorf("extract %s: %w", entry.Name, err)
}
if n > maxBinarySize {
return fmt.Errorf("executable in the release archive exceeds %s; refusing to extract", humanSize(maxBinarySize))
}
if n == 0 {
return fmt.Errorf("%s in the release archive is an empty file", want)
}
return f.Sync()
}
return fmt.Errorf("%s not found in the release archive", want)
}
// checkWritable confirms up front that the directory is writable. Without this step, running as non-root or having the binary in a system
// directory would only fail at the very moment of the swap, after tens of MB have already been downloaded.
func checkWritable(dir string) error {
probe, err := os.CreateTemp(dir, ".artex-update-probe-*")
if err != nil {
return fmt.Errorf("program directory %s is not writable; automatic update is impossible (check permissions or upgrade manually): %w", dir, err)
}
name := probe.Name()
_ = probe.Close()
_ = os.Remove(name)
return nil
}
// progressWriter counts written bytes and reports at a throttled rate, avoiding one SSE event per 32KiB chunk.
type progressWriter struct {
total int64
written int64
name string
prog Progress
last time.Time
}
func (w *progressWriter) Write(b []byte) (int, error) {
w.written += int64(len(b))
if time.Since(w.last) < 300*time.Millisecond {
return len(b), nil
}
w.last = time.Now()
pct := -1
if w.total > 0 {
pct = int(w.written * 100 / w.total)
}
w.prog(PhaseDownload, pct, fmt.Sprintf("Downloading %s / %s", humanSize(w.written), humanSize(w.total)))
return len(b), nil
}
func humanSize(n int64) string {
const unit = 1024
if n < unit {
return fmt.Sprintf("%d B", n)
}
div, exp := int64(unit), 0
for v := n / unit; v >= unit; v /= unit {
div *= unit
exp++
}
return fmt.Sprintf("%.1f %cB", float64(n)/float64(div), "KMGT"[exp])
}
func short(sum string) string {
if len(sum) > 12 {
return sum[:12] + "…"
}
return sum
}
-7
View File
@@ -664,13 +664,6 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("GET /api/logs/history", s.getLogsHistory)
mux.HandleFunc("GET /api/logs/stream", s.streamLogs)
// One-click page update. These go through the default JWT auth (auth.go only exempts /api/auth/* and
// /api/health), so the endpoints that modify the program itself naturally require a login.
mux.HandleFunc("GET /api/update/check", s.updateCheck)
mux.HandleFunc("POST /api/update/apply", s.updateApply)
mux.HandleFunc("POST /api/update/rollback", s.updateRollback)
mux.HandleFunc("GET /api/update/stream", s.updateStream)
mux.HandleFunc("GET /api/tasks", s.listTasks)
mux.HandleFunc("POST /api/tasks", s.createTask)
mux.HandleFunc("GET /api/skill-gaps", s.skillGaps)
-403
View File
@@ -1,403 +0,0 @@
package server
import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"net/http"
"runtime"
"sync"
"time"
"github.com/Autumn-27/artex/selfupdate"
)
// HTTP surface for one-click updates from the UI. The actual
// download/verify/swap logic lives in the selfupdate package; this file only
// handles the auth boundary, concurrency exclusion, progress broadcast, and
// telling main "time to exit".
//
// The restart is not performed by this process: after staging the new version
// the process exits with selfupdate.ExitRestart and the supervisor script
// (start.sh / start.bat, or ENTRYPOINT under Docker) brings it back up.
// restartCh is closed once an upgrade is staged or a rollback completes; main
// then exits with ExitRestart.
var (
restartOnce sync.Once
restartCh = make(chan struct{})
)
// RestartRequested returns a channel that is closed when it is time to exit and
// let the supervisor relaunch us.
func RestartRequested() <-chan struct{} { return restartCh }
func requestRestart() { restartOnce.Do(func() { close(restartCh) }) }
// bootState is selfupdate.Bootstrap's verdict for this boot (upgrade succeeded /
// just rolled back / staged file discarded), injected by main so that
// /api/update/check can faithfully tell the frontend how the last upgrade ended.
var (
bootStateMu sync.Mutex
bootState selfupdate.State
)
// SetBootUpdateState is called once by main at startup.
func SetBootUpdateState(st selfupdate.State) {
bootStateMu.Lock()
defer bootStateMu.Unlock()
bootState = st
}
func bootUpdateState() selfupdate.State {
bootStateMu.Lock()
defer bootStateMu.Unlock()
return bootState
}
// releaseCache caches the latest-release lookup against GitHub.
//
// The top-bar "new version available" indicator queries once per full page load,
// and the unauthenticated GitHub API allows 60 requests per IP per hour —
// without a cache, opening a few tabs or refreshing a few times burns through
// the quota, and a real update check would then fail to get through. When the
// user explicitly clicks "check for updates", force bypasses the cache.
type releaseCache struct {
mu sync.Mutex
rel *selfupdate.Release
err error
at time.Time
// fetch is the retrieval function, an injection point kept only for tests;
// nil means the real GitHub lookup.
fetch func(context.Context, *http.Client) (*selfupdate.Release, error)
}
const (
releaseTTL = 30 * time.Minute
// Failed results are also cached briefly, otherwise every page load would sit
// through a timeout while GitHub is unreachable; but the TTL is short so it
// heals itself soon after the network recovers.
releaseErrTTL = 2 * time.Minute
// Timeout for lookups. NewClient's 30-minute timeout is for downloading the
// full package; a version check cannot wait that long.
releaseTimeout = 20 * time.Second
)
var relCache = &releaseCache{}
// get returns the latest Release, serving from cache when possible.
//
// The lock is held for the whole fetch: concurrent requests queue up and share
// one lookup instead of each hitting GitHub (right after page load, when many
// tabs query at once, is exactly when rate limiting bites).
func (c *releaseCache) get(ctx context.Context, client *http.Client, force bool) (*selfupdate.Release, error) {
c.mu.Lock()
defer c.mu.Unlock()
if !force {
ttl := releaseTTL
if c.err != nil {
ttl = releaseErrTTL
}
if !c.at.IsZero() && time.Since(c.at) < ttl {
return c.rel, c.err
}
}
fetch := c.fetch
if fetch == nil {
fetch = selfupdate.FetchLatest
}
ctx, cancel := context.WithTimeout(ctx, releaseTimeout)
defer cancel()
rel, err := fetch(ctx, client)
// A cancelled request (user closed the tab) does not mean GitHub is broken;
// do not write it into the cache, or the next visitor gets a baffling
// "cancelled" error.
if err != nil && ctx.Err() != nil && errors.Is(ctx.Err(), context.Canceled) {
return c.rel, err
}
c.rel, c.err, c.at = rel, err, time.Now()
return rel, err
}
// updateProgress is one progress update pushed to the frontend.
type updateProgress struct {
Phase selfupdate.Phase `json:"phase"`
Percent int `json:"percent"` // only meaningful during the download phase; -1 otherwise
Message string `json:"message"`
Version string `json:"version,omitempty"`
Error string `json:"error,omitempty"`
}
// updateHub holds the progress of one upgrade and broadcasts it to SSE
// subscribers.
//
// running doubles as a mutex: a second POST /api/update/apply during an upgrade
// gets a 409, keeping two goroutines from writing the same artex.new at once.
type updateHub struct {
mu sync.Mutex
running bool
cur updateProgress
subs map[chan updateProgress]struct{}
}
var updHub = &updateHub{
cur: updateProgress{Phase: selfupdate.PhaseIdle, Percent: -1},
subs: map[chan updateProgress]struct{}{},
}
// begin claims the upgrade slot, returning false if one is already running.
func (h *updateHub) begin(version string) bool {
h.mu.Lock()
defer h.mu.Unlock()
if h.running {
return false
}
h.running = true
h.cur = updateProgress{Phase: selfupdate.PhaseDownload, Percent: 0, Message: "Preparing...", Version: version}
h.fanout(h.cur)
return true
}
// finish ends an upgrade. A nil err means staging succeeded and a restart is
// pending.
func (h *updateHub) finish(err error) {
h.mu.Lock()
defer h.mu.Unlock()
h.running = false
if err != nil {
h.cur = updateProgress{Phase: selfupdate.PhaseFailed, Percent: -1, Message: "Update failed", Error: err.Error(), Version: h.cur.Version}
} else {
h.cur = updateProgress{Phase: selfupdate.PhaseStaged, Percent: 100, Message: "New version ready, restarting...", Version: h.cur.Version}
}
h.fanout(h.cur)
}
func (h *updateHub) publish(ph selfupdate.Phase, pct int, msg string) {
h.mu.Lock()
defer h.mu.Unlock()
h.cur = updateProgress{Phase: ph, Percent: pct, Message: msg, Version: h.cur.Version}
h.fanout(h.cur)
}
// fanout must be called with h.mu held. Subscriber channels are buffered and
// drops are fine — progress is disposable, transient information, and a stuck
// SSE connection must never block the upgrade itself.
func (h *updateHub) fanout(p updateProgress) {
for ch := range h.subs {
select {
case ch <- p:
default:
}
}
}
func (h *updateHub) snapshot() (updateProgress, bool) {
h.mu.Lock()
defer h.mu.Unlock()
return h.cur, h.running
}
func (h *updateHub) subscribe() (<-chan updateProgress, func()) {
ch := make(chan updateProgress, 64)
h.mu.Lock()
h.subs[ch] = struct{}{}
h.mu.Unlock()
var once sync.Once
return ch, func() {
once.Do(func() {
h.mu.Lock()
delete(h.subs, ch)
h.mu.Unlock()
close(ch)
})
}
}
// updateCheck queries the latest stable release on GitHub and compares it with
// the current version.
//
// The frontend also talks to api.github.com directly (GitHub's CORS is *), but
// **this endpoint is authoritative**: the download is done by the backend, so an
// update is only possible if the backend can reach GitHub. It is common for the
// browser to have connectivity while the server does not (server on an internal
// network, or the proxy configured only in the browser); in that case clicking
// update would inevitably fail, so it is better to report the error honestly at
// the check step.
func (s *Server) updateCheck(w http.ResponseWriter, r *http.Request) {
current := BuildVersion
mode := "binary"
if selfupdate.InDocker() {
mode = "docker"
}
boot := bootUpdateState()
out := map[string]any{
"current": current,
"mode": mode,
"os": runtime.GOOS,
"arch": runtime.GOARCH,
"has_backup": selfupdate.HasBackup(),
"repo": selfupdate.Repo,
"boot_notice": boot.Detail,
"rolled_back": boot.RolledBack,
}
// The top-bar indicator uses the cache (default); clicking "check for
// updates" passes force=1 to refetch from the origin.
force := r.URL.Query().Get("force") != ""
client := selfupdate.NewClient(s.m.GlobalProxy())
rel, err := relCache.get(r.Context(), client, force)
if err != nil {
out["error"] = err.Error()
writeJSON(w, 200, out)
return
}
latest := rel.TagName
out["latest"] = latest
out["notes"] = rel.Body
out["html_url"] = rel.HTMLURL
if !rel.PublishedAt.IsZero() {
out["published_at"] = rel.PublishedAt.Format(time.RFC3339)
}
asset := selfupdate.AssetName(latest, runtime.GOOS, runtime.GOARCH)
out["asset"] = asset
if a, ok := rel.FindAsset(asset); ok {
out["asset_available"] = true
out["size"] = a.Size
} else {
out["asset_available"] = false
}
cmp, comparable := selfupdate.CompareVersions(current, latest)
out["comparable"] = comparable
out["has_update"] = comparable && cmp < 0
if !comparable {
// Development builds (dev / git describe with a suffix) have no
// comparable version number. Allowing the update would only overwrite
// the binary being debugged locally with a release build, so updates
// are refused outright.
out["reason"] = fmt.Sprintf("current version %q is not a stable release; one-click update is disabled", current)
}
writeJSON(w, 200, out)
}
// updateApply downloads and stages the new version, then lets the process exit
// so the supervisor script restarts it.
//
// It returns 202 immediately and does the work on a background goroutine: a
// full package download can take minutes and would be cut off by reverse-proxy
// timeouts if tied to the request. Progress goes over /api/update/stream.
func (s *Server) updateApply(w http.ResponseWriter, r *http.Request) {
current := BuildVersion
// Use the cache: ensure the installed version is exactly the one the user
// saw and confirmed in the UI.
client := selfupdate.NewClient(s.m.GlobalProxy())
rel, err := relCache.get(r.Context(), client, false)
if err != nil {
writeErr(w, 502, err.Error())
return
}
cmp, comparable := selfupdate.CompareVersions(current, rel.TagName)
if !comparable {
writeErr(w, 400, fmt.Sprintf("current version %q is not a stable release; one-click update is disabled", current))
return
}
if cmp >= 0 {
writeErr(w, 400, fmt.Sprintf("already on the latest version %s", current))
return
}
if !updHub.begin(rel.TagName) {
writeErr(w, 409, "an update is already in progress")
return
}
go func() {
// Deliberately use s.ctx rather than the request ctx: the request ends
// as soon as the HTTP response returns, and a download tied to it would
// be cancelled immediately.
err := selfupdate.Stage(s.ctx, client, rel, current, func(ph selfupdate.Phase, pct int, msg string) {
updHub.publish(ph, pct, msg)
})
updHub.finish(err)
if err != nil {
log.Printf("[update] update failed: %v", err)
return
}
log.Printf("[update] %s → %s staged, exiting to complete the swap", current, rel.TagName)
// Leave a moment to push the final progress to the frontend before
// triggering the exit.
time.Sleep(1500 * time.Millisecond)
requestRestart()
}()
writeJSON(w, 202, map[string]any{"ok": true, "target": rel.TagName})
}
// updateRollback actively reverts to the previous version (the artex.old backed
// up before the swap).
func (s *Server) updateRollback(w http.ResponseWriter, r *http.Request) {
if _, running := updHub.snapshot(); running {
writeErr(w, 409, "an update is in progress; cannot roll back")
return
}
if err := selfupdate.Rollback(); err != nil {
writeErr(w, 400, err.Error())
return
}
log.Printf("[update] manually rolled back to the previous version, exiting to complete the switch")
writeJSON(w, 202, map[string]any{"ok": true})
go func() {
time.Sleep(500 * time.Millisecond)
requestRestart()
}()
}
// updateStream pushes update progress over SSE.
func (s *Server) updateStream(w http.ResponseWriter, r *http.Request) {
flusher, ok := w.(http.Flusher)
if !ok {
writeErr(w, 500, "streaming unsupported")
return
}
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
w.Header().Set("X-Accel-Buffering", "no")
ch, unsub := updHub.subscribe()
defer unsub()
send := func(p updateProgress) {
b, _ := json.Marshal(p)
fmt.Fprintf(w, "data: %s\n\n", b)
flusher.Flush()
}
// Send the current state first, so a page refresh immediately shows an
// in-flight upgrade.
cur, _ := updHub.snapshot()
send(cur)
ctx := r.Context()
ping := time.NewTicker(20 * time.Second)
defer ping.Stop()
for {
select {
case <-ctx.Done():
return
case p, ok := <-ch:
if !ok {
return
}
send(p)
case <-ping.C:
fmt.Fprint(w, ": ping\n\n")
flusher.Flush()
}
}
}
-161
View File
@@ -1,161 +0,0 @@
package server
import (
"context"
"errors"
"net/http"
"testing"
"time"
"github.com/Autumn-27/artex/selfupdate"
)
// releaseCache is the layer that protects the GitHub quota: the unauthenticated
// API allows only 60 requests/hour/IP, and the top-bar "new version available"
// indicator queries once per full page load. Once the cache stops working, a
// user opening a few tabs would burn through the quota, and real update checks
// would then fail to get through.
func newTestCache(fetch func(context.Context, *http.Client) (*selfupdate.Release, error)) *releaseCache {
return &releaseCache{fetch: fetch}
}
func TestReleaseCacheServesFromCache(t *testing.T) {
calls := 0
c := newTestCache(func(context.Context, *http.Client) (*selfupdate.Release, error) {
calls++
return &selfupdate.Release{TagName: "v0.3.8"}, nil
})
for range 5 {
rel, err := c.get(t.Context(), nil, false)
if err != nil {
t.Fatalf("get: %v", err)
}
if rel.TagName != "v0.3.8" {
t.Fatalf("TagName = %q", rel.TagName)
}
}
if calls != 1 {
t.Errorf("5 queries should hit the origin only once, got %d", calls)
}
}
func TestReleaseCacheForceBypasses(t *testing.T) {
calls := 0
c := newTestCache(func(context.Context, *http.Client) (*selfupdate.Release, error) {
calls++
return &selfupdate.Release{TagName: "v0.3.8"}, nil
})
if _, err := c.get(t.Context(), nil, false); err != nil {
t.Fatal(err)
}
// Clicking "check for updates" must return a live result, otherwise a freshly
// published release stays invisible until the cache expires.
if _, err := c.get(t.Context(), nil, true); err != nil {
t.Fatal(err)
}
if calls != 2 {
t.Errorf("force should bypass the cache, want 2 origin fetches, got %d", calls)
}
}
func TestReleaseCacheExpiresAfterTTL(t *testing.T) {
calls := 0
c := newTestCache(func(context.Context, *http.Client) (*selfupdate.Release, error) {
calls++
return &selfupdate.Release{TagName: "v0.3.8"}, nil
})
if _, err := c.get(t.Context(), nil, false); err != nil {
t.Fatal(err)
}
// Move the stored timestamp back to just past expiry, simulating TTL timeout.
c.at = time.Now().Add(-releaseTTL - time.Second)
if _, err := c.get(t.Context(), nil, false); err != nil {
t.Fatal(err)
}
if calls != 2 {
t.Errorf("after TTL expiry it should fetch from the origin again, want 2, got %d", calls)
}
}
func TestReleaseCacheUsesShorterTTLForErrors(t *testing.T) {
calls := 0
c := newTestCache(func(context.Context, *http.Client) (*selfupdate.Release, error) {
calls++
return nil, errors.New("github unreachable")
})
if _, err := c.get(t.Context(), nil, false); err == nil {
t.Fatal("expected an error")
}
// Failed results must also be cached briefly, otherwise every page load waits
// through a timeout for nothing while GitHub is unreachable.
if _, err := c.get(t.Context(), nil, false); err == nil {
t.Fatal("expected an error")
}
if calls != 1 {
t.Errorf("errors should be cached briefly, want 1 origin fetch, got %d", calls)
}
// But the error TTL must be noticeably shorter than the success one, so the
// cache can heal quickly once the network recovers.
if releaseErrTTL >= releaseTTL {
t.Fatalf("error TTL (%v) must be shorter than success TTL (%v)", releaseErrTTL, releaseTTL)
}
c.at = time.Now().Add(-releaseErrTTL - time.Second)
if _, err := c.get(t.Context(), nil, false); err == nil {
t.Fatal("expected an error")
}
if calls != 2 {
t.Errorf("after error TTL expiry it should retry, want 2, got %d", calls)
}
}
func TestReleaseCacheDoesNotPoisonOnCallerCancel(t *testing.T) {
good := &selfupdate.Release{TagName: "v0.3.8"}
c := newTestCache(func(ctx context.Context, _ *http.Client) (*selfupdate.Release, error) {
return good, nil
})
if _, err := c.get(t.Context(), nil, false); err != nil {
t.Fatal(err)
}
// A visitor closing the tab cancels the request. That does not mean GitHub is
// broken, and the "cancelled" error must never be written into the cache —
// otherwise every visitor would get a baffling error for the next 30 minutes.
c.fetch = func(ctx context.Context, _ *http.Client) (*selfupdate.Release, error) {
return nil, ctx.Err()
}
c.at = time.Now().Add(-releaseTTL - time.Second) // expire the cache to force a refetch
ctx, cancel := context.WithCancel(context.Background())
cancel()
if _, err := c.get(ctx, nil, false); err == nil {
t.Fatal("when the caller has cancelled, the error should be passed through to it")
}
// Key invariant: the cancelled call leaves no trace — the cache holds neither
// the "cancelled" error nor loses the previous good result.
if c.err != nil {
t.Fatalf("cancel error must not be written into the cache, got %v", c.err)
}
if c.rel == nil || c.rel.TagName != "v0.3.8" {
t.Fatalf("cache should keep the previous good result, got %+v", c.rel)
}
// The cancellation fetched no new data, so the next visitor should refetch
// from the origin — and get a normal result, unaffected by the earlier cancel.
c.fetch = func(context.Context, *http.Client) (*selfupdate.Release, error) {
return good, nil
}
rel, err := c.get(t.Context(), nil, false)
if err != nil {
t.Fatalf("normal requests after a cancellation must not fail: %v", err)
}
if rel == nil || rel.TagName != "v0.3.8" {
t.Fatalf("should get a normal result, got %+v", rel)
}
}
+112
View File
@@ -0,0 +1,112 @@
---
name: bbot-recon
description: "Invoke this skill when a hunt task needs breadth-first recon on an authorized target: subdomain enumeration, subdomain-takeover surface, archived URLs, parameter mining, IP/ASN enrichment, or when one-shot tools (subfinder, httpx, nuclei) would need manual piping between steps. Runs bbot (recursive event-graph scanner, installed in the image) under hard program-policy gates: passive zero-touch mode when scanners are banned, active modules only when the program allows them."
---
# bbot-recon: recursive recon under program-policy gates
bbot (blacklanternsecurity/bbot, AGPL-3.0 like ARTEX, pinned v3.0.2 at
`/usr/local/bin/bbot`, venv `/opt/bbot`, module dependencies preinstalled in
the image) chains every discovery into an event graph and emits structured
JSON. Use it for breadth. Keep the single-purpose tools (subfinder, httpx,
katana, ffuf, nuclei) when you need exact control over individual requests or
a single host.
## Hard gates (read before running anything)
1. **Authorization**: targets and seeds must be inside the task's authorized
scope. Third-party sources are queried about the target, not exploited.
2. **Program policy first**: read the program rules, then pick the mode.
- Scanner ban, "no automated scanning", or a tight rate limit (<= 5 req/s):
**Mode P only** (passive, zero-touch; the target is never contacted).
- Active scanning allowed: Mode A is allowed; loud families still need the
program to be explicit about aggressive scanning.
- Loud/invasive families only under explicit permission, and prefer the
dedicated single tools (ffuf, nmap, nuclei) when they exist, for rate
control.
3. **Scope discipline**: pass in-scope targets with `-t`, add `-b` blacklist
entries for anything excluded (logout/auth endpoints, specific subdomains).
`--strict-scope` treats subdomains as out-of-scope unless listed. Events
with scope_distance > 0 are intel only: never probe them.
4. **No API keys by default**: keyless modules only. If the task vault
provides a key for a permitted source, pass it via
`-c modules.<name>.api_key=...` and never echo it.
5. **Never** run the `kitchen-sink` preset on a live bounty target.
6. Module deps are preinstalled in the image, so every run passes `--no-deps`.
If a module still complains about a dependency, record it as unavailable
and continue without it: runtime pip does not work under egress allowlists.
## Modes and module sets (v3.0.2, all keyless)
**Mode P (passive, zero-touch)**: run with seeds only (`-s`, no `-t`) so
nothing enters scope and no module contacts the target. Sources: `crt`,
`crt_db`, `certspotter`, `dnsdumpster`, `hackertarget`, `anubisdb`,
`rapiddns`, `subdomaincenter`, `urlscan`, `myssl`, `wayback`, `asn`,
`shodan_idb` (InternetDB), `ipneighbor`; email family `dnscaa`, `dnstlsrpt`,
`emailformat`, `pgp`, `skymem`.
**Mode A (active-safe)**, when the program allows active scanning: adds
`dnsresolve`, `dnscommonsrv`, `baddns`, `baddns_direct`, `baddns_zone`
(takeover), `sslcert`, `robots`, `securitytxt`, `oauth`,
`graphql_introspection`, `fingerprintx`, `reflected_parameters`, `retirejs`,
`badsecrets`, `gowitness`, `iis_shortnames` (loud but targeted),
`filedownload`, `newsletters`, `hunt`, and the `http` spider (cap it, see
rate control).
**Loud/invasive**, explicit permission only: `paramminer_getparams`,
`paramminer_headers`, `paramminer_cookies`, `virtualhost`, `waf_bypass`,
`bypass403`, `dnsbrute`, `dnsbrute_mutations`, `portscan`, `lightfuzz`,
`nuclei`, `legba`, `medusa`, `host_header`, `url_manipulation`,
`generic_ssrf`.
## Recipes
Common tail for every run: `-om json,txt -o /app/data/tasks/<task_id>/bbot -n <intent-name> -y --no-deps`.
`--no-deps` is mandatory: module dependencies are baked into the image, and a
scan-time dependency check would run pip, which fails (or hangs) under a
program's egress allowlist. One scan per intent, named after it
(`passive-subdoms`, `takeover`, `params`).
0. Sanity first: `bbot --version && bbot -l | head -40` (the table lists each
module's Needs-API-Key and flags; trust it over memory).
1. Passive subdomain sweep (P):
`bbot -s example.com -m crt,crt_db,certspotter,dnsdumpster,hackertarget,anubisdb,rapiddns,subdomaincenter,urlscan -n passive-subdoms`
2. Archived URLs (P): `bbot -s example.com -m wayback -n wayback-urls`
3. IP/ASN enrichment (P): `bbot -s 203.0.113.0/24 -m shodan_idb,ipneighbor,asn -n ip-enrich`
4. Takeover pass (A): `bbot -t example.com -p baddns-heavy -n takeover`
5. Single-host deep look (A):
`bbot -t host.example.com -m sslcert,robots,securitytxt,oauth,graphql_introspection,iis_shortnames -n host-deep`
6. Param mining (loud):
`bbot -t https://host.example.com -m paramminer_getparams,paramminer_headers,paramminer_cookies -n params`
7. Email perimeter (P): `bbot -s example.com -m dnscaa,dnstlsrpt,emailformat,pgp,skymem -n emails`
Rate control: fewer modules per scan; `--exclude-cdn` to skip CDN/WAF
backends; identify yourself when the program asks for it
(`--user-agent "..."`); cap the spider with `-c spider_distance=1
spider_depth=2` on sensitive programs. When in doubt, split into two passive
scans rather than one broad active scan.
## Ingestion
After each scan read `<out_dir>/<scan_name>/output.json` (one JSON event per
line; also `scan.log` and `error.log` in the same directory as the audit
trail). Useful fields: `type`, `data`, `host`, `resolved_hosts`, `tags`,
`module` (source), `scope_description`, `scope_distance`, `timestamp`.
- `DNS_NAME` (resolved), `IP_ADDRESS`, `OPEN_TCP_PORT`, `URL` become facts,
with tags as qualifiers (waf, cloudflare, cdn).
- `FINDING` / `VULNERABILITY` (baddns, badsecrets, iis_shortnames,
aspnet_bin_exposure, telerik, ...) become finding candidates: attach the
event description, the module, and the scan id. bbot findings are leads,
not proofs; verify independently with the single tools before any report.
- `EMAIL_ADDRESS` becomes intel facts.
Record the scan id and output path in the task so every fact is traceable.
## Boundaries
- Never mix passive (Mode P) and active modules in the same run.
- Do not edit bbot's global config or secrets files; pass everything per scan
via `-c` so runs stay reproducible.
- If a scan's error.log shows a module failed, retry once without it instead
of widening the scan.
-12
View File
@@ -10,11 +10,7 @@ rem
rem It does exactly one thing: run artex.exe and, once the process exits, decide by exit code whether to relaunch it.
rem
rem 0 user stopped normally -> exit the loop
rem 75 program requested a restart -> rerun immediately ("one-click update" or "rollback" was clicked in the web UI)
rem other crash -> rerun after backoff (1->2->4... up to 60 seconds)
rem
rem Download, SHA256 verification, and the swap are NOT done here; artex does all of them itself at startup
rem (the selfupdate package). The script stays dumb; see the notes at the top of start.sh.
setlocal enabledelayedexpansion
cd /d "%~dp0"
@@ -25,7 +21,6 @@ if not exist "%BIN%" (
exit /b 1
)
set "RESTART_CODE=75"
set "MAX_DELAY=60"
set /a delay=1
@@ -38,13 +33,6 @@ if "!code!"=="0" (
exit /b 0
)
if "!code!"=="%RESTART_CODE%" (
rem Update/rollback is ready: after the rerun, artex completes the swap at startup.
echo [artex] restart requested (applying new version)...
set /a delay=1
goto loop
)
echo [artex] abnormal exit ^(code=!code!^), restarting in !delay!s 1>&2
rem timeout fails in a redirected console, so fall back to ping (delaying N seconds needs N+1 pings).
set /a pings=!delay!+1
-12
View File
@@ -9,13 +9,7 @@
# It does exactly one thing: run artex and, once the process exits, decide by exit code whether to relaunch it.
#
# 0 user stopped normally -> exit the loop
# 75 program requested a restart -> rerun immediately ("one-click update" or "rollback" was clicked in the web UI)
# other crash -> rerun after backoff (1→2→4... up to 60 seconds)
#
# Download, SHA256 verification, and the swap are deliberately not done here: that logic would need two implementations in sh and bat,
# and it is precisely the step that must never fail — once an unrunnable binary is swapped in, this script would faithfully relaunch it
# forever, and the user could only fix it on the machine by hand. So all verification/swap logic stays in Go (the selfupdate package),
# done by artex itself at startup, keeping the script dumb.
set -u
cd "$(dirname "$0")" || exit 1
@@ -23,7 +17,6 @@ cd "$(dirname "$0")" || exit 1
BIN=./artex
[ -x "$BIN" ] || { echo "[artex] executable $BIN not found" >&2; exit 1; }
RESTART_CODE=75
MAX_DELAY=60
child=0
@@ -67,11 +60,6 @@ while :; do
echo "[artex] exited normally"
exit 0
;;
"$RESTART_CODE")
# Update/rollback is ready: after the rerun, artex completes the swap at startup (see selfupdate.Bootstrap).
echo "[artex] restart requested (applying new version)…"
delay=1
;;
*)
echo "[artex] abnormal exit (code=$code), restarting in ${delay}s" >&2
sleep "$delay"
@@ -14,7 +14,6 @@ import { AccountSwitcher } from "./sidebar/account-switcher";
import { LayoutControls } from "./sidebar/layout-controls";
import { SearchDialog } from "./sidebar/search-dialog";
import { ThemeSwitcher } from "./sidebar/theme-switcher";
import { UpdateBadge } from "./update-badge";
// Task detail pages stay as-is: they bring their own header/Tabs and padding, so no extra global header or padding is stacked here.
function isFullBleed(pathname: string) {
@@ -68,7 +67,6 @@ export function MainContent({ children }: { children: ReactNode }) {
{version && (
<span className="font-medium text-muted-foreground text-xs tabular-nums">Version · {version}</span>
)}
<UpdateBadge />
<LayoutControls />
<ThemeSwitcher />
<AccountSwitcher users={[currentUser]} />
@@ -1,58 +0,0 @@
"use client";
import * as React from "react";
import Link from "next/link";
import { ArrowUpCircleIcon } from "lucide-react";
import { api } from "@/lib/api";
/**
* Top-bar "new version" badge: checked once per full page load; lights up next to the version
* number when an update exists, and clicking jumps to the "Version and updates" card on the system settings page.
*
* The backend caches GitHub query results for 30 minutes, so checking once per mount here is safe
* — the unauthenticated GitHub API allows only 60 requests/hour/IP; without that cache layer, a few extra tabs
* would drain the quota, and a real update check would then stall.
*
* Failed checks stay silent: the top bar is not the place for errors — the reason shows on the settings page's "Check for updates".
*/
export function UpdateBadge() {
const [latest, setLatest] = React.useState("");
React.useEffect(() => {
let alive = true;
api
.checkUpdate()
.then((r) => {
// has_update already includes the "version is comparable" check, so development builds never light this up.
if (alive && r.has_update && r.latest) setLatest(r.latest.replace(/^v(?=\d)/, ""));
})
.catch(() => {
// Silent: no network / GitHub rate limiting must not raise an error in the top bar.
});
return () => {
alive = false;
};
}, []);
if (!latest) return null;
return (
<Link
href="/system/settings"
title={`New version ${latest} available — click to update`}
className="inline-flex items-center gap-1.5 rounded-full bg-primary px-2.5 py-1 font-medium text-primary-foreground text-xs transition-opacity hover:opacity-90"
>
{/* Breathing dot: the top bar is crowded and plain text is easy to miss; the animation makes it instantly visible. */}
<span className="relative flex size-1.5">
<span className="absolute inline-flex size-full animate-ping rounded-full bg-primary-foreground opacity-75" />
<span className="relative inline-flex size-1.5 rounded-full bg-primary-foreground" />
</span>
<ArrowUpCircleIcon className="size-3.5" />
<span className="hidden sm:inline">New version {latest}</span>
<span className="sm:hidden">New version</span>
</Link>
);
}
@@ -1,314 +0,0 @@
"use client";
import * as React from "react";
import {
CheckCircle2Icon,
DownloadIcon,
ExternalLinkIcon,
RefreshCwIcon,
RotateCcwIcon,
TriangleAlertIcon,
} from "lucide-react";
import { toast } from "sonner";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card";
import { Progress } from "@/components/ui/progress";
import { api, sseUrl } from "@/lib/api";
import type { UpdateCheck, UpdateProgress } from "@/lib/types";
/** The longest wait for a new version to come online. One upgrade goes through three process starts
* (stage → swap → new version), each taking seconds; three minutes covers slow disks and Docker container rebuilds. */
const RESTART_TIMEOUT_MS = 180_000;
function humanSize(n?: number): string {
if (!n || n <= 0) return "";
const units = ["B", "KB", "MB", "GB"];
let v = n;
let i = 0;
while (v >= 1024 && i < units.length - 1) {
v /= 1024;
i++;
}
return `${v.toFixed(i === 0 ? 0 : 1)} ${units[i]}`;
}
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
export function UpdateCard() {
const [info, setInfo] = React.useState<UpdateCheck | null>(null);
const [checking, setChecking] = React.useState(true);
const [progress, setProgress] = React.useState<UpdateProgress | null>(null);
// Separate from progress: once staging completes the process is gone, SSE drops, and we must switch to polling /api/health.
const [restarting, setRestarting] = React.useState(false);
const [busy, setBusy] = React.useState(false);
// quiet also decides whether to bypass the backend cache: the automatic check on page load uses the cache (the top bar just checked),
// while a manual "Check for updates" click forces a refresh — otherwise a freshly released version only appears after the cache expires.
const check = React.useCallback((quiet = false) => {
setChecking(true);
api
.checkUpdate(!quiet)
.then((r) => {
setInfo(r);
if (!quiet) {
if (r.error) toast.error("Update check failed: " + r.error);
else if (r.has_update) toast.success(`New version ${r.latest} found`);
else if (r.comparable) toast.success("Already on the latest version");
}
})
.catch((e) => {
if (!quiet) toast.error("Update check failed: " + (e as Error).message);
})
.finally(() => setChecking(false));
}, []);
React.useEffect(() => {
check(true);
}, [check]);
// Poll /api/health until the version changes.
//
// The criterion must be "the version changed", not "it connects again": during the swap the old version briefly comes back up once
// (that run only swaps in artex.new and exits immediately), so checking connectivity alone would misjudge success.
const waitForNewVersion = React.useCallback(async (fromVersion: string) => {
setRestarting(true);
const deadline = Date.now() + RESTART_TIMEOUT_MS;
while (Date.now() < deadline) {
await sleep(2000);
try {
const r = await fetch("/api/health", { cache: "no-store" });
if (r.ok) {
const j = (await r.json()) as { version?: string };
if (j.version && j.version !== fromVersion) {
toast.success(`Updated to ${j.version}, reloading the page`);
await sleep(800);
window.location.reload();
return;
}
}
} catch {
// Failing to connect during the restart window is expected; keep polling.
}
}
setRestarting(false);
toast.error("Timed out waiting for the service to restart. Check the backend logs, or confirm artex was started via start.sh / start.bat.");
}, []);
// Subscribe to update progress. SSE bypasses Next's /api rewrite (that layer buffers and events never come through).
const openStream = React.useCallback(
(fromVersion: string) => {
const es = new EventSource(sseUrl("/api/update/stream"));
es.onmessage = (ev) => {
let p: UpdateProgress;
try {
p = JSON.parse(ev.data) as UpdateProgress;
} catch {
return;
}
setProgress(p);
if (p.phase === "failed") {
es.close();
setBusy(false);
toast.error("Update failed: " + (p.error || p.message));
return;
}
if (p.phase === "staged") {
es.close();
void waitForNewVersion(fromVersion);
}
};
es.onerror = () => {
// SSE always drops when the process exits. If we are already waiting for the restart, that is normal —
// leave the verdict to the /api/health polling.
es.close();
};
return es;
},
[waitForNewVersion],
);
const doUpdate = () => {
if (!info) return;
const from = info.current;
const ok = window.confirm(
`Update to ${info.latest}?\n\n` +
"Updating restarts the program; running tasks will be interrupted.\n" +
(info.mode === "docker"
? "\nNote: an in-container update only replaces the program itself and does not update the toolchain in the image (playwright / nmap etc.);" +
"if the new version needs new tools, use docker compose pull instead."
: ""),
);
if (!ok) return;
setBusy(true);
setProgress({ phase: "downloading", percent: 0, message: "Preparing…" });
const es = openStream(from);
api.applyUpdate().catch((e) => {
es.close();
setBusy(false);
setProgress(null);
toast.error("Failed to start the update: " + (e as Error).message);
});
};
const doRollback = () => {
if (!info) return;
if (
!window.confirm(
"Roll back to the previous version?\n\nThe program will restart and running tasks will be interrupted.\nNote: the database schema is not rolled back; the old version may not recognize data written by the new one.",
)
)
return;
const from = info.current;
setBusy(true);
api
.rollbackUpdate()
.then(() => {
toast.success("Switched to the previous version, restarting…");
void waitForNewVersion(from);
})
.catch((e) => {
setBusy(false);
toast.error("Rollback failed: " + (e as Error).message);
});
};
const phase = progress?.phase;
const showProgress = busy || restarting;
// Only the download phase has a real percentage (computed from Content-Length). Verify/unpack/wait-for-restart are
// phases of unknown duration, so the bar fills up with a pulse animation meaning "busy, unsure how long".
const downloading = !restarting && phase === "downloading";
const pct = downloading ? Math.max(progress?.percent ?? 0, 0) : 100;
return (
// The settings page uses a multi-column masonry layout; each card handles its own row spacing and avoids breaking across columns (see the comment in page.tsx).
<Card className="mb-4 break-inside-avoid md:mb-6">
<CardHeader>
<CardTitle className="flex items-center gap-2 text-base">
<DownloadIcon className="size-4" />
Version & updates
</CardTitle>
<CardDescription>Check for and install new versions from GitHub. Updating restarts the program; running tasks will be interrupted.</CardDescription>
</CardHeader>
<CardContent className="space-y-4">
<div className="flex flex-wrap items-center gap-2 text-sm">
<span className="text-muted-foreground">Current version</span>
<Badge variant="secondary" className="font-mono">
{info?.current ?? "…"}
</Badge>
{info && (
<>
<Badge variant="outline" className="font-mono">
{info.os}/{info.arch}
</Badge>
<Badge variant="outline">{info.mode === "docker" ? "Docker" : "Standalone"}</Badge>
</>
)}
{info?.latest && (
<>
<span className="text-muted-foreground">Latest version</span>
<Badge variant={info.has_update ? "default" : "secondary"} className="font-mono">
{info.latest}
</Badge>
</>
)}
{info?.html_url && (
<a
href={info.html_url}
target="_blank"
rel="noreferrer"
className="inline-flex items-center gap-1 text-xs text-muted-foreground underline-offset-4 hover:underline"
>
Release notes <ExternalLinkIcon className="size-3" />
</a>
)}
</div>
{info?.boot_notice && (
<p className="flex items-start gap-2 rounded-md border border-amber-500/40 bg-amber-500/10 p-2 text-xs text-amber-700 dark:text-amber-400">
<TriangleAlertIcon className="mt-0.5 size-3.5 shrink-0" />
{info.boot_notice}
</p>
)}
{info?.error && (
<p className="flex items-start gap-2 rounded-md border border-destructive/40 bg-destructive/10 p-2 text-xs text-destructive">
<TriangleAlertIcon className="mt-0.5 size-3.5 shrink-0" />
Cannot reach GitHub: {info.error}
{" "}Configure the global proxy above and retry.
</p>
)}
{info && !info.comparable && info.reason && <p className="text-xs text-muted-foreground">{info.reason}</p>}
{info?.has_update && info.asset_available === false && (
<p className="flex items-start gap-2 rounded-md border border-destructive/40 bg-destructive/10 p-2 text-xs text-destructive">
<TriangleAlertIcon className="mt-0.5 size-3.5 shrink-0" />
{info.latest} has no release package for {info.os}/{info.arch} (missing {info.asset}); automatic update is unavailable.
</p>
)}
{info?.has_update && info.asset_available !== false && (
<p className="text-xs text-muted-foreground">
Will download <span className="font-mono">{info.asset}</span>
{info.size ? ` (${humanSize(info.size)})` : ""}; it is replaced only after the SHA256 check and a smoke test pass — on failure the current version is kept automatically.
</p>
)}
{info && !info.has_update && info.comparable && !info.error && (
<p className="flex items-center gap-2 text-xs text-muted-foreground">
<CheckCircle2Icon className="size-3.5 text-emerald-600" />
Already on the latest version.
</p>
)}
{info?.mode === "docker" && info.has_update && (
<p className="text-xs text-muted-foreground">
Under Docker an update only replaces the program itself, not the toolchain in the image (playwright / nmap etc.), and
<span className="font-mono"> docker compose up -d </span>
rebuilding the container reverts to the image's bundled version. To upgrade the image too, run
<span className="font-mono"> docker compose pull artex &amp;&amp; docker compose up -d artex</span>.
</p>
)}
{showProgress && (
<div className="space-y-1.5">
<Progress value={pct} className={downloading ? undefined : "animate-pulse"} />
<p className="text-xs text-muted-foreground">
{restarting ? "Restarting and applying the new version, please wait (the page will refresh automatically)…" : progress?.message}
</p>
</div>
)}
<div className="flex flex-wrap gap-2">
<Button variant="outline" size="sm" onClick={() => check(false)} disabled={checking || busy || restarting}>
<RefreshCwIcon className={checking ? "size-4 animate-spin" : "size-4"} />
Check for updates
</Button>
<Button
size="sm"
onClick={doUpdate}
disabled={busy || restarting || !info?.has_update || info?.asset_available === false}
>
<DownloadIcon className="size-4" />
{info?.has_update ? `Update to ${info.latest}` : "Update now"}
</Button>
{info?.has_backup && (
<Button variant="ghost" size="sm" onClick={doRollback} disabled={busy || restarting}>
<RotateCcwIcon className="size-4" />
Roll back to the previous version
</Button>
)}
</div>
<p className="text-xs text-muted-foreground">
One-click updates rely on the supervisor script to restart the program. Please start ARTEX via <span className="font-mono">start.sh</span> (Windows:
<span className="font-mono"> start.bat</span>); when running the artex binary directly, the program is not relaunched after exiting.
</p>
</CardContent>
</Card>
);
}
@@ -15,8 +15,6 @@ import { api } from "@/lib/api";
import { CHAT_SEND_MODE_OPTIONS, type ChatSendMode, setChatSendMode, useChatSendMode } from "@/lib/chat-send-mode";
import type { Settings } from "@/lib/types";
import { UpdateCard } from "./_components/update-card";
export default function SystemSettingsPage() {
const [trafficCapture, setTrafficCapture] = React.useState(false);
const [agentTrafficBinding, setAgentTrafficBinding] = React.useState(false);
@@ -267,8 +265,6 @@ export default function SystemSettingsPage() {
balance-fill by content height. Card spacing uses mb rather than gap — in a multi-column layout
column-gap only covers the gaps between columns; row spacing must come from the children. */}
<div className="columns-1 gap-4 md:gap-6 lg:columns-2">
<UpdateCard />
<Card className="mb-4 break-inside-avoid md:mb-6">
<CardHeader>
<CardTitle className="flex items-center gap-2 text-base">
-12
View File
@@ -102,7 +102,6 @@ import type {
TrafficEvidenceRole,
TrafficHost,
TrafficResp,
UpdateCheck,
UsageStats,
WorkspaceFile,
WorkspaceListing,
@@ -1310,15 +1309,4 @@ export const api = {
// ledger, exact per call, covering per-agent bindings / pooling / interrupted spend).
tokensByModel: (task: string) =>
get<{ models: ModelTokenStat[] }>(`/llm/records/by-model?task=${encodeURIComponent(task)}`),
// ---- One-click update ----
// The backend arbitrates update checks: it does the downloading, and "the browser reaches
// GitHub but the server cannot" is common (server on an intranet, proxy configured only in
// the browser) — clicking update would inevitably fail then. The backend caches GitHub
// query results for 30 minutes (the unauthenticated GitHub API allows 60 requests/hour/IP;
// the top bar queries on every full page load and would burn the quota fast without caching). force=true bypasses the cache, reserved for the user's explicit "Check for updates" click.
checkUpdate: (force = false) => get<UpdateCheck>(`/update/check${force ? "?force=1" : ""}`),
// Returns 202 immediately; the download runs in the background, progress via /api/update/stream.
applyUpdate: () => post<{ ok: boolean; target: string }>(`/update/apply`),
rollbackUpdate: () => post<{ ok: boolean }>(`/update/rollback`),
};
-41
View File
@@ -1591,47 +1591,6 @@ export interface FindingTrafficDetail {
response: EvidenceBodyPreview;
}
/** GET /api/update/check — comparison of the current version against the latest GitHub stable release. */
export interface UpdateCheck {
/** The currently running version; development builds are "dev" or a suffixed git describe form. */
current: string;
/** Runtime form. Under docker the swap only touches the container writable layer; rebuilding the container reverts to the image version. */
mode: "docker" | "binary";
os: string;
arch: string;
repo: string;
/** Whether a previous version exists to roll back to (artex.old). */
has_backup: boolean;
/** Conclusion of the self-update bootstrap at this startup (swap failed / rolled back, etc.); empty when nothing happened. */
boot_notice?: string;
rolled_back?: boolean;
/** Reason given when the GitHub query fails; the fields below are absent then. */
error?: string;
latest?: string;
notes?: string;
html_url?: string;
published_at?: string;
/** The release asset name for the current platform, and whether that Release actually carries it. */
asset?: string;
asset_available?: boolean;
size?: number;
has_update?: boolean;
/** Whether the two version numbers are comparable; false for development builds, which disables one-click update. */
comparable?: boolean;
/** Explanation shown when comparable is false. */
reason?: string;
}
/** One update progress event pushed by /api/update/stream. */
export interface UpdateProgress {
phase: "idle" | "downloading" | "verifying" | "extracting" | "staged" | "failed";
/** Meaningful only during the downloading phase (0-100); -1 otherwise. */
percent: number;
message: string;
version?: string;
error?: string;
}
// Original execution selected from an approval, never submitted to the reviewer.
export interface InterceptExecution {
conversation_id: number | null;