diff --git a/CHANGELOG.md b/CHANGELOG.md index 94aad22..2315be4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,11 +23,13 @@ All notable changes to this project are documented in this file. The format is b - **`ask_operator` target warning**: a request whose question names a host or URL outside the goal targets is tagged as containing target-derived text (#197). - **Opt-in update banner**: `ARTEX_UPDATE_CHECK` makes the backend check the latest GitHub release at most once per 24 hours and the UI show a banner; off by default (#202). - **Release provenance**: release archives carry GitHub build provenance attestations and a CycloneDX SBOM (`gh attestation verify --repo BaaRai/ARTEX`); release notes are the matching `CHANGELOG.md` section (#219, #201). -- **Capture-proxy rewrite rules, storage and API**: `GET`, `POST`, `PUT` and `DELETE /api/traffic/rewrite-rules` and `POST /api/traffic/rewrite-rules/{id}/toggle` manage rules that match a request or a response (method, host, URL, headers, body, status) and describe a rewrite (method, path and query, headers, body, status). Rules are validated on save, limited to 100 of which 50 enabled, created disabled, and stored in the new table `traffic_rewrite_rules` (migration `0005_traffic_rewrite_rules.sql`) with `matchers` and `actions` encrypted at rest. The value of an action marked `sensitive` is never returned, and an update sent without it keeps the stored one. The server compiles the enabled rules at startup and after every change and hands the snapshot to the proxy. **The proxy does not apply the rules yet: capture is unchanged.** See `docs/traffic-rewrite-rules.md`. +- **Capture-proxy rewrite rules, storage and API**: `GET`, `POST`, `PUT` and `DELETE /api/traffic/rewrite-rules` and `POST /api/traffic/rewrite-rules/{id}/toggle` manage rules that match a request or a response (method, host, URL, headers, body, status) and describe a rewrite (method, path and query, headers, body, status). Rules are validated on save, limited to 100 of which 50 enabled, created disabled, and stored in the new table `traffic_rewrite_rules` (migration `0005_traffic_rewrite_rules.sql`) with `matchers` and `actions` encrypted at rest. The value of an action marked `sensitive` is never returned, and an update sent without it keeps the stored one. The server compiles the enabled rules at startup and after every change and hands the snapshot to the proxy. The proxy applies them from the next entry. See `docs/traffic-rewrite-rules.md`. +- **Capture-proxy rewrite rules applied**: the capture proxy applies enabled rules to each HTTP request before it is sent to the target and to each response before it reaches the client. Each action that ran is recorded in the new table `exchange_rewrites` in the same transaction as the exchange, `traffic_get` shows a `REWRITTEN` line, and `traffic_search` never finds a value a rule wrote. SSE status and headers are rewritten at the start of the stream. **WebSocket handshakes are not rewritten**: the capture library runs its hook after the upgrade, so the status and headers cannot be changed, and frames pass unchanged. **Responses of 5 MiB or more** are forwarded unchanged and not recorded in full, because the library does not call the response hook for them. Snapshots of original requests are capped at 1024 entries and 64 MiB. Replays go through the proxy, so the rules apply to them. Archives, a per-replay rule toggle, the web UI and the proof flag come later. ### Changed - **Password changes revoke sessions**: a password change stores a monotonic `auth.password_version` in the same transaction as the hash. Tokens carry that version, so a token from an older version is rejected, including when two changes land in the same second. `reset-password.sh` writes the same counter and requires an ARTEX restart, because the process reads the version at startup. A failed write leaves the previous session valid. Log, activity, and side-question streams re-check the token and close after the version moves. If the version cannot be read at startup, sessions are rejected until the next login reads it. +- **Task budgets survive a restart**: planner, worker, judge, and compactor rows in `llm_usage` are counted back into `ARTEX_TASK_BUDGET_*`. A restored task already inside the wrap-up reserve only has that reserve left, so the run wraps up instead of starting another worker. The per-worker estimate is not restored and starts from the defaults (50,000 tokens and 10 turns) until a worker finishes in this process. Chat, goal decomposition, and side questions stay outside the cap. - **Untrusted-output boundary is now on by default**: Norma frames tool results for the worker, planner, chat and main agent unless `ARTEX_UNTRUSTED_BOUNDARY` is set to `0`, `false`, `off` or `no`. It was opt-in via `ARTEX_UNTRUSTED_BOUNDARY=1` in 0.6.0 (#166); the default flips here (#203). - **Versioned migrations**: the `db` package applies numbered `db/migrations/NNNN_slug.sql` files once, in order, under the schema advisory lock, and records a SHA-256 checksum in `schema_migrations`; missing, renamed, changed or out-of-order migrations fail closed (#226). - **Coverage graph performance**: the coverage graph batches its connector lookups (three queries instead of one per root and per company) and adds the partial index `idx_ts_icp`; on a task of about 3,000 assets the p50 dropped from 126 ms to 27 ms and the queries per call from 221 to 3 (#230). diff --git a/README.md b/README.md index ab353d5..ad8efec 100644 --- a/README.md +++ b/README.md @@ -255,7 +255,10 @@ worker remains confined to the intent it claimed. - **Budgets**: `ARTEX_TASK_BUDGET_TOKENS` / `ARTEX_TASK_BUDGET_TURNS` cap a whole task (planner and workers); `ARTEX_INTENT_BUDGET_TOKENS` / `ARTEX_INTENT_BUDGET_TURNS` cap one worker run. Near the limit the task enters a forced wrap-up; a worker is not started when the remaining budget cannot - cover it. Budgets are in memory: a restart starts fresh. + cover it. Spend already in `llm_usage` is restored after a restart. A task that is already inside + the wrap-up reserve only has that reserve left, so the revived run wraps up instead of starting + another worker. The per-worker cost estimate starts again from the defaults (50,000 tokens and + 10 turns) until a worker finishes in this process. - **Noise governor**: `ARTEX_NOISE_PROFILE=quiet|balanced|aggressive` paces requests ARTEX sends to targets (per-host rate, burst, in-flight limit, jitter, backoff on 429/503, short GET caching). With a profile set, agents also receive low-and-slow flags for the installed scanners. @@ -455,7 +458,13 @@ workers, else 50,000 tokens and 10 turns). Near the wrap-up reserve (the larger tokens, and 2 turns) the task enters a forced wrap-up: the planner writes a summary and final goal judgment, and workers may only write back. The task then ends `done` if its goals are met, else `timeout`. Practical floor: below 50,000 tokens no worker starts; 50,000-70,000 typically gives one -worker then wrap-up; pick well above 70,000 for real exploration. +worker then wrap-up; pick well above 70,000 for real exploration. Planner, worker, judge, and +compactor rows in `llm_usage` are counted back into the cap when a ledger is created, including +after a restart and after a revived run drops its ledger. A restored task already inside the +wrap-up reserve does not get another worker: only that reserve remains, and the run wraps up. +Chat, goal decomposition, and side questions are stored on the same task and are not counted. +The per-worker average is not restored (usage rows have no run boundary), so the estimate starts +from the defaults until a worker finishes in this process. **Common flags**: `./start.sh -addr :8787 -proxy :8788` (`-addr` frontend/API, `-proxy` traffic recording proxy). diff --git a/agent/budget.go b/agent/budget.go index f22028e..4eb2925 100644 --- a/agent/budget.go +++ b/agent/budget.go @@ -92,7 +92,8 @@ func BudgetsFromEnv() (task, intent Budget) { NewBudget(env("ARTEX_INTENT_BUDGET_TOKENS"), env("ARTEX_INTENT_BUDGET_TURNS")) } -// BudgetLedger meters one task, in memory (a process restart starts afresh). +// BudgetLedger meters one task. The total is restored from llm_usage when the +// ledger is created, including after a restart. Per-worker averages are not. type BudgetLedger struct { task, intent Budget mu sync.Mutex @@ -110,6 +111,21 @@ func NewBudgetLedger(task, intent Budget) *BudgetLedger { return &BudgetLedger{task: task, intent: intent} } +// RestoreSpend adds usage that was already recorded before this ledger existed. +// It only adds to the task total. Per-worker averages stay at the defaults +// until a worker finishes against this ledger, because llm_usage rows do not +// mark where one worker run ends and the next begins. A nil ledger (no budget +// configured) is a no-op. +func (l *BudgetLedger) RestoreSpend(spent Spend) { + if l == nil || (spent.Tokens == 0 && spent.Turns == 0) { + return + } + l.mu.Lock() + l.spent.Tokens += spent.Tokens + l.spent.Turns += spent.Turns + l.mu.Unlock() +} + // Record meters one task-level call (the planner's own rounds). func (l *BudgetLedger) Record(u llm.Usage) { l.mu.Lock() diff --git a/agent/budget_test.go b/agent/budget_test.go index b35f358..2c4c10a 100644 --- a/agent/budget_test.go +++ b/agent/budget_test.go @@ -144,3 +144,14 @@ func TestBudgetCountsMatchUsageFixture(t *testing.T) { t.Fatalf("spent %+v remaining %+v estimate %+v, want {%d 5}, remaining, {10400 3}", st.Spent, st.Remaining, st.Estimate, want) } } + +func TestRestoreSpendCountsTowardCap(t *testing.T) { + l := NewBudgetLedger(NewBudget(100, 10), NewBudget(0, 0)) + l.RestoreSpend(Spend{Tokens: 40, Turns: 4}) + st := l.State() + if st.Spent != (Spend{40, 4}) || st.Remaining != (Budget{60, 6}) { + t.Fatalf("spent=%+v remaining=%+v", st.Spent, st.Remaining) + } + var none *BudgetLedger + none.RestoreSpend(Spend{Tokens: 1, Turns: 1}) +} diff --git a/agent/coverage.go b/agent/coverage.go new file mode 100644 index 0000000..6ad9b40 --- /dev/null +++ b/agent/coverage.go @@ -0,0 +1,22 @@ +package agent + +import ( + "log" + + "github.com/Autumn-27/artex/db" +) + +// coverageOn reports whether asset-coverage tools stay available for this task. +// A nil store is the historical default (enabled). A database error is logged and +// treated as enabled so a transient read failure does not drop the tools mid-run. +func coverageOn(as *db.AssetStore, taskID int64) bool { + if as == nil { + return true + } + enabled, err := as.CoverageEnabled(taskID) + if err != nil { + log.Printf("[coverage] task %d: reading the coverage flag failed; leaving coverage on: %v", taskID, err) + return true + } + return enabled +} diff --git a/agent/coverage_flag_test.go b/agent/coverage_flag_test.go new file mode 100644 index 0000000..7bcbf4b --- /dev/null +++ b/agent/coverage_flag_test.go @@ -0,0 +1,113 @@ +package agent + +import ( + "bytes" + "fmt" + "log" + "strings" + "testing" + "time" + + "github.com/Autumn-27/artex/db" +) + +// A failed coverage-flag read stays enabled: the log says so, the worker keeps +// record_coverage and list_untested_assets, and AddAutoScope still writes. That +// write happens even when the stored flag is off, which is the outage behaviour. +func TestCoverageReadFailureKeepsToolsAndWritesAutoScope(t *testing.T) { + d := testDB(t) + defer d.Close() + off := false + task, err := d.CreateTaskWithOptions("coverage outage", "goal", db.TaskCreateOptions{CoverageEnabled: &off}) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = d.DeleteTask(task.ID) }) + + live := d.Assets() + stored, err := live.CoverageEnabled(task.ID) + if err != nil || stored { + t.Fatalf("stored flag enabled=%v err=%v, want off", stored, err) + } + if coverageOn(live, task.ID) { + t.Fatal("a readable off flag was treated as on") + } + + dsn, _, err := db.DSN() + if err != nil { + t.Fatal(err) + } + broken, err := db.Open(dsn) + if err != nil { + t.Fatal(err) + } + if err := broken.Close(); err != nil { + t.Fatal(err) + } + var logged bytes.Buffer + prev := log.Default().Writer() + log.SetOutput(&logged) + t.Cleanup(func() { log.SetOutput(prev) }) + if !coverageOn(broken.Assets(), task.ID) { + t.Fatal("a failed coverage read turned the feature off") + } + if !strings.Contains(logged.String(), "leaving coverage on") { + t.Fatalf("coverage read failure was not logged: %q", logged.String()) + } + + // record_coverage is the worker's coverage tool; list_untested_assets is the + // planner's, so each is checked on the tool set that actually carries it. + hasPlannerTool := func(ts *ToolSet, name string) bool { + for _, tool := range ts.DropCoverageTools(ts.PlannerTools()) { + if tool.Name() == name { + return true + } + } + return false + } + offWorker := NewToolSet(nil, "worker-off") + offWorker.SetCoverageEnabled(false) + if hasTool(offWorker, "record_coverage") { + t.Fatal("record_coverage stayed available to a worker with coverage off") + } + offPlanner := NewToolSet(nil, "planner-off") + offPlanner.SetCoverageEnabled(false) + if hasPlannerTool(offPlanner, "list_untested_assets") { + t.Fatal("list_untested_assets stayed available to the planner with coverage off") + } + outage := coverageOn(broken.Assets(), task.ID) + outWorker := NewToolSet(nil, "worker-outage") + outWorker.SetCoverageEnabled(outage) + if !hasTool(outWorker, "record_coverage") { + t.Fatal("record_coverage dropped from the worker when the coverage read failed") + } + outPlanner := NewToolSet(nil, "planner-outage") + outPlanner.SetCoverageEnabled(outage) + if !hasPlannerTool(outPlanner, "list_untested_assets") { + t.Fatal("list_untested_assets dropped from the planner when the coverage read failed") + } + + keptOff := fmt.Sprintf("off-%d.invalid", time.Now().UnixNano()) + writeIntentAutoScope(live, task.ID, []*db.Asset{{Type: "root_domain", Domain: keptOff}}, false) + wrote := fmt.Sprintf("outage-%d.invalid", time.Now().UnixNano()) + writeIntentAutoScope(live, task.ID, []*db.Asset{{Type: "root_domain", Domain: wrote}}, coverageOn(broken.Assets(), task.ID)) + scope, err := live.ListTaskScope(task.ID) + if err != nil { + t.Fatal(err) + } + var sawKept, sawWrote bool + for _, row := range scope { + if row.Domain == keptOff { + sawKept = true + } + if row.Domain == wrote { + sawWrote = true + } + } + if sawKept { + t.Fatal("auto scope was written while coverage was off") + } + if !sawWrote { + t.Fatal("auto scope was not written when the coverage read failed") + } +} diff --git a/agent/mainagent.go b/agent/mainagent.go index 5b9f376..92eb443 100644 --- a/agent/mainagent.go +++ b/agent/mainagent.go @@ -118,7 +118,7 @@ func (m *MainAgent) Chat(ctx context.Context, taskID int64, mainSeg int, as *db. tsx.SetAssetStore(as, as.Companies()) } tsx.SetTaskID(taskID) - tsx.SetCoverageEnabled(as == nil || as.CoverageEnabled(taskID)) + tsx.SetCoverageEnabled(coverageOn(as, taskID)) tsx.SetNotify(notify) // generic wake-up (write ops without a dedicated callback go through it, debounced) tsx.SetResumeTask(resume) // set_goals adds a goal → pull a completed/paused task back to running tsx.SetNotifyGoal(notifyGoal) // set_goals adds a goal → record a "human added a goal: …" trigger for the planner diff --git a/agent/planner.go b/agent/planner.go index 5ed68f8..00751cc 100644 --- a/agent/planner.go +++ b/agent/planner.go @@ -473,7 +473,7 @@ func (p *Planner) Plan(ctx context.Context, taskID int64, as *db.AssetStore, ts tsx.SetAssetStore(as, as.Companies()) } tsx.SetTaskID(taskID) - tsx.SetCoverageEnabled(as == nil || as.CoverageEnabled(taskID)) + tsx.SetCoverageEnabled(coverageOn(as, taskID)) tsx.killWork = p.killWork // enable kill_work tool (nil = unavailable) tsx.steerWork = p.steerWork // enable steer_work tool (nil = unavailable) if origin, _ := ts.OriginFactID(); origin > 0 { diff --git a/agent/worker.go b/agent/worker.go index 1eb5301..c43f94c 100644 --- a/agent/worker.go +++ b/agent/worker.go @@ -387,6 +387,21 @@ func renderWorkerGraphOverview(data map[string]any, env dataEnvelope) string { sections } +// writeIntentAutoScope records the intent's target assets as auto scope when +// coverage is on. coverageEnabled is fail-open: a failed flag read is true, so +// an outage still writes, including when the stored flag is off. +func writeIntentAutoScope(as *db.AssetStore, taskID int64, assets []*db.Asset, coverageEnabled bool) { + if as == nil || !coverageEnabled { + return + } + for _, a := range assets { + if a == nil { + continue + } + _ = as.AddAutoScope(taskID, a.Type, a.Domain, a.URL, a.IP) + } +} + // Execute runs one intent. hooks (the per-task Guard) gates every tool call; may // be nil. emit, if non-nil, receives one ActivityRecord per execution step. // notifyFinding, if non-nil, is called (intentID, summary) when this worker writes @@ -412,7 +427,7 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db. tsx := NewToolSet(ts, name) tsx.SetFindingRecorder(w.findingRecorder) tsx.SetTaskID(taskID) - coverageEnabled := as == nil || as.CoverageEnabled(taskID) + coverageEnabled := coverageOn(as, taskID) tsx.SetCoverageEnabled(coverageEnabled) if as != nil { tsx.SetAssetStore(as, as.Companies()) @@ -473,12 +488,9 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db. // toward coverage only under a declared domain or IP root (db.declaredScopeFilter); // coverage scope, not enforced. // When the asset-coverage feature is off, the test scope (denominator) is no - // longer accumulated. - if coverageEnabled { - for _, a := range assets { - _ = as.AddAutoScope(taskID, a.Type, a.Domain, a.URL, a.IP) - } - } + // longer accumulated. A failed flag read is treated as on, so an outage + // still writes, including for a task whose stored flag is off. + writeIntentAutoScope(as, taskID, assets, coverageEnabled) } } } diff --git a/cmd/artex/main.go b/cmd/artex/main.go index 6595a28..1cba3d4 100644 --- a/cmd/artex/main.go +++ b/cmd/artex/main.go @@ -110,10 +110,15 @@ func run() int { log.Printf("[config] trusted proxies: %v", trusted) } srv := server.New(ctx, mgr, skillDir, *dataDir, keyDir, server.WithSecretScrub(), server.WithTrustedProxies(trusted)) + // ReadTimeout bounds how long the request headers and body may take to arrive. + // It does not limit how long a handler runs, so an event stream stays open. + // WriteTimeout is left unset: a write deadline would cut that stream. httpSrv := &http.Server{ Addr: *addr, Handler: srv.Handler(), ReadHeaderTimeout: 10 * time.Second, + ReadTimeout: 30 * time.Minute, + IdleTimeout: 2 * time.Minute, } go func() { @@ -142,6 +147,9 @@ func run() int { if drainErr != nil { log.Printf("task drain: %v", drainErr) } + // The health writer flushes on ctx.Done(). Wait for it before the deferred + // store close, so the last cooling window is not written at a closed database. + srv.FlushLLMHealth() if httpErr != nil || drainErr != nil { return 1 } diff --git a/db/llm_usage.go b/db/llm_usage.go index a721aa4..5945974 100644 --- a/db/llm_usage.go +++ b/db/llm_usage.go @@ -33,6 +33,24 @@ func (d *DB) EnsureLLMUsageTable() error { return d.execEmbeddedMigration(llmUsageMigration) } +// TaskLLMSpend is the token and turn total the in-memory task ledger would +// already have counted. One row is one turn. The ledger sees planner and worker +// completions, plus judge and compactor calls made on those contexts. Main-agent +// chat, goal decomposition, and side questions write llm_usage rows for the same +// task and never enter the ledger, so they are left out: counting them would +// tighten the turn cap after a restart. Cache read and write are not added: +// input_tokens already includes them. Every status counts, because a failed call +// is metered too. +func (d *DB) TaskLLMSpend(taskID string) (tokens, turns int64, err error) { + if d == nil || taskID == "" { + return 0, 0, nil + } + err = d.QueryRow(`SELECT COALESCE(SUM(input_tokens + output_tokens), 0), COUNT(*) +FROM llm_usage +WHERE task_id=$1 AND worker IN ('planner', 'worker', 'judge', 'compactor')`, taskID).Scan(&tokens, &turns) + return tokens, turns, err +} + // InsertLLMUsage appends one metering row. Best-effort: callers log and continue on // error (a lost metering row must never break the LLM call). func (d *DB) InsertLLMUsage(u *LLMUsage) error { diff --git a/db/task_llm_spend_test.go b/db/task_llm_spend_test.go new file mode 100644 index 0000000..7c807f2 --- /dev/null +++ b/db/task_llm_spend_test.go @@ -0,0 +1,40 @@ +package db + +import ( + "fmt" + "testing" + "time" +) + +// Judge calls share the planner or worker sink and count toward the task cap. +// Chat rows for the same task do not, or a restart would tighten the cap. +func TestTaskLLMSpendCountsJudgeNotChat(t *testing.T) { + d, err := Open(testDSN(t)) + if err != nil { + t.Skipf("postgres unavailable (%v)", err) + } + defer d.Close() + + taskID := fmt.Sprintf("spend-%d", time.Now().UnixNano()) + t.Cleanup(func() { + _, _ = d.Exec(`DELETE FROM llm_usage WHERE task_id=$1`, taskID) + }) + insert := func(worker string, in, out int) { + t.Helper() + if err := d.InsertLLMUsage(&LLMUsage{ + TaskID: taskID, Worker: worker, InputTokens: in, OutputTokens: out, Status: "ok", + }); err != nil { + t.Fatal(err) + } + } + insert("judge", 30, 10) + insert("chat", 1000, 1000) + + tokens, turns, err := d.TaskLLMSpend(taskID) + if err != nil { + t.Fatal(err) + } + if tokens != 40 || turns != 1 { + t.Fatalf("tokens=%d turns=%d, want the judge row only (40 tokens, 1 turn)", tokens, turns) + } +} diff --git a/db/task_scope.go b/db/task_scope.go index bc9defe..bbbdee8 100644 --- a/db/task_scope.go +++ b/db/task_scope.go @@ -297,17 +297,22 @@ type Coverage struct { } // CoverageEnabled reports whether a task has the asset-coverage feature turned on -// (tasks.coverage_enabled). Missing row / error → true (fail open to the default), -// so unknown/legacy tasks keep the historical behavior. taskID<=0 → true. -func (s *AssetStore) CoverageEnabled(taskID int64) bool { +// (tasks.coverage_enabled). A missing row stays enabled, which is the historical +// default for unknown and legacy tasks. taskID<=0 stays enabled. A query error is +// returned with enabled=true so a database blip does not turn the feature off. +func (s *AssetStore) CoverageEnabled(taskID int64) (bool, error) { if taskID <= 0 { - return true + return true, nil } var enabled bool - if err := s.db.QueryRow(`SELECT COALESCE(coverage_enabled,true) FROM tasks WHERE id=$1`, taskID).Scan(&enabled); err != nil { - return true + err := s.db.QueryRow(`SELECT COALESCE(coverage_enabled,true) FROM tasks WHERE id=$1`, taskID).Scan(&enabled) + if err == sql.ErrNoRows { + return true, nil } - return enabled + if err != nil { + return true, err + } + return enabled, nil } // declaredScopeFilter keeps a task_scope row ts in the coverage match only when it is diff --git a/db/task_scope_test.go b/db/task_scope_test.go index 8ac7e8a..25f7d7a 100644 --- a/db/task_scope_test.go +++ b/db/task_scope_test.go @@ -2,6 +2,39 @@ package db import "testing" +func TestCoverageEnabledWithoutTask(t *testing.T) { + on, err := (&AssetStore{}).CoverageEnabled(0) + if err != nil || !on { + t.Fatalf("enabled=%v err=%v", on, err) + } +} + +func TestCoverageEnabledMissingRowStaysOn(t *testing.T) { + d, err := Open(testDSN(t)) + if err != nil { + t.Skipf("postgres unavailable (%v)", err) + } + defer d.Close() + on, err := d.Assets().CoverageEnabled(9_000_000_000_001) + if err != nil || !on { + t.Fatalf("missing task: enabled=%v err=%v, want enabled with no error", on, err) + } +} + +func TestCoverageEnabledQueryErrorStaysOn(t *testing.T) { + d, err := Open(testDSN(t)) + if err != nil { + t.Skipf("postgres unavailable (%v)", err) + } + if err := d.Close(); err != nil { + t.Fatal(err) + } + on, err := d.Assets().CoverageEnabled(1) + if err == nil || !on { + t.Fatalf("failed query: enabled=%v err=%v, want enabled and the error", on, err) + } +} + // TestStripHostPort covers the ip:port / [ipv6]:port stripping used by the // ip/cidr scope path so a target like "10.0.188.136:3000" no longer 404s. func TestStripHostPort(t *testing.T) { diff --git a/docs/traffic-rewrite-rules.md b/docs/traffic-rewrite-rules.md index 59f24f4..8bc92fe 100644 --- a/docs/traffic-rewrite-rules.md +++ b/docs/traffic-rewrite-rules.md @@ -3,9 +3,10 @@ A rewrite rule matches a request or a response that goes through the capture proxy and describes how to change it: the method, the path and query, headers, the body, the status code. -**Status: rules are stored, validated and compiled. The proxy does not apply them yet.** Creating -or enabling a rule changes nothing in what the proxy forwards or records. Applying the rules, the -per-exchange trace, the archive format and the web UI come in a later change. +**Status: the capture proxy applies enabled rules to HTTP traffic (request and response phases), +and records what it did.** Not yet in this change: archive format, a per-replay rule toggle, the web UI, +and the proof flag. See [Application in the capture proxy](#application-in-the-capture-proxy) for +what is covered and what is not. Rules are a convenience for the operator, not a security control: a rule that fails leaves the message unchanged (fail open). @@ -145,3 +146,66 @@ cannot restore them, and ARTEX refuses to start under a key that cannot open the The server compiles the enabled rules into an immutable snapshot at startup and after every rule change, and hands it to the proxy without a restart. + +## Application in the capture proxy + +The proxy applies the snapshot to each exchange in two phases. The request phase runs before the +request is sent to the target; the response phase runs before the response is returned to the +client. Each rule sees the output of the previous one. Only rules whose phase matches the step +are run, and a rule that fails leaves the message as it was (fail open). + +### What is recorded + +For a rewritten exchange, the record keeps the request as the client sent it and the response as +the target sent it. The rewritten values are not stored as the exchange's original data. + +- Each action that ran (applied, skipped or failed) is written in the SQLite table + `exchange_rewrites`, in the same transaction as the exchange itself, with the rule id, a copy of + the rule as it was applied, the action, the outcome and the reason. +- The before value is kept as its SHA-256. The after value is kept as the value itself, inline or + in the blob store when it is large, with its SHA-256. A sensitive action stores neither value nor + plain hash: its after value is not stored, and the hashes it leaves are a keyed HMAC when a key is + available (empty otherwise). The same HMAC applies to later actions on a field a sensitive action + wrote. +- `traffic_get` shows a `=== REWRITTEN ===` line naming the rule ids when the exchange was changed. + The line says that the shown values are the originals. +- `traffic_search` searches only the original values. It never finds a value that a rule wrote. + +Reasons a rule is not applied are recorded in the trace, for example `not rewritten: binary`, +`not rewritten: stream`, or `not rewritten: snapshot capacity`. + +### Streams: SSE and WebSocket + +- **SSE** (`text/event-stream`): status and headers are rewritten at the start of the stream + (SSEStart), since they are sent once. Later event frames pass unchanged. +- **WebSocket**: not rewritten. The capture library calls its start hook after the upgrade is + already done, and it short-circuits the upgrade before the request and response hooks run, so + the handshake status and headers cannot be changed. Frames pass unchanged. The exchange is + traced `not rewritten: stream`. + +### Large responses + +go-mitmproxy does not call the response hook for a body of 5 MiB or more (its `StreamLargeBodies` +default). Such a response is forwarded to the client unchanged and is not recorded in full, and no +response rule runs on it. + +### Snapshot limits + +Each exchange with a rewrite trace (at least one rule matched and produced a line: applied, skipped or failed) keeps an original copy in memory until it finishes. Limits: + +| Limit | Value | +|---|---| +| Entries holding an original request | 1024 | +| Bytes held by those entries | 64 MiB | +| Tombstones (slots whose data was released) | 4096 | +| Time after which the slot of a live flow may be turned into a tombstone (see below) | 15 minutes | + +When there is no room, the request is not rewritten. The exchange is traced `not rewritten: snapshot capacity` when a tombstone slot is free; when the tombstone table is full, the request is refused with no trace line at all (the refusal is counted and logged, at most once a minute). +The slot of a finished flow is swept when its connection closes (at most once per second) and at +least once a minute while slots exist. The 15-minute TTL never deletes a live flow's slot: past it, +the slot may be converted into a tombstone if a new reservation needs the room. + +### Replay + +Replay goes through the recording proxy like any other traffic, so the enabled rules apply to a +replay and the resulting exchange is traced like a captured one. There is no per-replay toggle yet. diff --git a/llmpool/health.go b/llmpool/health.go index 017f6c8..7f3e301 100644 --- a/llmpool/health.go +++ b/llmpool/health.go @@ -46,7 +46,9 @@ type Registry struct { m map[int64]*State // persist / forget mirror state to the DB so a cooling-off window survives a - // restart. Both may be nil (no DB); both are called off the hot path. + // restart. Both may be nil (no DB). Both run while r.mu is held, so a later + // change to the same profile cannot enqueue ahead of this one. They must not + // call back into the registry or wait on database I/O. persist func(id int64, st State) forget func(id int64) @@ -118,16 +120,14 @@ func (r *Registry) IsOpen(id int64) bool { return r.Get(id).Open() } // recovers is fully trusted again (and drops the persisted row). func (r *Registry) Pass(id int64) { r.mu.Lock() + defer r.mu.Unlock() st := r.m[id] if st == nil || (st.Fails == 0 && st.Trips == 0 && st.OpenUntil.IsZero()) { - r.mu.Unlock() - return // already clean — nothing to write + return // already clean; nothing to write } delete(r.m, id) - forget := r.forget - r.mu.Unlock() - if forget != nil { - forget(id) + if r.forget != nil { + r.forget(id) } } @@ -137,6 +137,7 @@ func (r *Registry) Pass(id int64) { // true when this call is what opened the breaker, so the caller can log it once. func (r *Registry) Trip(id int64, errMsg string, hard bool) (tripped bool) { r.mu.Lock() + defer r.mu.Unlock() st := r.m[id] if st == nil { st = &State{} @@ -152,11 +153,8 @@ func (r *Registry) Trip(id int64, errMsg string, hard bool) (tripped bool) { st.Fails = 0 // counted into this trip; start fresh for the half-open probe tripped = true } - snap := *st - persist := r.persist - r.mu.Unlock() - if persist != nil { - persist(id, snap) + if r.persist != nil { + r.persist(id, *st) } return tripped } @@ -164,11 +162,10 @@ func (r *Registry) Trip(id int64, errMsg string, hard bool) (tripped bool) { // Reset clears one profile's state — the UI's "Recover now" action. func (r *Registry) Reset(id int64) { r.mu.Lock() + defer r.mu.Unlock() delete(r.m, id) - forget := r.forget - r.mu.Unlock() - if forget != nil { - forget(id) + if r.forget != nil { + r.forget(id) } } diff --git a/llmpool/health_order_test.go b/llmpool/health_order_test.go new file mode 100644 index 0000000..5767224 --- /dev/null +++ b/llmpool/health_order_test.go @@ -0,0 +1,51 @@ +package llmpool + +import ( + "sync/atomic" + "testing" + "time" +) + +// persist and forget run under the registry lock. A second update on the same +// profile cannot enqueue until the first callback returns, so the queue cannot +// observe the later state first. +func TestHealthCallbacksStayInStateOrder(t *testing.T) { + var entered atomic.Int32 + firstIn := make(chan struct{}) + release := make(chan struct{}) + block := func() { + if entered.Add(1) == 1 { + close(firstIn) + <-release + } + } + reg := NewRegistry( + func(int64, State) { block() }, + func(int64) { block() }, + ) + go reg.Trip(1, "down", true) + select { + case <-firstIn: + case <-time.After(2 * time.Second): + t.Fatal("Trip did not reach persist") + } + passed := make(chan struct{}) + go func() { + reg.Pass(1) + close(passed) + }() + select { + case <-passed: + t.Fatal("Pass enqueued while Trip's persist was still running") + case <-time.After(50 * time.Millisecond): + } + close(release) + select { + case <-passed: + case <-time.After(2 * time.Second): + t.Fatal("Pass did not finish after Trip released the lock") + } + if entered.Load() != 2 { + t.Fatalf("callbacks=%d, want Trip then Pass", entered.Load()) + } +} diff --git a/server/asset_intercept.go b/server/asset_intercept.go index 56b677e..53091f5 100644 --- a/server/asset_intercept.go +++ b/server/asset_intercept.go @@ -36,7 +36,7 @@ func (s *Server) assetInterceptCreateRule(w http.ResponseWriter, r *http.Request return } var req assetInterceptRuleReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -63,7 +63,7 @@ func (s *Server) assetInterceptUpdateRule(w http.ResponseWriter, r *http.Request return } var req assetInterceptRuleReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -109,7 +109,7 @@ func (s *Server) assetInterceptToggleRule(w http.ResponseWriter, r *http.Request var req struct { Enabled bool `json:"enabled"` } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } diff --git a/server/auth.go b/server/auth.go index b3532d1..3b7b657 100644 --- a/server/auth.go +++ b/server/auth.go @@ -425,7 +425,7 @@ func (s *Server) authInit(w http.ResponseWriter, r *http.Request) { Password string `json:"password"` SetupToken string `json:"setup_token"` } - if err := decode(r, &req); err != nil || req.Password == "" { + if err := decode(w, r, &req); err != nil || req.Password == "" { writeErr(w, 400, "Password cannot be empty") return } @@ -483,7 +483,7 @@ func (s *Server) authChangePassword(w http.ResponseWriter, r *http.Request) { OldPassword string `json:"old_password"` NewPassword string `json:"new_password"` } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, "Invalid request format") return } @@ -549,7 +549,7 @@ func (s *Server) login(w http.ResponseWriter, r *http.Request, getSetting func(s Username string `json:"username"` Password string `json:"password"` } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, "Invalid request format") return } diff --git a/server/conversations.go b/server/conversations.go index 92acd1a..7c0d916 100644 --- a/server/conversations.go +++ b/server/conversations.go @@ -27,7 +27,7 @@ const ( func decodeConversationRequest(w http.ResponseWriter, r *http.Request, value any) bool { r.Body = http.MaxBytesReader(w, r.Body, maxConversationRequestBytes) - if err := decode(r, value); err != nil { + if err := decode(w, r, value); err != nil { var tooLarge *http.MaxBytesError if errors.As(err, &tooLarge) { writeErr(w, http.StatusRequestEntityTooLarge, "Request body too large") @@ -396,7 +396,7 @@ func (s *Server) pgSendConversationMessage(w http.ResponseWriter, r *http.Reques Message string `json:"message"` Attachments []chatAttachment `json:"attachments,omitempty"` // files uploaded via method 1 (paths relative to the conversation working dir) } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } diff --git a/server/customtool.go b/server/customtool.go index e10f09a..09531ae 100644 --- a/server/customtool.go +++ b/server/customtool.go @@ -52,7 +52,7 @@ func (s *Server) pgCreateCustomTool(w http.ResponseWriter, r *http.Request) { return } var req customToolReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -104,7 +104,7 @@ func (s *Server) pgUpdateCustomTool(w http.ResponseWriter, r *http.Request) { return } var req customToolReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } diff --git a/server/decode_limit_test.go b/server/decode_limit_test.go new file mode 100644 index 0000000..5922673 --- /dev/null +++ b/server/decode_limit_test.go @@ -0,0 +1,69 @@ +package server + +import ( + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +func TestDecodeAcceptsSmallBody(t *testing.T) { + w := httptest.NewRecorder() + r := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(`{"x":"ok"}`)) + var v map[string]string + if err := decode(w, r, &v); err != nil || v["x"] != "ok" { + t.Fatalf("decode: %v %+v", err, v) + } +} + +func TestDecodeCapsBody(t *testing.T) { + w := httptest.NewRecorder() + body := strings.NewReader(`{"x":"` + strings.Repeat("a", decodeMaxBytes) + `"}`) + r := httptest.NewRequest(http.MethodPost, "/", body) + var v map[string]string + err := decode(w, r, &v) + var tooLarge *http.MaxBytesError + if !errors.As(err, &tooLarge) { + t.Fatalf("err=%v, want the body cap", err) + } +} + +func TestRawJSONBodyOverTheCapIs413(t *testing.T) { + h := limitRequestBody(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var v map[string]string + if err := json.NewDecoder(r.Body).Decode(&v); err != nil { + writeErr(w, 400, err.Error()) + return + } + writeJSON(w, 200, v) + })) + small := httptest.NewRecorder() + h.ServeHTTP(small, httptest.NewRequest(http.MethodPost, "/", strings.NewReader(`{"x":"ok"}`))) + if small.Code != 200 { + t.Fatalf("small body status=%d", small.Code) + } + large := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(`{"x":"`+strings.Repeat("a", decodeMaxBytes)+`"}`)) + h.ServeHTTP(large, req) + if large.Code != http.StatusRequestEntityTooLarge { + t.Fatalf("status=%d body=%s, want 413", large.Code, large.Body.String()) + } +} + +func TestMultipartBodyIsNotGivenTheJSONCap(t *testing.T) { + h := limitRequestBody(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if _, ok := r.Body.(*limitedBody); ok { + t.Errorf("multipart body was given the JSON cap") + } + w.WriteHeader(http.StatusNoContent) + })) + r := httptest.NewRequest(http.MethodPost, "/", strings.NewReader("not-read")) + r.Header.Set("Content-Type", "multipart/form-data; boundary=abc") + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + if w.Code != http.StatusNoContent { + t.Fatalf("status=%d", w.Code) + } +} diff --git a/server/engine_budget.go b/server/engine_budget.go index b6021b2..fc6d45a 100644 --- a/server/engine_budget.go +++ b/server/engine_budget.go @@ -17,8 +17,36 @@ func (e *Engine) budgetFor(taskID string) *agent.BudgetLedger { if v, ok := e.budgets.Load(taskID); ok { return v.(*agent.BudgetLedger) } - v, _ := e.budgets.LoadOrStore(taskID, agent.NewBudgetLedger(agent.BudgetsFromEnv())) - return v.(*agent.BudgetLedger) + created := agent.NewBudgetLedger(agent.BudgetsFromEnv()) + e.seedBudget(taskID, created) + v, loaded := e.budgets.LoadOrStore(taskID, created) + if loaded { + return v.(*agent.BudgetLedger) + } + return created +} + +// seedBudget rebuilds spent tokens and turns from llm_usage for every new +// ledger. budgetFor calls it on first use, and again after resetSettledRun or +// StopTask deletes the ledger in this same process. The in-memory sink runs +// before the row is inserted, so a call that happens after this seed is +// counted once. A nil ledger means no budget is configured and nothing is read. +// The restore does not rebuild per-worker averages, so the next worker estimate +// starts from the defaults until a worker finishes against this ledger. +func (e *Engine) seedBudget(taskID string, ledger *agent.BudgetLedger) { + if ledger == nil || e == nil || e.m == nil || e.m.pg == nil || taskID == "" { + return + } + tokens, turns, err := e.m.pg.TaskLLMSpend(taskID) + if err != nil { + log.Printf("[budget] task %s: cannot restore spend from llm_usage: %v", taskID, err) + return + } + if tokens == 0 && turns == 0 { + return + } + ledger.RestoreSpend(agent.Spend{Tokens: tokens, Turns: turns}) + log.Printf("[budget] task %s restored %d tokens / %d turns from llm_usage", taskID, tokens, turns) } // budgetStop is the enforcement point of a planner round: it logs the remaining diff --git a/server/engine_budget_test.go b/server/engine_budget_test.go index 26365bd..c3a8ebd 100644 --- a/server/engine_budget_test.go +++ b/server/engine_budget_test.go @@ -2,9 +2,12 @@ package server import ( "context" + "strconv" "testing" + "time" "github.com/Autumn-27/artex/agent" + pgdb "github.com/Autumn-27/artex/db" "github.com/Autumn-27/norma/llm" ) @@ -67,3 +70,64 @@ func TestResetSettledRunRestartsBudget(t *testing.T) { t.Fatalf("revived run: ledger not fresh, spent %d tokens", got) } } + +// With Postgres, resetSettledRun drops the ledger and the next one is seeded +// again from llm_usage. In-memory spend that was never written does not come +// back. A restore that lands on the wrap-up reserve leaves only that reserve, +// and the worker estimate starts from the defaults. +func TestResetSettledRunReseedsBudgetFromUsage(t *testing.T) { + for _, k := range []string{"ARTEX_TASK_BUDGET_TURNS", "ARTEX_INTENT_BUDGET_TOKENS", "ARTEX_INTENT_BUDGET_TURNS"} { + t.Setenv(k, "") + } + t.Setenv("ARTEX_TASK_BUDGET_TOKENS", "100000") + m, err := NewManager(t.TempDir(), "") + if err != nil { + t.Skipf("postgres unavailable (%v)", err) + } + defer m.Close() + + taskID := strconv.FormatInt(time.Now().UnixNano(), 10) + t.Cleanup(func() { + _, _ = m.pg.Exec(`DELETE FROM llm_usage WHERE task_id=$1`, taskID) + }) + // 100000 - wrap-up reserve (max(5%, 20000)) = 80000. One planner row. + if err := m.pg.InsertLLMUsage(&pgdb.LLMUsage{ + TaskID: taskID, Worker: "planner", InputTokens: 80_000, Status: "ok", + }); err != nil { + t.Fatal(err) + } + e := NewEngine(m) + task := &Task{ID: taskID} + l := e.budgetFor(task.ID) + st := l.State() + if st.Spent.Tokens != 80_000 || st.Spent.Turns != 1 { + t.Fatalf("seed spent=%+v, want 80000 tokens and 1 turn", st.Spent) + } + if st.Remaining.Tokens != st.Margin.Tokens || !st.AtMargin || st.Dispatch { + t.Fatalf("restored task near the cap: remaining=%d margin=%d at=%v dispatch=%v", + st.Remaining.Tokens, st.Margin.Tokens, st.AtMargin, st.Dispatch) + } + if st.Estimate.Tokens != agent.DefaultWorkerEstimateTokens || st.Estimate.Turns != agent.DefaultWorkerEstimateTurns { + t.Fatalf("estimate=%+v, want the defaults (worker history is not restored)", st.Estimate) + } + + l.Record(llm.Usage{InputTokens: 1000}) + l.BeginWrapup() + e.settling.Store(task.ID, true) + e.resetSettledRun(task.ID) + if e.isSettling(task.ID) { + t.Fatal("revived run is still marked settling") + } + again := e.budgetFor(task.ID) + if again == l { + t.Fatal("revive reused the deleted ledger") + } + got := again.State() + if got.Spent.Tokens != 80_000 || got.Spent.Turns != 1 { + t.Fatalf("reseed spent=%+v, want the usage row and not the in-memory extra", got.Spent) + } + if got.Remaining.Tokens != got.Margin.Tokens || got.Dispatch { + t.Fatalf("reseed still past the wrap-up margin: remaining=%d margin=%d dispatch=%v", + got.Remaining.Tokens, got.Margin.Tokens, got.Dispatch) + } +} diff --git a/server/engine_timeout.go b/server/engine_timeout.go index c8869c3..fbd0ab2 100644 --- a/server/engine_timeout.go +++ b/server/engine_timeout.go @@ -110,7 +110,8 @@ func (e *Engine) resetTimeoutRevival(taskID string) { } // resetSettledRun clears the settle state of a finished run (settling and the -// budget ledger) so a revived task can plan and claim again with a fresh budget. +// budget ledger) so a revived task can plan and claim again. The next ledger +// is seeded from llm_usage, so the spend already recorded still counts. // It also clears the persisted end_reason of the previous run (ApplyTaskAdmission // already did in the same UPDATE as the revive; this is idempotent and never // touches a task that is terminal again). diff --git a/server/finding_workflow.go b/server/finding_workflow.go index 8b11fac..8e2ecdc 100644 --- a/server/finding_workflow.go +++ b/server/finding_workflow.go @@ -29,6 +29,16 @@ func (s *Server) seedFindingWorkflowTools() { _ = s.m.pg.SetSetting(hostSearchDescriptionFlag, "true") } } + const getRewriteDescriptionFlag = "traffic_get_rewrite_description_v1" + if value, _, _ := s.m.pg.GetSetting(getRewriteDescriptionFlag); value != "true" { + // Same rule: only the original built-in text is replaced. + legacy := "Fetch the raw request/response of one captured traffic record by id (truncated when too large). Use together with traffic_search to avoid re-curling. A WebSocket (status 101) or SSE (text/event-stream) record shows only its handshake here; read its messages with traffic_messages." + if _, err := s.m.pg.Exec(`UPDATE tools SET description=$1,updated_at=now() WHERE key='traffic_get' AND system AND description=$2`, traffic.TrafficGetDescription, legacy); err != nil { + log.Printf("[evidence] upgrade traffic_get description: %v", err) + } else { + _ = s.m.pg.SetSetting(getRewriteDescriptionFlag, "true") + } + } const flag = "finding_workflow_tools_v2_reporter" if value, _, _ := s.m.pg.GetSetting(flag); value == "true" { return diff --git a/server/intercept.go b/server/intercept.go index b4b3542..179d0c4 100644 --- a/server/intercept.go +++ b/server/intercept.go @@ -141,7 +141,7 @@ func (s *Server) interceptCreateRule(w http.ResponseWriter, r *http.Request) { return } var req interceptRuleReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -169,7 +169,7 @@ func (s *Server) interceptUpdateRule(w http.ResponseWriter, r *http.Request) { return } var req interceptRuleReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -217,7 +217,7 @@ func (s *Server) interceptToggleRule(w http.ResponseWriter, r *http.Request) { var req struct { Enabled bool `json:"enabled"` } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -400,7 +400,7 @@ func (s *Server) interceptDecide(w http.ResponseWriter, r *http.Request) { var req struct { Decision string `json:"decision"` // "allowed" | "denied" } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -445,7 +445,7 @@ func (s *Server) interceptBatchDecide(w http.ResponseWriter, r *http.Request) { IDs []int64 `json:"ids"` Decision string `json:"decision"` // "allowed" | "denied" } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -647,7 +647,7 @@ func (s *Server) targetGateSetExemptions(w http.ResponseWriter, r *http.Request) var req struct { Exemptions []db.TargetExemption `json:"exemptions"` } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -697,7 +697,7 @@ func (s *Server) interceptSetToolConfig(w http.ResponseWriter, r *http.Request) var req struct { EnabledTools []string `json:"enabled_tools"` } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -722,7 +722,7 @@ func (s *Server) interceptGetJudgeConfig(w http.ResponseWriter, r *http.Request) // interceptSetJudgeConfig persists the judge configuration. func (s *Server) interceptSetJudgeConfig(w http.ResponseWriter, r *http.Request) { var req intercept.JudgeConfig - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } diff --git a/server/llmpool.go b/server/llmpool.go index a3901fc..bbb1d1c 100644 --- a/server/llmpool.go +++ b/server/llmpool.go @@ -1,7 +1,9 @@ package server import ( + "context" "log" + "sync" "time" "github.com/Autumn-27/artex/agent" @@ -17,26 +19,19 @@ import ( // - Circuit-breaker state is process-wide (s.llmHealth); rebuilding the pool does not clear it. // newLLMHealthRegistry builds the process-wide circuit-breaker registry, mirroring -// state into PG so a cooling-off window survives a restart. Writes are async and -// best-effort — the in-memory copy is authoritative. -func newLLMHealthRegistry(pg *db.DB) *llmpool.Registry { +// state into PG so a cooling-off window survives a restart. Writes are async. +// The in-memory copy is authoritative. The returned syncer is nil when there is +// no database; otherwise it flushes when ctx is cancelled. +func newLLMHealthRegistry(ctx context.Context, pg *db.DB) (*llmpool.Registry, *healthSync) { if pg == nil { - return llmpool.NewRegistry(nil, nil) + return llmpool.NewRegistry(nil, nil), nil } + syncer := newHealthSync(ctx, pg.SaveLLMHealth, pg.ClearLLMHealth) persist := func(id int64, st llmpool.State) { - h := db.LLMHealth{ProfileID: id, Fails: st.Fails, Trips: st.Trips, LastError: st.LastError} - if !st.OpenUntil.IsZero() { - t := st.OpenUntil - h.OpenUntil = &t - } - go func() { - if err := pg.SaveLLMHealth(h); err != nil { - log.Printf("[llmpool] failed to persist circuit-breaker state: %v", err) - } - }() + syncer.enqueue(id, healthOp{st: st}) } forget := func(id int64) { - go func() { _ = pg.ClearLLMHealth(id) }() + syncer.enqueue(id, healthOp{clear: true}) } reg := llmpool.NewRegistry(persist, forget) // Restore only windows that haven't expired (LoadLLMHealth filters), so a @@ -51,7 +46,199 @@ func newLLMHealthRegistry(pg *db.DB) *llmpool.Registry { log.Printf("[llmpool] restored circuit-breaker state: profile #%d cooling until %s", h.ProfileID, st.OpenUntil.Format(time.RFC3339)) } } - return reg + return reg, syncer +} + +// healthOp is one circuit-breaker write for a profile. clear drops the row. +// seq orders writes for that profile: a newer op wins over one still in flight. +type healthOp struct { + seq uint64 + clear bool + st llmpool.State +} + +// healthSync writes circuit-breaker state from one goroutine. A newer update for +// the same profile replaces one that has not been written yet, so a burst of +// state changes becomes one database call. The database call runs without mu, +// so enqueue does not wait on a slow write. writeMu keeps those calls in order. +// Before each call, an op that is already older than the applied or pending +// sequence is dropped. save and clear must not call enqueue. +type healthSync struct { + save func(db.LLMHealth) error + clear func(int64) error + retryAfter time.Duration // tests set a short delay; zero uses one second + mu sync.Mutex + writeMu sync.Mutex // orders database calls; enqueue does not take it + pending map[int64]healthOp + applied map[int64]uint64 + seq uint64 + stopped bool + wake chan struct{} + done chan struct{} +} + +func newHealthSync(ctx context.Context, save func(db.LLMHealth) error, clear func(int64) error) *healthSync { + if ctx == nil { + ctx = context.Background() + } + h := &healthSync{ + save: save, + clear: clear, + pending: map[int64]healthOp{}, + applied: map[int64]uint64{}, + wake: make(chan struct{}, 1), + done: make(chan struct{}), + } + go h.loop(ctx) + return h +} + +// FlushLLMHealth waits until the circuit-breaker writer has flushed the ops +// queued before shutdown. Call it after the server context is cancelled and +// before the database closes. +func (s *Server) FlushLLMHealth() { + if s == nil || s.llmHealthSync == nil { + return + } + select { + case <-s.llmHealthSync.done: + case <-time.After(5 * time.Second): + log.Printf("[llmpool] timed out flushing circuit-breaker state") + } +} + +func (h *healthSync) enqueue(id int64, op healthOp) { + h.mu.Lock() + h.seq++ + op.seq = h.seq + if h.stopped { + h.mu.Unlock() + if err := h.apply(id, op); err != nil { + log.Printf("[llmpool] write circuit-breaker profile #%d: %v", id, err) + } + return + } + if prev, ok := h.pending[id]; ok && prev.seq > op.seq { + h.mu.Unlock() + return + } + h.pending[id] = op + h.mu.Unlock() + select { + case h.wake <- struct{}{}: + default: + } +} + +func (h *healthSync) take() map[int64]healthOp { + h.mu.Lock() + defer h.mu.Unlock() + if len(h.pending) == 0 { + return nil + } + out := h.pending + h.pending = map[int64]healthOp{} + return out +} + +func (h *healthSync) restore(id int64, op healthOp) { + h.mu.Lock() + defer h.mu.Unlock() + if h.stopped || h.applied[id] >= op.seq { + return + } + if cur, ok := h.pending[id]; ok && cur.seq >= op.seq { + return + } + h.pending[id] = op +} + +// superseded reports whether a newer op for id is already applied or waiting. +// The caller holds writeMu, so the decision and the database call stay in order. +func (h *healthSync) superseded(id int64, seq uint64) bool { + h.mu.Lock() + defer h.mu.Unlock() + if seq > 0 && h.applied[id] >= seq { + return true + } + cur, ok := h.pending[id] + return ok && cur.seq > seq +} + +func (h *healthSync) apply(id int64, op healthOp) error { + h.writeMu.Lock() + defer h.writeMu.Unlock() + if h.superseded(id, op.seq) { + return nil + } + var err error + if op.clear { + if h.clear != nil { + err = h.clear(id) + } + } else if h.save != nil { + row := db.LLMHealth{ProfileID: id, Fails: op.st.Fails, Trips: op.st.Trips, LastError: op.st.LastError} + if !op.st.OpenUntil.IsZero() { + openUntil := op.st.OpenUntil + row.OpenUntil = &openUntil + } + err = h.save(row) + } + if err != nil { + return err + } + h.mu.Lock() + if h.applied[id] < op.seq { + h.applied[id] = op.seq + } + h.mu.Unlock() + return nil +} + +func (h *healthSync) write(batch map[int64]healthOp) (failed bool) { + for id, op := range batch { + if err := h.apply(id, op); err != nil { + log.Printf("[llmpool] write circuit-breaker profile #%d: %v", id, err) + h.restore(id, op) + failed = true + } + } + return failed +} + +func (h *healthSync) finish() { + h.mu.Lock() + h.stopped = true + batch := h.pending + h.pending = map[int64]healthOp{} + h.mu.Unlock() + for id, op := range batch { + if err := h.apply(id, op); err != nil { + log.Printf("[llmpool] write circuit-breaker profile #%d: %v", id, err) + } + } +} + +func (h *healthSync) loop(ctx context.Context) { + defer close(h.done) + for { + failed := h.write(h.take()) + delay := h.retryAfter + if delay <= 0 { + delay = time.Second + } + var retry <-chan time.Time + if failed { + retry = time.After(delay) + } + select { + case <-ctx.Done(): + h.finish() + return + case <-h.wake: + case <-retry: + } + } } // poolMember builds one chain member from a profile, reusing the per-profile diff --git a/server/llmpool_health_test.go b/server/llmpool_health_test.go new file mode 100644 index 0000000..c9e7a1e --- /dev/null +++ b/server/llmpool_health_test.go @@ -0,0 +1,183 @@ +package server + +import ( + "context" + "errors" + "sync" + "testing" + "time" + + "github.com/Autumn-27/artex/db" + "github.com/Autumn-27/artex/llmpool" +) + +func TestHealthSyncLoopWritesLatest(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + var mu sync.Mutex + saved := map[int64]db.LLMHealth{} + cleared := map[int64]int{} + savedCh := make(chan struct{}, 4) + clearedCh := make(chan struct{}, 4) + h := newHealthSync(ctx, + func(row db.LLMHealth) error { + mu.Lock() + saved[row.ProfileID] = row + mu.Unlock() + savedCh <- struct{}{} + return nil + }, + func(id int64) error { + mu.Lock() + cleared[id]++ + delete(saved, id) + mu.Unlock() + clearedCh <- struct{}{} + return nil + }, + ) + openUntil := time.Now().Add(time.Minute).UTC().Truncate(time.Second) + h.enqueue(8, healthOp{st: llmpool.State{Fails: 1, LastError: "first"}}) + h.enqueue(8, healthOp{st: llmpool.State{Fails: 4, Trips: 1, OpenUntil: openUntil, LastError: "down"}}) + deadline := time.After(2 * time.Second) + var latest db.LLMHealth + for { + mu.Lock() + latest = saved[8] + mu.Unlock() + if latest.Fails == 4 && latest.OpenUntil != nil && latest.OpenUntil.Equal(openUntil) { + break + } + select { + case <-savedCh: + case <-deadline: + t.Fatalf("loop saved=%+v, want the latest trip", latest) + } + } + h.enqueue(7, healthOp{clear: true}) + select { + case <-clearedCh: + case <-time.After(2 * time.Second): + t.Fatal("loop did not clear") + } + mu.Lock() + defer mu.Unlock() + got := saved[8] + if got.Fails != 4 || got.Trips != 1 || got.LastError != "down" || got.OpenUntil == nil || !got.OpenUntil.Equal(openUntil) { + t.Fatalf("saved=%+v, want the latest trip including OpenUntil", got) + } + if cleared[7] != 1 { + t.Fatalf("cleared=%v", cleared) + } +} + +func TestHealthSyncRetriesFailedSave(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + var mu sync.Mutex + calls := 0 + done := make(chan struct{}) + h := newHealthSync(ctx, + func(row db.LLMHealth) error { + mu.Lock() + calls++ + n := calls + mu.Unlock() + if n == 1 { + return errors.New("database down") + } + if n == 2 { + close(done) + } + return nil + }, + func(int64) error { return nil }, + ) + h.retryAfter = 20 * time.Millisecond + h.enqueue(3, healthOp{st: llmpool.State{Fails: 2, LastError: "blip"}}) + select { + case <-done: + case <-time.After(2 * time.Second): + t.Fatal("failed save was not retried") + } + mu.Lock() + defer mu.Unlock() + if calls < 2 { + t.Fatalf("save calls=%d, want a retry", calls) + } +} + +func TestHealthSyncEnqueueDoesNotWaitOnSave(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + started := make(chan struct{}) + release := make(chan struct{}) + later := make(chan struct{}) + h := newHealthSync(ctx, + func(row db.LLMHealth) error { + if row.Fails == 1 { + close(started) + <-release + return nil + } + if row.Fails == 9 { + close(later) + } + return nil + }, + func(int64) error { return nil }, + ) + h.enqueue(4, healthOp{st: llmpool.State{Fails: 1}}) + select { + case <-started: + case <-time.After(2 * time.Second): + t.Fatal("save did not start") + } + done := make(chan struct{}) + go func() { + h.enqueue(4, healthOp{st: llmpool.State{Fails: 9, LastError: "later"}}) + close(done) + }() + select { + case <-done: + case <-time.After(200 * time.Millisecond): + t.Fatal("enqueue blocked while a save was in flight") + } + close(release) + select { + case <-later: + case <-time.After(2 * time.Second): + t.Fatal("the later state was not written") + } +} + +func TestHealthSyncDropsStaleWrite(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + writes := 0 + h := newHealthSync(ctx, + func(db.LLMHealth) error { + writes++ + return nil + }, + func(int64) error { return nil }, + ) + h.mu.Lock() + h.seq = 4 + h.applied[5] = 2 + h.pending[6] = healthOp{seq: 4, clear: true} + h.mu.Unlock() + if err := h.apply(5, healthOp{seq: 1, st: llmpool.State{Fails: 1}}); err != nil { + t.Fatal(err) + } + if err := h.apply(6, healthOp{seq: 3, st: llmpool.State{Fails: 3}}); err != nil { + t.Fatal(err) + } + if writes != 0 { + t.Fatalf("stale writes=%d", writes) + } +} diff --git a/server/server.go b/server/server.go index aeba273..4bb6f7a 100644 --- a/server/server.go +++ b/server/server.go @@ -128,7 +128,9 @@ type Server struct { // It deliberately lives OUTSIDE the provider caches: rebuilding the chain // (saving an unrelated profile, flipping a setting) must not erase what we // learned about which backends are out of credit / rate-limited. - llmHealth *llmpool.Registry + // llmHealthSync flushes that state when ctx is cancelled, before the database closes. + llmHealth *llmpool.Registry + llmHealthSync *healthSync // Explicit task chains use one task-scoped dynamic provider shared by goals, // planner, workers, and the main agent. Bundles are stable; their runtime reads @@ -174,9 +176,10 @@ func New(ctx context.Context, m *Manager, skillDir string, dataDir string, keyDi chatCancel: map[string]context.CancelCauseFunc{}, triggerQ: map[string][]triggeredRun{}, triggerActive: map[string]int{}, triggerCfg: map[string]triggerBehavior{}, profChatAgents: map[int64]*agent.ChatAgent{}, - provByProfile: map[int64]*provEntry{}, llmHealth: newLLMHealthRegistry(m.pg), - taskAgents: map[string]*taskAgentBundle{}, archiveWake: make(chan struct{}, 1), + provByProfile: map[int64]*provEntry{}, + taskAgents: map[string]*taskAgentBundle{}, archiveWake: make(chan struct{}, 1), loginLimit: newLoginLimiter(time.Now)} + s.llmHealth, s.llmHealthSync = newLLMHealthRegistry(ctx, m.pg) for _, o := range opts { o(s) } @@ -1036,7 +1039,7 @@ func (s *Server) Handler() http.Handler { root := http.NewServeMux() root.Handle("/api/", api) root.Handle("/", s.webuiHandler()) - return root + return limitRequestBody(root) } // --- handlers --- diff --git a/server/server_mgmt.go b/server/server_mgmt.go index c51e9fb..0367e46 100644 --- a/server/server_mgmt.go +++ b/server/server_mgmt.go @@ -9,6 +9,7 @@ import ( "fmt" "io" "log" + "mime" "net/http" "net/url" "os" @@ -79,7 +80,87 @@ func pathInt(r *http.Request, name string) (int64, bool) { return n, err == nil } -func decode(r *http.Request, v any) error { return json.NewDecoder(r.Body).Decode(v) } +// decodeMaxBytes is the default JSON body cap. Multipart uploads are not capped +// here; those handlers set their own limit. A smaller MaxBytesReader installed +// by a handler still wins, because it sits inside this one. +const decodeMaxBytes = 32 << 20 + +// limitedBody records a MaxBytesError so a later 400 can be answered as 413. +type limitedBody struct { + io.ReadCloser + tooLarge bool +} + +func (b *limitedBody) Read(p []byte) (int, error) { + n, err := b.ReadCloser.Read(p) + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + b.tooLarge = true + } + return n, err +} + +type bodyLimitWriter struct { + http.ResponseWriter + tooLarge func() bool +} + +func (b *bodyLimitWriter) WriteHeader(code int) { + if code == http.StatusBadRequest && b.tooLarge != nil && b.tooLarge() { + code = http.StatusRequestEntityTooLarge + } + b.ResponseWriter.WriteHeader(code) +} + +func (b *bodyLimitWriter) Flush() { + if f, ok := b.ResponseWriter.(http.Flusher); ok { + f.Flush() + } +} + +func (b *bodyLimitWriter) Unwrap() http.ResponseWriter { return b.ResponseWriter } + +// limitRequestBody caps a non-multipart body at decodeMaxBytes and passes the +// real ResponseWriter into MaxBytesReader. A handler that then answers 400 for +// that read is rewritten to 413. +func limitRequestBody(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Body == nil || r.Body == http.NoBody || isMultipart(r) { + next.ServeHTTP(w, r) + return + } + body := &limitedBody{ReadCloser: http.MaxBytesReader(w, r.Body, decodeMaxBytes)} + r.Body = body + next.ServeHTTP(&bodyLimitWriter{ResponseWriter: w, tooLarge: func() bool { return body.tooLarge }}, r) + }) +} + +func isMultipart(r *http.Request) bool { + media, _, err := mime.ParseMediaType(r.Header.Get("Content-Type")) + if err != nil { + return false + } + return strings.HasPrefix(media, "multipart/") +} + +func decode(w http.ResponseWriter, r *http.Request, v any) error { + if r.Body != nil { + if _, already := r.Body.(*limitedBody); !already { + r.Body = &limitedBody{ReadCloser: http.MaxBytesReader(w, r.Body, decodeMaxBytes)} + } + } + err := json.NewDecoder(r.Body).Decode(v) + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + if lb, ok := r.Body.(*limitedBody); ok { + lb.tooLarge = true + } + // Return the original error: a caller that maps 413 itself keeps working, + // and the middleware still rewrites a 400 for callers that do not. + return tooLarge + } + return err +} // ---------- tasks (delete) ---------- @@ -138,7 +219,7 @@ func (s *Server) pgDeleteTask(w http.ResponseWriter, r *http.Request) { return } var opts DeleteTaskOptions - if err := decode(r, &opts); err != nil && err != io.EOF { + if err := decode(w, r, &opts); err != nil && err != io.EOF { writeErr(w, 400, "invalid JSON: "+err.Error()) return } @@ -250,7 +331,7 @@ func (s *Server) pgCreateAgent(w http.ResponseWriter, r *http.Request) { return } var req struct{ Key, Name, Description string } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -290,7 +371,7 @@ func (s *Server) pgUpdateAgent(w http.ResponseWriter, r *http.Request) { return } var req struct{ Name, Description string } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -516,7 +597,7 @@ func (s *Server) pgSavePrompt(w http.ResponseWriter, r *http.Request) { return } var body struct{ Template, Note string } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -565,7 +646,7 @@ func (s *Server) pgSaveWrapup(w http.ResponseWriter, r *http.Request) { Prompt string `json:"prompt"` MaxTurns *int `json:"max_turns"` } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -619,7 +700,7 @@ func (s *Server) pgSaveTaskTimeoutWrapup(w http.ResponseWriter, r *http.Request) Prompt string `json:"prompt"` MaxTurns *int `json:"max_turns"` } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -689,7 +770,7 @@ func (s *Server) pgPreviewPrompt(w http.ResponseWriter, r *http.Request) { Template string `json:"template"` Sample map[string]string `json:"sample"` } - _ = decode(r, &body) + _ = decode(w, r, &body) vars, _ := pg.PromptVars(a.ID) if body.Template == "" { body.Template, _ = pg.CurrentPrompt(a.ID) @@ -724,7 +805,7 @@ func (s *Server) pgSetAgentVisibility(w http.ResponseWriter, r *http.Request) { MCP []int64 `json:"mcp"` Skill []string `json:"skill"` } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -797,7 +878,7 @@ func (s *Server) pgUpdateTool(w http.ResponseWriter, r *http.Request) { Agents []string `json:"agents"` Enabled bool `json:"enabled"` } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -963,7 +1044,7 @@ func (s *Server) pgSaveMCP(w http.ResponseWriter, r *http.Request) { return } var m db.MCPServer - if err := decode(r, &m); err != nil { + if err := decode(w, r, &m); err != nil { writeErr(w, 400, err.Error()) return } @@ -1182,7 +1263,7 @@ func (s *Server) fsCreateSkill(w http.ResponseWriter, r *http.Request) { MCPs []string `json:"mcps"` // optional; MCP servers this skill unlocks Instructions string `json:"instructions"` // optional; scaffolded if empty } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -1267,7 +1348,7 @@ func (s *Server) fsUpdateSkillMeta(w http.ResponseWriter, r *http.Request) { License *string `json:"license"` Compatibility *string `json:"compatibility"` } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -1788,7 +1869,7 @@ func (s *Server) fsWriteFile(w http.ResponseWriter, r *http.Request) { var body struct { Content string `json:"content"` } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -1819,7 +1900,7 @@ func (s *Server) fsCreateDir(w http.ResponseWriter, r *http.Request) { var body struct { Path string `json:"path"` } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -1888,7 +1969,7 @@ func (s *Server) pgToggleSkillVisibility(w http.ResponseWriter, r *http.Request) SkillName string `json:"skill_name"` Visible bool `json:"visible"` } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -1926,7 +2007,7 @@ func (s *Server) pgToggleVisibility(w http.ResponseWriter, r *http.Request) { ResourceID int64 `json:"resource_id"` Visible bool `json:"visible"` } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -1967,7 +2048,7 @@ func (s *Server) pgSaveProfile(w http.ResponseWriter, r *http.Request) { APIKey string `json:"api_key"` Streaming *bool `json:"streaming"` } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -2036,7 +2117,7 @@ func (s *Server) pgSaveLLMRetryPolicy(w http.ResponseWriter, r *http.Request) { return } var pol db.LLMRetryPolicy - if err := decode(r, &pol); err != nil { + if err := decode(w, r, &pol); err != nil { writeErr(w, 400, err.Error()) return } @@ -2108,7 +2189,7 @@ func (s *Server) pgActivateProfile(w http.ResponseWriter, r *http.Request) { var body struct { ID int64 `json:"id"` } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -2141,7 +2222,7 @@ func (s *Server) pgLLMPoolReset(w http.ResponseWriter, r *http.Request) { var body struct { ID int64 `json:"id"` // 0 / omitted = all } - if err := decode(r, &body); err != nil { + if err := decode(w, r, &body); err != nil { writeErr(w, 400, err.Error()) return } @@ -2168,7 +2249,7 @@ func (s *Server) pgListModels(w http.ResponseWriter, r *http.Request) { Proxy string `json:"proxy"` ProfileID *int64 `json:"profile_id"` // fallback: use stored key from this profile } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } diff --git a/server/sync_scopesentry.go b/server/sync_scopesentry.go index 76372b5..849357c 100644 --- a/server/sync_scopesentry.go +++ b/server/sync_scopesentry.go @@ -124,7 +124,7 @@ func (s *Server) syncSSDatasource(w http.ResponseWriter, r *http.Request) { URL string `json:"url"` APIKey string `json:"api_key"` } - _ = decode(r, &body) // empty body = just create the placeholder + _ = decode(w, r, &body) // empty body = just create the placeholder body.URL = strings.TrimSpace(body.URL) body.APIKey = strings.TrimSpace(body.APIKey) @@ -266,7 +266,7 @@ func (s *Server) syncSSRun(w http.ResponseWriter, r *http.Request) { return } var req ssSyncReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, "invalid JSON: "+err.Error()) return } diff --git a/server/task_archives.go b/server/task_archives.go index b73dfb2..836c2d8 100644 --- a/server/task_archives.go +++ b/server/task_archives.go @@ -512,7 +512,7 @@ func (s *Server) queueTaskArchiveDelete(w http.ResponseWriter, r *http.Request) func (s *Server) queueTaskArchivesBatch(w http.ResponseWriter, r *http.Request) { var request archiveBatchRequest - if err := decode(r, &request); err != nil { + if err := decode(w, r, &request); err != nil { writeErr(w, 400, err.Error()) return } @@ -552,7 +552,7 @@ func (s *Server) queueTaskArchivesBatch(w http.ResponseWriter, r *http.Request) func (s *Server) queueTaskArchiveActionBatch(w http.ResponseWriter, r *http.Request, action string) { var request archiveBatchRequest - if err := decode(r, &request); err != nil { + if err := decode(w, r, &request); err != nil { writeErr(w, 400, err.Error()) return } diff --git a/server/task_assets.go b/server/task_assets.go index bdca4a5..bc66c97 100644 --- a/server/task_assets.go +++ b/server/task_assets.go @@ -33,7 +33,7 @@ func (s *Server) attachTaskAssets(w http.ResponseWriter, r *http.Request) { SourceSummary string `json:"source_summary"` Scope companyScopeInputs `json:"scope"` } - if err := decode(r, &request); err != nil { + if err := decode(w, r, &request); err != nil { var tooLarge *http.MaxBytesError if errors.As(err, &tooLarge) { writeErr(w, http.StatusRequestEntityTooLarge, "request body too large") diff --git a/server/task_categories.go b/server/task_categories.go index fee8e45..03d2ad9 100644 --- a/server/task_categories.go +++ b/server/task_categories.go @@ -21,7 +21,7 @@ type taskCategoryRequest struct { func decodeTaskCategoryRequest(w http.ResponseWriter, r *http.Request) (*taskCategoryRequest, bool) { r.Body = http.MaxBytesReader(w, r.Body, maxTaskCategoryRequestBytes) var request taskCategoryRequest - if err := decode(r, &request); err != nil { + if err := decode(w, r, &request); err != nil { var tooLarge *http.MaxBytesError if errors.As(err, &tooLarge) { writeErr(w, http.StatusRequestEntityTooLarge, "request body too large") @@ -150,7 +150,7 @@ func (s *Server) updateTaskCategory(w http.ResponseWriter, r *http.Request) { var request struct { CategoryID json.RawMessage `json:"category_id"` } - if err := decode(r, &request); err != nil { + if err := decode(w, r, &request); err != nil { writeErr(w, http.StatusBadRequest, err.Error()) return } @@ -184,7 +184,7 @@ func (s *Server) updateTasksCategoryBatch(w http.ResponseWriter, r *http.Request TaskIDs []string `json:"task_ids"` CategoryID json.RawMessage `json:"category_id"` } - if err := decode(r, &request); err != nil { + if err := decode(w, r, &request); err != nil { var tooLarge *http.MaxBytesError if errors.As(err, &tooLarge) { writeErr(w, http.StatusRequestEntityTooLarge, "request body too large") diff --git a/server/task_intercept.go b/server/task_intercept.go index c6eeefb..4635b1b 100644 --- a/server/task_intercept.go +++ b/server/task_intercept.go @@ -91,7 +91,7 @@ func (s *Server) taskInterceptCreateRule(w http.ResponseWriter, r *http.Request) return } var req taskInterceptRuleReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -123,7 +123,7 @@ func (s *Server) taskInterceptUpdateRule(w http.ResponseWriter, r *http.Request) return } var req taskInterceptRuleReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -180,7 +180,7 @@ func (s *Server) taskInterceptToggleRule(w http.ResponseWriter, r *http.Request) var req struct { Enabled bool `json:"enabled"` } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } diff --git a/server/task_metadata.go b/server/task_metadata.go index 7bcd738..dde601a 100644 --- a/server/task_metadata.go +++ b/server/task_metadata.go @@ -30,7 +30,7 @@ func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) { PlanQuietMode *bool `json:"plan_quiet_mode"` TargetGate *bool `json:"target_gate"` } - if err := decode(r, &request); err != nil { + if err := decode(w, r, &request); err != nil { var tooLarge *http.MaxBytesError if errors.As(err, &tooLarge) { writeErr(w, http.StatusRequestEntityTooLarge, "request body too large") diff --git a/server/traffic_get_description_test.go b/server/traffic_get_description_test.go new file mode 100644 index 0000000..f91ae05 --- /dev/null +++ b/server/traffic_get_description_test.go @@ -0,0 +1,44 @@ +package server + +import ( + "encoding/json" + "testing" + + "github.com/Autumn-27/artex/traffic" +) + +// The traffic_get description that says traffic_search never finds a value a +// rewrite rule wrote replaces the original built-in text once, and never a +// description the user edited. +func TestTrafficGetDescriptionUpgrade(t *testing.T) { + s, _, _ := trafficEvidenceServer(t) + pg := s.m.pg + old, err := pg.GetTool("traffic_get") + if err != nil || old == nil { + t.Fatal("missing tool traffic_get", err) + } + bindings, _ := json.Marshal(old.Agents) + t.Cleanup(func() { pg.UpdateTool("traffic_get", old.Description, old.Schema, bindings, old.Enabled) }) + const flag = "traffic_get_rewrite_description_v1" + legacy := "Fetch the raw request/response of one captured traffic record by id (truncated when too large). Use together with traffic_search to avoid re-curling. A WebSocket (status 101) or SSE (text/event-stream) record shows only its handshake here; read its messages with traffic_messages." + + for _, c := range []struct{ stored, want string }{ + {legacy, traffic.TrafficGetDescription}, + {"USER DESCRIPTION", "USER DESCRIPTION"}, + } { + if err := pg.UpdateTool("traffic_get", c.stored, old.Schema, bindings, old.Enabled); err != nil { + t.Fatal(err) + } + if err := pg.SetSetting(flag, "false"); err != nil { + t.Fatal(err) + } + s.seedFindingWorkflowTools() + row, _ := pg.GetTool("traffic_get") + if row.Description != c.want { + t.Fatalf("stored %q: description %q, want %q", c.stored, row.Description, c.want) + } + if v, _, _ := pg.GetSetting(flag); v != "true" { + t.Fatal("flag not set") + } + } +} diff --git a/server/traffic_rewrite.go b/server/traffic_rewrite.go index b937794..8038fc3 100644 --- a/server/traffic_rewrite.go +++ b/server/traffic_rewrite.go @@ -216,7 +216,7 @@ func (s *Server) rewriteCreateRule(w http.ResponseWriter, r *http.Request) { return } var req rewriteRuleReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -249,7 +249,7 @@ func (s *Server) rewriteUpdateRule(w http.ResponseWriter, r *http.Request) { return } var req rewriteRuleReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -308,7 +308,7 @@ func (s *Server) rewriteToggleRule(w http.ResponseWriter, r *http.Request) { var req struct { Enabled bool `json:"enabled"` } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } diff --git a/server/triggers.go b/server/triggers.go index 1064066..0b60837 100644 --- a/server/triggers.go +++ b/server/triggers.go @@ -60,7 +60,7 @@ func (s *Server) pgCreateTrigger(w http.ResponseWriter, r *http.Request) { return } var req triggerReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -96,7 +96,7 @@ func (s *Server) pgUpdateTrigger(w http.ResponseWriter, r *http.Request) { return } var req triggerReq - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } diff --git a/server/workspace.go b/server/workspace.go index df78e3e..0307343 100644 --- a/server/workspace.go +++ b/server/workspace.go @@ -139,7 +139,7 @@ func (s *Server) wsWrite(w http.ResponseWriter, r *http.Request) { Path string `json:"path"` Content string `json:"content"` } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } @@ -168,7 +168,7 @@ func (s *Server) wsMkdir(w http.ResponseWriter, r *http.Request) { var req struct { Path string `json:"path"` } - if err := decode(r, &req); err != nil { + if err := decode(w, r, &req); err != nil { writeErr(w, 400, err.Error()) return } diff --git a/traffic/external.go b/traffic/external.go index 41c9dba..9e849dc 100644 --- a/traffic/external.go +++ b/traffic/external.go @@ -76,6 +76,11 @@ func (t *Traffic) insertExchange(id string, ts int64, host, method, urlTemplate, } defer tx.Rollback() //nolint:errcheck // no-op once committed + // An id can repeat (seconds plus a rolling counter); the trace of the + // exchange it replaces must not stay attached to this one. + if _, err := tx.Exec(`DELETE FROM exchange_rewrites WHERE exchange_id=?`, id); err != nil { + return fmt.Errorf("clearing rewrite trace: %w", err) + } res, err := tx.Exec(`INSERT OR REPLACE INTO exchanges(id,ts,host,method,url_template,url,status,content_type,req_len,resp_len,path) VALUES(?,?,?,?,?,?,?,?,?,?,'')`, id, ts, host, method, urlTemplate, rawURL, status, contentType, reqLen, respLen) diff --git a/traffic/pace.go b/traffic/pace.go index 756c330..438a839 100644 --- a/traffic/pace.go +++ b/traffic/pace.go @@ -40,11 +40,15 @@ func (s *sink) Requestheaders(f *mproxy.Flow) { } } -// ClientDisconnected forgets a closed connection's mark. +// ClientDisconnected forgets a closed connection's mark, and asks for a sweep +// of the request snapshots its flows may have left (rewrite_flow.go). func (s *sink) ClientDisconnected(c *mproxy.ClientConn) { if c != nil { s.paced.Delete(c.Id) } + if s.t != nil { + s.t.snaps.requestSweep() + } } // pace waits for the request's host under the current noise profile, bounded diff --git a/traffic/rewrite/compile.go b/traffic/rewrite/compile.go index ed663f9..87079de 100644 --- a/traffic/rewrite/compile.go +++ b/traffic/rewrite/compile.go @@ -80,6 +80,14 @@ func (s *Set) LoadErrors() map[int64]string { return s.loadErrors } +// NoteError records a run-time error of a rule seen outside the engine, such +// as the transport failure of a request the rule rewrote. +func (s *Set) NoteError(id int64, class string) { + if s != nil { + s.errors.record(id, class) + } +} + // LastError is the most recent run-time error of a rule. It lives in memory // only; the per-exchange trace keeps the history. type LastError struct { diff --git a/traffic/rewrite/trace.go b/traffic/rewrite/trace.go index 506a85e..5bf494d 100644 --- a/traffic/rewrite/trace.go +++ b/traffic/rewrite/trace.go @@ -30,6 +30,9 @@ const ( ErrClassInvalidURL = "invalid_url" // a rewritten path and query is not a valid request URI ErrClassExpansion = "expansion_limit" // a replace result exceeds MaxExpansion ErrClassPanic = "panic" + // ErrClassTransport is recorded by the proxy, not the engine: the request a + // rule rewrote could not be sent or answered. + ErrClassTransport = "transport_error" ) // TraceLine is one row of an exchange's rewrite trace, in order. An exchange is diff --git a/traffic/rewrite_flow.go b/traffic/rewrite_flow.go new file mode 100644 index 0000000..b740093 --- /dev/null +++ b/traffic/rewrite_flow.go @@ -0,0 +1,663 @@ +package traffic + +import ( + "database/sql" + "io" + "log" + "net/http" + "net/url" + "sync" + "sync/atomic" + "time" + + "github.com/Autumn-27/artex/traffic/rewrite" + mproxy "github.com/lqqyt2423/go-mitmproxy/proxy" +) + +// Applying the rewrite rules to the proxied flows. +// +// The request phase runs in the Request hook. A request a rule rewrote leaves +// the proxy changed, but its exchange records what the client sent: the +// original is kept in a per-flow request snapshot until the Response hook +// records the exchange. The response phase runs at the start of the Response +// hook (status and headers only at SSEStart), and the exchange records the +// response as the target sent it. The trace of both phases is written in the +// exchange's own transaction (exchange_rewrites). +// +// A request is rewritten only once its snapshot slot is reserved, and a slot +// stays until the flow's exchange is recorded or the flow is finished, so the +// Response hook never takes a rewritten request for the client's. + +// Limits of the request snapshots. +const ( + maxSnapEntries = 1024 // entries holding an original request + maxSnapBytes = 64 << 20 // what those entries hold + maxSnapTombs = 4096 // tombstones: slots whose data was released + // snapTTL deletes nothing: past it, the entry of a live flow may be turned + // into a tombstone when a new reservation finds no room. + snapTTL = 15 * time.Minute + sweepDebounce = time.Second // at most one sweep per second + sweepPeriod = time.Minute // a sweep at least this often while slots exist + // libraryStreamBody is go-mitmproxy's StreamLargeBodies default: a body this + // large reaches neither the Request nor the Response hook. + libraryStreamBody = 5 << 20 + + reasonSnapshotCapacity = "not rewritten: snapshot capacity" + reasonOriginalUnavailable = "original request unavailable" +) + +// testRuleHook, when a test sets it, runs inside each rule that checks a body +// ("rule") and in the sink right after the engine ("request", "response"), so +// a test can make a rule, or the sink itself, panic. +var testRuleHook atomic.Pointer[func(stage string)] + +func runTestHook(stage string) { + if h := testRuleHook.Load(); h != nil { + (*h)(stage) + } +} + +// rulesIsBinary is the binary predicate handed to the engine: the recorder's +// own, so "not rewritten: binary" follows what the recorder treats as binary. +func rulesIsBinary(contentType string, body []byte) bool { + runTestHook("rule") + return isBinaryBody(contentType, body) +} + +type flowKey [16]byte + +func keyOf(f *mproxy.Flow) flowKey { return flowKey(f.Id) } + +// reqSnap is one slot of the table: an entry, which holds the original of a +// request, or a tombstone, which holds only why it has none. +type reqSnap struct { + // done is the flow's Done() channel: closed, no hook will come for the + // slot any more. The flow itself is not referenced. + done <-chan struct{} + rules *rewrite.Set // the snapshot of rules the flow runs with, both phases + host string // host name: the only part of the request a log shows + tomb bool + + // Entry. + created time.Time + cost int64 + rewritten bool // the request phase changed the request + method string + url *url.URL + header http.Header + rawHost string + body []byte // the client's body, kept when a body action applied + keepBody bool + lines []rewrite.TraceLine // request-phase trace + + // Tombstone. + reason string // reasonSnapshotCapacity or reasonOriginalUnavailable + ruleIDs []int64 // rules that had rewritten the request +} + +// requestRewritten reports a flow whose request left the proxy changed. +func (s *reqSnap) requestRewritten() bool { + if s.tomb { + return s.reason == reasonOriginalUnavailable + } + return s.rewritten +} + +func (s *reqSnap) appliedRuleIDs() []int64 { + if s.tomb { + return s.ruleIDs + } + _, ids := appliedRules(s.lines) + return ids +} + +// snapTable holds the slots, keyed by flow id. Its mutex is a leaf lock. +type snapTable struct { + mu sync.Mutex + slots map[flowKey]*reqSnap + entries int + bytes int64 + tombs int + lastSweep time.Time + lastFullLog time.Time + sweepWanted atomic.Bool + + now func() time.Time + maxEntries int + maxBytes int64 + maxTombs int + ttl time.Duration + + sweeps, capacity, tombDrops, conversions, largeResponses atomic.Int64 +} + +func newSnapTable() *snapTable { + return &snapTable{slots: map[flowKey]*reqSnap{}, now: time.Now, + maxEntries: maxSnapEntries, maxBytes: maxSnapBytes, maxTombs: maxSnapTombs, ttl: snapTTL} +} + +func doneClosed(ch <-chan struct{}) bool { + if ch == nil { + return false // a nil channel is never ready: the flow counts as live + } + select { + case <-ch: + return true + default: + return false + } +} + +// reserve stores the entry of a rewritten request. On failure the request must +// leave unchanged: a capacity tombstone then stands for the flow, when there is +// room for one. +func (t *snapTable) reserve(k flowKey, e *reqSnap) bool { + if t == nil { + return false + } + t.mu.Lock() + defer t.mu.Unlock() + t.removeLocked(k) + if !t.fitsLocked(e.cost) { + // A failed reservation asks for an immediate sweep, under the same + // debounce; a sweep refused by the debounce fails the reservation. + now := t.now() + if now.Sub(t.lastSweep) < sweepDebounce { + t.refuseLocked(k, e) + return false + } + t.sweepLocked(now) + if !t.fitsLocked(e.cost) && !t.convertLocked(e.cost) { + t.refuseLocked(k, e) + return false + } + } + e.created = t.now() + t.slots[k] = e + t.entries++ + t.bytes += e.cost + return true +} + +func (t *snapTable) fitsLocked(cost int64) bool { + return t.entries < t.maxEntries && t.bytes+cost <= t.maxBytes +} + +// convertLocked turns the oldest entry of a live flow older than the TTL into +// a tombstone, in place, so the flow is never without a slot. It does so only +// when releasing that entry lets a reservation of cost succeed, and when a +// tombstone place is free (after evicting tombstones of finished flows only). +// Otherwise nothing changes and it reports false. +func (t *snapTable) convertLocked(cost int64) bool { + now := t.now() + var victim *reqSnap + for _, s := range t.slots { + if s.tomb || doneClosed(s.done) || now.Sub(s.created) < t.ttl { + continue + } + if t.entries-1 >= t.maxEntries || t.bytes-s.cost+cost > t.maxBytes { + continue // released, it would still leave no room + } + if victim == nil || s.created.Before(victim.created) { + victim = s + } + } + if victim == nil { + return false + } + if t.tombs >= t.maxTombs { + t.evictFinishedTombsLocked() + } + if t.tombs >= t.maxTombs { + return false + } + ids := victim.appliedRuleIDs() + t.entries-- + t.bytes -= victim.cost + t.tombs++ + *victim = reqSnap{done: victim.done, rules: victim.rules, host: victim.host, tomb: true, reason: reasonOriginalUnavailable, ruleIDs: ids} + t.conversions.Add(1) + log.Printf("[traffic] rewrite snapshot of a request to %s (rules %v) released under pressure: its exchange will be recorded as original request unavailable", victim.host, ids) + return true +} + +// refuseLocked is the capacity path: the request is not rewritten, and a +// tombstone records why, when a place is free. +func (t *snapTable) refuseLocked(k flowKey, e *reqSnap) { + t.capacity.Add(1) + if t.tombs >= t.maxTombs { + t.evictFinishedTombsLocked() + } + full := t.tombs >= t.maxTombs + if full { + t.tombDrops.Add(1) + } else { + t.slots[k] = &reqSnap{done: e.done, rules: e.rules, host: e.host, tomb: true, reason: reasonSnapshotCapacity} + t.tombs++ + } + if now := t.now(); now.Sub(t.lastFullLog) >= time.Minute { + t.lastFullLog = now + log.Printf("[traffic] rewrite snapshot table full: requests are sent without rewriting (%d refused, %d without a trace so far)", t.capacity.Load(), t.tombDrops.Load()) + } +} + +func (t *snapTable) evictFinishedTombsLocked() { + for k, s := range t.slots { + if s.tomb && doneClosed(s.done) { + t.removeLocked(k) + } + } +} + +// sweepLocked deletes every slot whose flow is finished. The slot of a live +// flow is never touched, whatever its age. +func (t *snapTable) sweepLocked(now time.Time) { + for k, s := range t.slots { + if doneClosed(s.done) { + t.removeLocked(k) + } + } + t.lastSweep = now + t.sweepWanted.Store(false) + t.sweeps.Add(1) +} + +func (t *snapTable) removeLocked(k flowKey) { + s := t.slots[k] + if s == nil { + return + } + if s.tomb { + t.tombs-- + } else { + t.entries-- + t.bytes -= s.cost + } + delete(t.slots, k) +} + +// take removes and returns the slot of a flow, nil when it has none. +func (t *snapTable) take(k flowKey) *reqSnap { + if t == nil { + return nil + } + t.mu.Lock() + defer t.mu.Unlock() + s := t.slots[k] + t.removeLocked(k) + return s +} + +// requestSweep asks the sweeper for a pass; it runs at most once a second. +func (t *snapTable) requestSweep() { + if t != nil { + t.sweepWanted.Store(true) + } +} + +// tick is one beat of the sweeper: a sweep when one was asked for and the last +// is a second old, or when the last is a minute old. Nothing on an empty table. +func (t *snapTable) tick() { + t.mu.Lock() + defer t.mu.Unlock() + if len(t.slots) == 0 { + t.sweepWanted.Store(false) + return + } + now := t.now() + since := now.Sub(t.lastSweep) + if (t.sweepWanted.Load() && since >= sweepDebounce) || since >= sweepPeriod { + t.sweepLocked(now) + } +} + +// counts reports the entries and tombstones in the table. +func (t *snapTable) counts() (entries, tombs int) { + t.mu.Lock() + defer t.mu.Unlock() + return t.entries, t.tombs +} + +// sweepSnapshots is the single sweeper goroutine, started by Open and stopped +// by Close. +func (t *Traffic) sweepSnapshots() { + tick := time.NewTicker(time.Second) + defer tick.Stop() + for { + select { + case <-t.closed: + return + case <-tick.C: + t.snaps.tick() + } + } +} + +// RewriteStats are the proxy-side counters of the rewrite rules since start. +type RewriteStats struct { + // LargeResponses counts flows with a snapshot whose response reached the + // library's 5 MiB stream threshold: passed through, not recorded. + LargeResponses int64 `json:"large_responses"` + // CapacityRefusals counts requests sent without rewriting for lack of room. + CapacityRefusals int64 `json:"capacity_refusals"` + // TombstoneDrops counts those of them that got no capacity tombstone either. + TombstoneDrops int64 `json:"tombstone_drops"` + // Conversions counts entries released under pressure. + Conversions int64 `json:"conversions"` + Sweeps int64 `json:"sweeps"` +} + +func (t *Traffic) RewriteStats() RewriteStats { + s := t.snaps + if s == nil { + return RewriteStats{} + } + return RewriteStats{LargeResponses: s.largeResponses.Load(), CapacityRefusals: s.capacity.Load(), + TombstoneDrops: s.tombDrops.Load(), Conversions: s.conversions.Load(), Sweeps: s.sweeps.Load()} +} + +// newEntry builds the snapshot of a request before the rules' result is put on +// the flow: URL and headers cloned, the body referenced, and only when a body +// action applied (otherwise the flow's body is still the client's). +func newEntry(f *mproxy.Flow, set *rewrite.Set, res rewrite.Result) *reqSnap { + r := f.Request + u := *r.URL + e := &reqSnap{done: f.Done(), rules: set, host: r.URL.Hostname(), rewritten: res.Changed, + method: r.Method, url: &u, header: r.Header.Clone(), lines: res.Lines} + if raw := r.Raw(); raw != nil { + e.rawHost = raw.Host + } + for _, l := range res.Lines { + if l.Outcome == rewrite.OutcomeApplied && l.Field == "body" && len(r.Body) <= rewrite.MaxBodySize { + e.body, e.keepBody = r.Body, true + break + } + } + e.cost = int64(len(e.method) + len(u.String()) + len(e.rawHost) + len(e.body)) + for k, vs := range e.header { + for _, v := range vs { + e.cost += int64(len(k) + len(v)) + } + } + for _, l := range e.lines { + e.cost += int64(len(l.After) + len(l.RuleSnapshot)) + } + return e +} + +// originalRequest is the request an exchange records, with the request-phase +// trace: the flow's own request when no rule rewrote it, the entry's copy when +// one did, and only what Raw() still knows when the entry was released. +func originalRequest(f *mproxy.Flow, snap *reqSnap) (requestView, []rewrite.TraceLine) { + switch { + case snap == nil: + return flowRequestView(f.Request), nil + case !snap.tomb: + v := requestView{method: snap.method, url: snap.url, proto: f.Request.Proto, header: snap.header, rawHost: snap.rawHost, body: f.Request.Body} + if snap.keepBody { + v.body = snap.body + } + return v, snap.lines + case snap.reason == reasonSnapshotCapacity: + // The request was not rewritten: the flow's request is the client's. + return flowRequestView(f.Request), []rewrite.TraceLine{{Phase: rewrite.PhaseRequest, Outcome: rewrite.OutcomeSkipped, Reason: reasonSnapshotCapacity}} + default: + // Rewritten, original released: never record the rewritten request as + // the client's. Raw() is the client's request line and Host. + v := requestView{unavailable: true} + if raw := f.Request.Raw(); raw != nil { + v.method, v.url, v.proto, v.rawHost = raw.Method, raw.URL, raw.Proto, raw.Host + } + lines := []rewrite.TraceLine{{Phase: rewrite.PhaseRequest, Outcome: rewrite.OutcomeError, Reason: reasonOriginalUnavailable}} + for _, id := range snap.ruleIDs { + lines = append(lines, rewrite.TraceLine{Phase: rewrite.PhaseRequest, RuleID: id, Outcome: rewrite.OutcomeApplied}) + } + return v, lines + } +} + +func engineRequest(r *mproxy.Request) rewrite.Request { + return rewrite.Request{Method: r.Method, URL: r.URL, Header: r.Header, Body: r.Body} +} + +// Request runs the request phase. The result goes on the flow only once the +// request's snapshot slot is reserved; without one the request leaves as the +// client sent it. Rules fail open: a panic leaves the request unchanged. +func (s *sink) Request(f *mproxy.Flow) { + if s.t == nil || f.Request == nil || f.Request.URL == nil { + return + } + set := s.t.rewriteRules.Load() + if set.Len(rewrite.PhaseRequest) == 0 { + return + } + host := f.Request.URL.Hostname() + defer func() { + if recover() != nil { + log.Printf("[traffic] rewrite rules stopped on a request to %s (class %s): it is sent as the client sent it", host, rewrite.ErrClassPanic) + } + }() + res := set.Apply(rewrite.PhaseRequest, rewrite.Input{Request: engineRequest(f.Request), IsBinary: rulesIsBinary}) + runTestHook("request") + if len(res.Lines) == 0 { + return + } + if !s.t.snaps.reserve(keyOf(f), newEntry(f, set, res)) { + return + } + if res.Changed { + r := f.Request + r.Method, r.URL, r.Header, r.Body = res.Request.Method, res.Request.URL, res.Request.Header, res.Request.Body + } +} + +// Response records the exchange and runs the response phase: the exchange +// keeps the client's request and the target's response, the client gets the +// rewritten response. Without a slot and without response rules this is the +// plain record. +func (s *sink) Response(f *mproxy.Flow) { + if f.Request == nil || f.Response == nil { + return + } + snap := s.t.snaps.take(keyOf(f)) + set := s.t.rewriteRules.Load() + if snap != nil { + set = snap.rules + } + if snap == nil && set.Len(rewrite.PhaseResponse) == 0 { + s.t.record(f) + return + } + rq, lines := originalRequest(f, snap) + rs := flowResponseView(f.Response) + res := s.applyResponse(f, set, rewrite.Input{ + Request: engineRequest(f.Request), // as sent to the target + Response: rewrite.Response{Status: rs.status, Header: rs.header, Body: rs.body}, + IsBinary: rulesIsBinary, + }) + s.t.recordViews(rq, rs, append(lines, res.Lines...)) + if res.Changed { + f.Response.StatusCode, f.Response.Header, f.Response.Body = res.Response.Status, res.Response.Header, res.Response.Body + } +} + +// applyResponse runs the response phase; a panic yields no change and no line. +func (s *sink) applyResponse(f *mproxy.Flow, set *rewrite.Set, in rewrite.Input) (res rewrite.Result) { + if set.Len(rewrite.PhaseResponse) == 0 { + return rewrite.Result{} + } + defer func() { + if recover() != nil { + res = rewrite.Result{} + log.Printf("[traffic] rewrite rules stopped on a response from %s (class %s): it is sent as the target sent it", f.Request.URL.Hostname(), rewrite.ErrClassPanic) + } + }() + r := set.Apply(rewrite.PhaseResponse, in) + runTestHook("response") + return r +} + +// rewriteStream prepares the exchange of a stream at SSEStart or +// WebSocketStart: the client's request, the target's status and headers (SSE), +// and the trace. For SSE, the response phase runs in stream mode: status and +// header actions apply (res, put on the flow by applyStreamResponse), rules +// that read or write the response body are traced "not rewritten: stream". +// WebSocketStart runs once go-mitmproxy has already answered the client's +// upgrade, with no response on the flow: nothing of the handshake can change +// there, so every candidate rule is traced "not rewritten: stream". +func (s *sink) rewriteStream(f *mproxy.Flow, kind string) (rq requestView, rs responseView, lines []rewrite.TraceLine, res rewrite.Result) { + snap := s.t.snaps.take(keyOf(f)) + set := s.t.rewriteRules.Load() + if snap != nil { + set = snap.rules + } + rq, lines = originalRequest(f, snap) + if kind == "sse" { + rs = flowResponseView(f.Response) + } + cur := engineRequest(f.Request) + if kind != "websocket" { + res = s.applyResponse(f, set, rewrite.Input{Request: cur, Response: rewrite.Response{Status: rs.status, Header: rs.header}, Stream: true, IsBinary: rulesIsBinary}) + return rq, rs, append(lines, res.Lines...), res + } + if set.Len(rewrite.PhaseRequest)+set.Len(rewrite.PhaseResponse) == 0 { + return rq, rs, lines, rewrite.Result{} + } + // The engine never changes a request with Upgrade; a ws:// stream arrives + // on its CONNECT flow, which has none. + if cur.Header.Get("Upgrade") == "" { + cur.Header = cur.Header.Clone() + if cur.Header == nil { + cur.Header = http.Header{} + } + cur.Header.Set("Upgrade", "websocket") + } + if snap == nil { + lines = append(lines, s.applyRequestTrace(f, set, cur)...) + } + r := s.applyResponse(f, set, rewrite.Input{Request: cur, Response: rewrite.Response{Status: http.StatusSwitchingProtocols}, Stream: true, IsBinary: rulesIsBinary}) + return rq, rs, append(lines, r.Lines...), rewrite.Result{} +} + +// applyRequestTrace runs the request phase for its trace only. +func (s *sink) applyRequestTrace(f *mproxy.Flow, set *rewrite.Set, req rewrite.Request) (lines []rewrite.TraceLine) { + defer func() { + if recover() != nil { + lines = nil + log.Printf("[traffic] rewrite rules stopped on a request to %s (class %s)", f.Request.URL.Hostname(), rewrite.ErrClassPanic) + } + }() + return set.Apply(rewrite.PhaseRequest, rewrite.Input{Request: req, IsBinary: rulesIsBinary}).Lines +} + +// applyStreamResponse puts the rewritten status and headers of an SSE +// response on the flow, before go-mitmproxy replies to the client. +func (s *sink) applyStreamResponse(f *mproxy.Flow, res rewrite.Result) { + if res.Changed && f.Response != nil { + f.Response.StatusCode, f.Response.Header = res.Response.Status, res.Response.Header + } +} + +// RequestError fires when a request through an established MITM tunnel fails. If +// the failure looks proxy/protocol-caused (h2 quirks, HEAD-with-body, protocol +// errors) and not a plain target-unreachable error, the host is tunneled for +// passthroughTTL: this request has already failed, later requests in the window +// connect straight through, and the first one after the window tries +// interception again. A request a rule rewrote never flags its host: the rule, +// not the interception, may be at fault. Its log names the host, the error class +// and the rules, never the URL or the error text. +func (s *sink) RequestError(f *mproxy.Flow, err error) { + if f != nil { + if snap := s.t.snaps.take(keyOf(f)); snap != nil && snap.requestRewritten() { + ids := snap.appliedRuleIDs() + for _, id := range ids { + snap.rules.NoteError(id, rewrite.ErrClassTransport) + } + log.Printf("[traffic] a request to %s rewritten by rules %v failed (class %s); the host stays intercepted", snap.host, ids, rewrite.ErrClassTransport) + return + } + } + s.t.maybePassthrough(f, err) +} + +// StreamResponseModifier passes every body through as received, nil included, +// without wrapping it. A response that reached the library's stream threshold +// is never recorded: the slot of its flow, if any, is released here. +func (s *sink) StreamResponseModifier(f *mproxy.Flow, in io.Reader) (out io.Reader) { + out = in + defer func() { + if recover() != nil { + out = in + } + }() + if s == nil || s.t == nil || f == nil || !f.Stream || f.SSE != nil { + return in + } + snap := s.t.snaps.take(keyOf(f)) + if snap == nil { + return in + } + s.t.snaps.largeResponses.Add(1) + log.Printf("[traffic] the response to a request to %s with rewrite rules %v reached %d MiB: passed through, not recorded", snap.host, snap.appliedRuleIDs(), libraryStreamBody>>20) + return in +} + +// spillTrace stores the values of the trace lines as record stores bodies. +// Called under wmu, before the transaction. +func (t *Traffic) spillTrace(lines []rewrite.TraceLine) []storedBody { + out := make([]storedBody, len(lines)) + for i, l := range lines { + out[i] = t.spill(l.After, "") + } + return out +} + +// writeTrace inserts an exchange's trace rows and the blob references of their +// spilled values, inside the exchange's transaction. +func writeTrace(tx *sql.Tx, id string, lines []rewrite.TraceLine, trace []storedBody) error { + for _, b := range trace { + if b.hash == "" { + continue + } + if _, err := tx.Exec(`INSERT OR IGNORE INTO blob_refs(hash,exchange_id) VALUES(?,?)`, b.hash, id); err != nil { + return &recordError{"registering blob reference", err} + } + } + for i, l := range lines { + var ruleID, after any + if l.RuleID != 0 { + ruleID = l.RuleID + } + if trace[i].inline != nil { + after = trace[i].inline + } + if _, err := tx.Exec(`INSERT INTO exchange_rewrites(exchange_id,seq,phase,rule_id,rule_snapshot,action,field,outcome,reason,before_hash,after_hash,after_inline,after_blob) +VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?)`, + id, i, string(l.Phase), ruleID, nullIfEmpty(l.RuleSnapshot), nullIfEmpty(l.Action), nullIfEmpty(l.Field), l.Outcome, l.Reason, + nullIfEmpty(l.BeforeHash), nullIfEmpty(l.AfterHash), after, nullIfEmpty(trace[i].hash)); err != nil { + return &recordError{"writing rewrite trace", err} + } + } + return nil +} + +// RewriteInfo reports whether rules rewrote an exchange (at least one applied +// trace line) and which rules did, by id. +func (t *Traffic) RewriteInfo(id string) (rewritten bool, ruleIDs []int64, err error) { + if err := t.db.QueryRow(`SELECT EXISTS(SELECT 1 FROM exchange_rewrites r WHERE r.exchange_id=? AND r.outcome='applied')`, id).Scan(&rewritten); err != nil || !rewritten { + return false, nil, err + } + rows, err := t.db.Query(`SELECT DISTINCT rule_id FROM exchange_rewrites WHERE exchange_id=? AND outcome='applied' AND rule_id IS NOT NULL ORDER BY rule_id`, id) + if err != nil { + return false, nil, err + } + defer rows.Close() + for rows.Next() { + var rid int64 + if err := rows.Scan(&rid); err != nil { + return false, nil, err + } + ruleIDs = append(ruleIDs, rid) + } + return true, ruleIDs, rows.Err() +} diff --git a/traffic/rewrite_flow_test.go b/traffic/rewrite_flow_test.go new file mode 100644 index 0000000..a549d34 --- /dev/null +++ b/traffic/rewrite_flow_test.go @@ -0,0 +1,1316 @@ +package traffic + +import ( + "bytes" + "context" + "crypto/hmac" + "crypto/sha256" + "database/sql" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "log" + "net" + "net/http" + "net/http/httptest" + "net/url" + "os" + "slices" + "sort" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/Autumn-27/artex/traffic/rewrite" + "github.com/gorilla/websocket" + mproxy "github.com/lqqyt2423/go-mitmproxy/proxy" +) + +// --- helpers --------------------------------------------------------------- + +// rules compiles enabled rules, failing the test on a load error. +func rules(t *testing.T, key []byte, errs *rewrite.ErrorLog, rs ...rewrite.Rule) *rewrite.Set { + t.Helper() + for i := range rs { + rs[i].Enabled = true + if rs[i].Name == "" { + rs[i].Name = fmt.Sprintf("rule %d", rs[i].ID) + } + } + set := rewrite.Compile(rs, rewrite.Options{TraceKey: key, Errors: errs}) + if len(set.LoadErrors()) != 0 { + t.Fatalf("load errors: %v", set.LoadErrors()) + } + return set +} + +type traceRow struct { + seq int + phase, outcome, reason string + ruleID sql.NullInt64 + snapshot, action, field sql.NullString + beforeHash, afterHash, blob sql.NullString + inline []byte +} + +func traceRows(t *testing.T, tr *Traffic, id string) []traceRow { + t.Helper() + rows, err := tr.DB().Query(`SELECT seq,phase,rule_id,rule_snapshot,action,field,outcome,reason,before_hash,after_hash,after_inline,after_blob +FROM exchange_rewrites WHERE exchange_id=? ORDER BY seq`, id) + if err != nil { + t.Fatal(err) + } + defer rows.Close() + var out []traceRow + for rows.Next() { + var r traceRow + if err := rows.Scan(&r.seq, &r.phase, &r.ruleID, &r.snapshot, &r.action, &r.field, &r.outcome, &r.reason, &r.beforeHash, &r.afterHash, &r.inline, &r.blob); err != nil { + t.Fatal(err) + } + out = append(out, r) + } + return out +} + +func countRows(t *testing.T, tr *Traffic, table string) int { + t.Helper() + var n int + if err := tr.DB().QueryRow(`SELECT COUNT(*) FROM ` + table).Scan(&n); err != nil { + t.Fatal(err) + } + return n +} + +func exchangeIDByPath(t *testing.T, tr *Traffic, suffix string) string { + t.Helper() + var id string + if err := tr.DB().QueryRow(`SELECT id FROM exchanges WHERE url LIKE ?`, "%"+suffix).Scan(&id); err != nil { + t.Fatalf("exchange %s: %v", suffix, err) + } + return id +} + +// logBuf captures the standard logger, where every ARTEX log goes. +type logBuf struct { + mu sync.Mutex + b bytes.Buffer +} + +func (l *logBuf) Write(p []byte) (int, error) { + l.mu.Lock() + defer l.mu.Unlock() + return l.b.Write(p) +} + +func (l *logBuf) String() string { + l.mu.Lock() + defer l.mu.Unlock() + return l.b.String() +} + +func captureLog(t *testing.T) *logBuf { + b := &logBuf{} + prev := log.Writer() + log.SetOutput(b) + t.Cleanup(func() { log.SetOutput(prev) }) + return b +} + +func setTestHook(t *testing.T, h func(stage string)) { + testRuleHook.Store(&h) + t.Cleanup(func() { testRuleHook.Store(nil) }) +} + +// limits changes the table's bounds and clock for a test. +func (t *snapTable) limits(maxEntries int, maxBytes int64, maxTombs int, now func() time.Time) { + t.mu.Lock() + defer t.mu.Unlock() + t.maxEntries, t.maxBytes, t.maxTombs = maxEntries, maxBytes, maxTombs + if now != nil { + t.now = now + } +} + +func (t *snapTable) slot(k flowKey) *reqSnap { + t.mu.Lock() + defer t.mu.Unlock() + if s := t.slots[k]; s != nil { + c := *s + return &c + } + return nil +} + +func key(n int) flowKey { + var k flowKey + k[0], k[1] = byte(n), byte(n>>8) + return k +} + +// withID gives a synthetic flow a distinct id, as real flows have. +func withID(f *mproxy.Flow, n int) *mproxy.Flow { + f.Id[0], f.Id[1], f.Id[15] = byte(n), byte(n>>8), 0xAB + return f +} + +type received struct { + mu sync.Mutex + reqs []string +} + +func (r *received) add(s string) { r.mu.Lock(); r.reqs = append(r.reqs, s); r.mu.Unlock() } +func (r *received) all() []string { + r.mu.Lock() + defer r.mu.Unlock() + return slices.Clone(r.reqs) +} + +func sendThrough(t *testing.T, tr *Traffic, proxyURL, method, target, body string, header http.Header) (*http.Response, []byte) { + t.Helper() + req, err := http.NewRequest(method, target, strings.NewReader(body)) + if err != nil { + t.Fatal(err) + } + for k, vs := range header { + req.Header[k] = vs + } + resp, err := proxyClient(t, tr, proxyURL).Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + b, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatal(err) + } + return resp, b +} + +// --- integration: rewritten exchange ----------------------------------------- + +// A request and a response rewritten by rules: the target gets the rewritten +// request, the client the rewritten response, and the exchange records what +// the client sent and what the target answered, with the trace of both phases. +// Lengths stay coherent after a body rewrite on both sides, and full-text +// search finds the originals only. +func TestRewriteAppliedRecordsOriginals(t *testing.T) { + var got received + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + got.add(fmt.Sprintf("%s %s X-Client=%s len=%d %s", r.Method, r.RequestURI, r.Header.Get("X-Client"), r.ContentLength, body)) + w.Header().Set("X-Target", "t") + w.Header().Set("Content-Length", "11") + w.WriteHeader(http.StatusTeapot) + io.WriteString(w, "target body") + })) + t.Cleanup(target.Close) + tr, _, proxyURL := startProxy(t) + tr.SetRewriteRules(rules(t, nil, nil, + rewrite.Rule{ID: 1, Phase: rewrite.PhaseRequest, Actions: []rewrite.Action{ + {Type: rewrite.ActionSetMethod, Value: "PUT"}, + {Type: rewrite.ActionSetPathQuery, Value: "/rewritten?r=1"}, + {Type: rewrite.ActionSetHeader, Name: "X-Client", Value: "rewritten"}, + {Type: rewrite.ActionSetBody, Value: "rewritten request body"}}}, + rewrite.Rule{ID: 2, Phase: rewrite.PhaseResponse, Actions: []rewrite.Action{ + {Type: rewrite.ActionSetStatus, Value: "200"}, + {Type: rewrite.ActionSetHeader, Name: "X-Rewritten", Value: "1"}, + {Type: rewrite.ActionReplaceBody, Regex: "target", Value: "REWRITTEN-RESPONSE"}}}, + )) + + resp, body := sendThrough(t, tr, proxyURL, "POST", target.URL+"/path?q=1", "client body", http.Header{"X-Client": {"c"}}) + + if want := []string{"PUT /rewritten?r=1 X-Client=rewritten len=22 rewritten request body"}; !slices.Equal(got.all(), want) { + t.Fatalf("target received %q, want %q", got.all(), want) + } + if resp.StatusCode != 200 || resp.Header.Get("X-Target") != "t" || resp.Header.Get("X-Rewritten") != "1" || string(body) != "REWRITTEN-RESPONSE body" { + t.Fatalf("client received %d %v %q", resp.StatusCode, resp.Header, body) + } + if resp.ContentLength != -1 && resp.ContentLength != int64(len(body)) { + t.Fatalf("client Content-Length %d for a %d-byte body", resp.ContentLength, len(body)) + } + + id := onlyExchangeID(t, tr) + var method, rawURL string + var status, reqLen, respLen int + if err := tr.DB().QueryRow(`SELECT method,url,status,req_len,resp_len FROM exchanges WHERE id=?`, id).Scan(&method, &rawURL, &status, &reqLen, &respLen); err != nil { + t.Fatal(err) + } + if method != "POST" || rawURL != target.URL+"/path?q=1" || status != 418 || reqLen != 11 || respLen != 11 { + t.Fatalf("exchange row %s %s %d %d %d", method, rawURL, status, reqLen, respLen) + } + reqText, respText, err := tr.Get(id) + if err != nil { + t.Fatal(err) + } + host := strings.TrimPrefix(target.URL, "http://") + for _, want := range []string{"POST /path?q=1 HTTP/1.1\n", "Host: " + host + "\n", "X-Client: c\n", "\n\nclient body"} { + if !strings.Contains(reqText, want) { + t.Fatalf("recorded request lacks %q:\n%s", want, reqText) + } + } + for _, want := range []string{"HTTP 418\n", "X-Target: t\n", "\n\ntarget body"} { + if !strings.Contains(respText, want) { + t.Fatalf("recorded response lacks %q:\n%s", want, respText) + } + } + if strings.Contains(strings.ToLower(reqText+respText), "rewritten") { + t.Fatalf("a rewritten value was recorded as original:\n%s\n%s", reqText, respText) + } + + lines := traceRows(t, tr, id) + var summary []string + for _, l := range lines { + summary = append(summary, fmt.Sprintf("%s/%d/%s/%s", l.phase, l.ruleID.Int64, l.field.String, l.outcome)) + } + want := []string{"request/1/method/applied", "request/1/url/applied", "request/1/header:X-Client/applied", "request/1/body/applied", + "response/2/header:X-Rewritten/applied", "response/2/body/applied", "response/2/status/applied"} + if !slices.Equal(summary, want) { + t.Fatalf("trace %q, want %q", summary, want) + } + for i, l := range lines { + first := i == 0 || l.ruleID != lines[i-1].ruleID + if l.snapshot.Valid != first { + t.Fatalf("line %d: rule_snapshot present=%v, want it on the first line of each rule only", i, l.snapshot.Valid) + } + } + if string(lines[3].inline) != "rewritten request body" || string(lines[5].inline) != "REWRITTEN-RESPONSE body" { + t.Fatalf("body lines keep %q and %q", lines[3].inline, lines[5].inline) + } + + if ok, ids, err := tr.RewriteInfo(id); err != nil || !ok || !slices.Equal(ids, []int64{1, 2}) { + t.Fatalf("RewriteInfo = %v %v %v", ok, ids, err) + } + out := toolText(t, tr, "traffic_get", `{"id":"`+id+`"}`) + if !strings.HasPrefix(out, "=== REWRITTEN === rewrite rules [1 2] ") || !strings.Contains(out, "client body") { + t.Fatalf("traffic_get output:\n%s", out) + } + + h := "127.0.0.1" + for term, found := range map[string]bool{"client body": true, "target body": true, "rewritten request": false, "REWRITTEN-RESPONSE": false, "/rewritten": false} { + rows, err := tr.query(h, "", term, 0, 10) + if err != nil { + t.Fatal(err) + } + if (len(rows) == 1) != found { + t.Fatalf("full-text search for %q found %d exchanges, want found=%v", term, len(rows), found) + } + } + if e, tb := tr.snaps.counts(); e != 0 || tb != 0 { + t.Fatalf("snapshot table holds %d entries, %d tombstones after Response", e, tb) + } +} + +// toolText runs one of the traffic agent tools and returns its text. +func toolText(t *testing.T, tr *Traffic, name, input string) string { + t.Helper() + for _, tool := range tr.Tools() { + if tool.Name() != name { + continue + } + res, err := tool.Call(context.Background(), json.RawMessage(input), nil) + if err != nil { + t.Fatal(err) + } + return res.Flatten() + } + t.Fatalf("no tool %s", name) + return "" +} + +// --- non-regression ---------------------------------------------------------- + +type recordedRow struct { + exchange, bodies string +} + +func dumpOnly(t *testing.T, tr *Traffic) recordedRow { + t.Helper() + var host, method, tmpl, u, ct, path, reqHead, respHead string + var status, reqLen, respLen int + var reqBody, respBody []byte + var reqBlob, respBlob sql.NullString + if err := tr.DB().QueryRow(`SELECT e.host,e.method,e.url_template,e.url,e.status,e.content_type,e.req_len,e.resp_len,e.path, +b.req_head,b.req_body,b.req_blob,b.resp_head,b.resp_body,b.resp_blob FROM exchanges e JOIN exchange_bodies b ON b.id=e.id`). + Scan(&host, &method, &tmpl, &u, &status, &ct, &reqLen, &respLen, &path, &reqHead, &reqBody, &reqBlob, &respHead, &respBody, &respBlob); err != nil { + t.Fatal(err) + } + return recordedRow{ + exchange: fmt.Sprintf("%q %q %q %q %d %q %d %d %q", host, method, tmpl, u, status, ct, reqLen, respLen, path), + bodies: fmt.Sprintf("%q %q %v %q %q %v", reqHead, reqBody, reqBlob, respHead, respBody, respBlob), + } +} + +// Without a rule that matches, the hooks record exactly what record() records, +// byte for byte, and write no trace row. +func TestRewriteNoRuleRecordsUnchanged(t *testing.T) { + flow := func(n int) *mproxy.Flow { + return withID(newFlow("api.example.com", "POST", "/v1/items/42?x=1", []byte(`{"a":1}`), []byte(`{"ok":true}`)), n) + } + plain, _ := openTraffic(t) + plain.record(flow(1)) + want := dumpOnly(t, plain) + + for name, set := range map[string]*rewrite.Set{ + "no snapshot": nil, + "no match": rules(t, nil, nil, + rewrite.Rule{ID: 1, Phase: rewrite.PhaseRequest, Matchers: rewrite.Matchers{Host: "other.example"}, Actions: []rewrite.Action{{Type: rewrite.ActionSetBody, Value: "x"}}}, + rewrite.Rule{ID: 2, Phase: rewrite.PhaseResponse, Matchers: rewrite.Matchers{Host: "*.other.example"}, Actions: []rewrite.Action{{Type: rewrite.ActionSetStatus, Value: "500"}}}), + } { + tr, _ := openTraffic(t) + tr.SetRewriteRules(set) + s := tr.proxy.Addons[0].(*sink) + f := flow(2) + s.Request(f) + s.Response(f) + if got := dumpOnly(t, tr); got != want { + t.Fatalf("%s: recorded\n%+v\nwant\n%+v", name, got, want) + } + if n := countRows(t, tr, "exchange_rewrites"); n != 0 { + t.Fatalf("%s: %d trace rows", name, n) + } + if f.Response.StatusCode != 200 || string(f.Request.Body) != `{"a":1}` { + t.Fatalf("%s: the flow changed", name) + } + } +} + +// Through the real proxy, a snapshot whose rules match nothing gives the same +// client response, target request and recorded exchange as no snapshot. +func TestRewriteNoMatchThroughProxy(t *testing.T) { + var got received + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + got.add(r.Method + " " + r.RequestURI + " " + string(body)) + w.Header().Set("Content-Type", "text/plain") + io.WriteString(w, "target body") + })) + t.Cleanup(target.Close) + canon := func(raw string) string { + head, body, _ := strings.Cut(raw, "\n\n") + lines := strings.Split(head, "\n") + sort.Strings(lines[1:]) + lines = slices.DeleteFunc(lines, func(l string) bool { return strings.HasPrefix(l, "Date: ") }) + return strings.Join(lines, "\n") + "\n\n" + body + } + exchange := func(set *rewrite.Set) string { + tr, _, proxyURL := startProxy(t) + tr.SetRewriteRules(set) + resp, body := sendThrough(t, tr, proxyURL, "POST", target.URL+"/p?q=1", "client body", nil) + reqText, respText, err := tr.Get(onlyExchangeID(t, tr)) + if err != nil { + t.Fatal(err) + } + if n := countRows(t, tr, "exchange_rewrites"); n != 0 { + t.Fatalf("%d trace rows", n) + } + return fmt.Sprintf("%d %s|%s|%s", resp.StatusCode, body, canon(reqText), canon(respText)) + } + base := exchange(nil) + other := exchange(rules(t, nil, nil, rewrite.Rule{ID: 1, Phase: rewrite.PhaseRequest, Matchers: rewrite.Matchers{Methods: []string{"DELETE"}}, + Actions: []rewrite.Action{{Type: rewrite.ActionSetBody, Value: "x"}}})) + if base != other { + t.Fatalf("without rules:\n%s\nwith non-matching rules:\n%s", base, other) + } + if all := got.all(); len(all) != 2 || all[0] != all[1] || all[0] != "POST /p?q=1 client body" { + t.Fatalf("target received %q", all) + } +} + +// --- atomicity and log redaction --------------------------------------------- + +// A trace that cannot be written rolls the whole exchange back: no rewritten +// exchange is ever left without its trace. The client is not affected, and the +// log names the host, the step and the rules, never the URL. +func TestRewriteTraceFailureRollsBack(t *testing.T) { + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, "target body") })) + t.Cleanup(target.Close) + tr, _, proxyURL := startProxy(t) + if _, err := tr.DB().Exec(`CREATE TRIGGER fail_trace BEFORE INSERT ON exchange_rewrites BEGIN SELECT RAISE(ABORT, 'forced'); END`); err != nil { + t.Fatal(err) + } + logs := captureLog(t) + tr.SetRewriteRules(rules(t, nil, nil, rewrite.Rule{ID: 7, Phase: rewrite.PhaseRequest, + Actions: []rewrite.Action{{Type: rewrite.ActionSetPathQuery, Value: "/secret-path?token=XYZ123"}}})) + resp, body := sendThrough(t, tr, proxyURL, "GET", target.URL+"/client-path?c=1", "", nil) + if resp.StatusCode != 200 || string(body) != "target body" { + t.Fatalf("client got %d %q", resp.StatusCode, body) + } + for _, table := range []string{"exchanges", "exchange_bodies", "exchange_rewrites", "blob_refs"} { + if n := countRows(t, tr, table); n != 0 { + t.Fatalf("%s holds %d rows after a failed trace insert", table, n) + } + } + out := logs.String() + if !strings.Contains(out, "failed to record a rewritten exchange of 127.0.0.1 (writing rewrite trace; rules [7])") { + t.Fatalf("log: %q", out) + } + for _, leak := range []string{"secret-path", "XYZ123", "client-path", "forced"} { + if strings.Contains(out, leak) { + t.Fatalf("log contains %q: %q", leak, out) + } + } +} + +// A transport error on a rewritten request clears its slot, keeps the host +// intercepted, and logs neither the URL a rule wrote nor the error text. +func TestRewriteRequestErrorRedactedAndCleared(t *testing.T) { + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + dead := ln.Addr().String() + ln.Close() + tr, _, proxyURL := startProxy(t) + logs := captureLog(t) + tr.SetRewriteRules(rules(t, nil, nil, rewrite.Rule{ID: 3, Phase: rewrite.PhaseRequest, + Actions: []rewrite.Action{{Type: rewrite.ActionReplaceURL, Regex: "c=1", Value: "token=XYZ123"}}})) + resp, _ := sendThrough(t, tr, proxyURL, "GET", "http://"+dead+"/client-path?c=1", "", nil) + if resp.StatusCode != http.StatusBadGateway { + t.Fatalf("status %d", resp.StatusCode) + } + waitFor(t, "empty snapshot table", func() bool { e, tb := tr.snaps.counts(); return e == 0 && tb == 0 }) + out := logs.String() + if !strings.Contains(out, "a request to 127.0.0.1 rewritten by rules [3] failed (class transport_error)") { + t.Fatalf("log: %q", out) + } + for _, leak := range []string{"XYZ123", "client-path", dead, "refused"} { + if strings.Contains(out, leak) { + t.Fatalf("log contains %q: %q", leak, out) + } + } + if n := countRows(t, tr, "exchanges"); n != 0 { + t.Fatalf("%d exchanges", n) + } +} + +// RequestError never switches the host of a rewritten request to passthrough, +// records the rules' last error, and still does so for a request no rule changed. +func TestRewriteRequestErrorKeepsInterception(t *testing.T) { + tr, _ := openTraffic(t) + s := tr.proxy.Addons[0].(*sink) + errs := &rewrite.ErrorLog{} + tr.SetRewriteRules(rules(t, nil, errs, rewrite.Rule{ID: 9, Phase: rewrite.PhaseRequest, Matchers: rewrite.Matchers{Host: "rw.example"}, + Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-A", Value: "1"}}})) + protoErr := errors.New("http2: protocol error") + + f := withID(newFlow("rw.example", "GET", "/", nil, nil), 1) + s.Request(f) + if f.Request.Header.Get("X-A") != "1" { + t.Fatal("request not rewritten") + } + s.RequestError(f, protoErr) + if _, flagged := tr.pass.Load("rw.example"); flagged { + t.Fatal("a rewritten request flagged its host for passthrough") + } + if e, ok := errs.Get(9); !ok || e.Class != rewrite.ErrClassTransport { + t.Fatalf("last error %+v %v", e, ok) + } + g := withID(newFlow("plain.example", "GET", "/", nil, nil), 2) + s.Request(g) + s.RequestError(g, protoErr) + if _, flagged := tr.pass.Load("plain.example"); !flagged { + t.Fatal("an unrewritten request no longer flags its host") + } + if e, tb := tr.snaps.counts(); e != 0 || tb != 0 { + t.Fatalf("table holds %d entries, %d tombstones", e, tb) + } +} + +// --- snapshot table ---------------------------------------------------------- + +func liveEntry(cost int64) (*reqSnap, chan struct{}) { + ch := make(chan struct{}) + return &reqSnap{done: ch, cost: cost, host: "h.example", lines: []rewrite.TraceLine{{Phase: rewrite.PhaseRequest, RuleID: 4, Outcome: rewrite.OutcomeApplied}}, rewritten: true}, ch +} + +type fakeClock struct{ ns atomic.Int64 } + +func (c *fakeClock) now() time.Time { return time.Unix(0, c.ns.Load()) } +func (c *fakeClock) add(d time.Duration) { c.ns.Add(int64(d)) } + +func newFakeClock() *fakeClock { + c := &fakeClock{} + c.ns.Store(time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC).UnixNano()) + return c +} + +// The sweep a ClientDisconnected asks for removes the slots of finished flows +// only (Response and RequestError are covered by the proxy tests above). +func TestSnapshotClearedOnDisconnectSweep(t *testing.T) { + tr, _ := openTraffic(t) + s := tr.proxy.Addons[0].(*sink) + finished, done := liveEntry(10) + close(done) + live, _ := liveEntry(10) + if !tr.snaps.reserve(key(1), finished) || !tr.snaps.reserve(key(2), live) { + t.Fatal("reserve") + } + s.ClientDisconnected(&mproxy.ClientConn{}) + waitFor(t, "the sweep after a disconnect", func() bool { e, _ := tr.snaps.counts(); return e == 1 }) + if tr.snaps.slot(key(2)) == nil || tr.snaps.slot(key(1)) != nil { + t.Fatal("the sweep must remove the finished flow's slot and keep the live one") + } +} + +// One sweep per second at most, whatever the number of disconnects; a periodic +// pass without any; a live slot survives, even past the TTL. +func TestSnapshotSweepDebounce(t *testing.T) { + clk := newFakeClock() + tb := newSnapTable() + tb.limits(maxSnapEntries, maxSnapBytes, maxSnapTombs, clk.now) + tb.tick() + if tb.sweeps.Load() != 0 { + t.Fatal("an empty table was swept") + } + live, _ := liveEntry(10) + tb.reserve(key(1), live) + tb.tick() // first sweep: none before + start := tb.sweeps.Load() + for i := 0; i < 100; i++ { + tb.requestSweep() + tb.tick() + } + if tb.sweeps.Load() != start { + t.Fatalf("%d sweeps within the debounce", tb.sweeps.Load()-start) + } + clk.add(sweepDebounce) + tb.tick() + if tb.sweeps.Load() != start+1 { + t.Fatal("the requested sweep did not run after the debounce") + } + clk.add(30 * time.Second) + tb.tick() + if tb.sweeps.Load() != start+1 { + t.Fatal("swept without a request before the period") + } + clk.add(sweepPeriod + snapTTL) + tb.tick() + if tb.sweeps.Load() != start+2 { + t.Fatal("no periodic sweep") + } + if e, _ := tb.counts(); e != 1 { + t.Fatal("a live slot past the TTL was swept") + } +} + +// Close stops the sweeper goroutine. +func TestSnapshotSweeperStopsOnClose(t *testing.T) { + tr, err := Open(t.TempDir(), "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + done := make(chan struct{}) + go func() { _ = tr.Close(); close(done) }() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("Close did not return: the sweeper did not stop") + } +} + +// Past 1024 entries a reservation fails: the request is not rewritten and a +// capacity tombstone stands for the flow. +func TestSnapshotCapacityTombstone(t *testing.T) { + tb := newSnapTable() + for i := 0; i < maxSnapEntries; i++ { + e, _ := liveEntry(1) + if !tb.reserve(key(i), e) { + t.Fatalf("reservation %d failed", i) + } + } + e, _ := liveEntry(1) + if tb.reserve(key(5000), e) { + t.Fatal("reservation past 1024 entries succeeded") + } + if n, tombs := tb.counts(); n != maxSnapEntries || tombs != 1 { + t.Fatalf("%d entries, %d tombstones", n, tombs) + } + if s := tb.slot(key(5000)); s == nil || !s.tomb || s.reason != reasonSnapshotCapacity { + t.Fatalf("slot %+v", s) + } + if tb.capacity.Load() != 1 || tb.conversions.Load() != 0 { + t.Fatal("counters") + } + + // The 64 MiB budget bounds the entries too. + tb = newSnapTable() + for i := 0; i < 64; i++ { + e, _ := liveEntry(1 << 20) + if !tb.reserve(key(i), e) { + t.Fatalf("reservation %d failed", i) + } + } + if e, _ := liveEntry(1); tb.reserve(key(100), e) { + t.Fatal("reservation past 64 MiB succeeded") + } +} + +// Through the sink: a refused reservation leaves the request as the client +// sent it, and the exchange carries one global "snapshot capacity" line. +func TestSnapshotCapacityRecordsGlobalLine(t *testing.T) { + tr, _ := openTraffic(t) + s := tr.proxy.Addons[0].(*sink) + tr.snaps.limits(0, maxSnapBytes, maxSnapTombs, nil) + tr.SetRewriteRules(rules(t, nil, nil, rewrite.Rule{ID: 1, Phase: rewrite.PhaseRequest, Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-A", Value: "1"}}})) + f := withID(newFlow("api.example.com", "GET", "/", nil, []byte("ok")), 1) + s.Request(f) + if f.Request.Header.Get("X-A") != "" { + t.Fatal("the request was rewritten without a snapshot") + } + s.Response(f) + lines := traceRows(t, tr, onlyExchangeID(t, tr)) + if len(lines) != 1 || lines[0].ruleID.Valid || lines[0].outcome != "skipped" || lines[0].reason != reasonSnapshotCapacity { + t.Fatalf("trace %+v", lines) + } + if ok, _, _ := tr.RewriteInfo(onlyExchangeID(t, tr)); ok { + t.Fatal("an unrewritten exchange is flagged") + } +} + +// An entry of a live flow older than the TTL is turned into a tombstone in +// place when a reservation fails; the flow keeps its slot throughout. +func TestSnapshotConversionInPlace(t *testing.T) { + clk := newFakeClock() + tb := newSnapTable() + tb.limits(2, maxSnapBytes, maxSnapTombs, clk.now) + old, _ := liveEntry(10) + tb.reserve(key(1), old) + clk.add(time.Minute) + young, _ := liveEntry(10) + tb.reserve(key(2), young) + clk.add(snapTTL) + e, _ := liveEntry(10) + if !tb.reserve(key(3), e) { + t.Fatal("reservation failed although an old entry was convertible") + } + s := tb.slot(key(1)) + if s == nil || !s.tomb || s.reason != reasonOriginalUnavailable || !slices.Equal(s.ruleIDs, []int64{4}) || s.header != nil || s.lines != nil { + t.Fatalf("converted slot %+v", s) + } + if n, tombs := tb.counts(); n != 2 || tombs != 1 || tb.conversions.Load() != 1 { + t.Fatalf("%d entries %d tombstones %d conversions", n, tombs, tb.conversions.Load()) + } + if s := tb.slot(key(2)); s == nil || s.tomb { + t.Fatal("the younger entry was converted") + } +} + +// With every tombstone place held by a live flow, the conversion is refused: +// the old entry stays, the new reservation fails and, with no room for its +// tombstone either, is only counted. +func TestSnapshotConversionRefusedWithoutTombstoneRoom(t *testing.T) { + clk := newFakeClock() + tb := newSnapTable() + tb.limits(1, maxSnapBytes, 1, clk.now) + old, _ := liveEntry(10) + tb.reserve(key(1), old) + blocker, _ := liveEntry(10) + tb.reserve(key(2), blocker) // refused while old is young: the single capacity tombstone + clk.add(snapTTL + time.Minute) + e, _ := liveEntry(10) + if tb.reserve(key(3), e) { + t.Fatal("reservation succeeded") + } + if s := tb.slot(key(1)); s == nil || s.tomb { + t.Fatal("the old entry was converted without a tombstone place") + } + if tb.slot(key(3)) != nil || tb.tombDrops.Load() != 1 || tb.conversions.Load() != 0 { + t.Fatalf("drops %d conversions %d", tb.tombDrops.Load(), tb.conversions.Load()) + } + + // A tombstone of a finished flow is evicted to make room. + tb = newSnapTable() + tb.limits(1, maxSnapBytes, 1, clk.now) + old, _ = liveEntry(10) + tb.reserve(key(1), old) + gone, done := liveEntry(10) + tb.reserve(key(2), gone) // capacity tombstone of a flow that then ends + close(done) + clk.add(snapTTL + time.Minute) + tb.mu.Lock() // no sweep here: only the eviction can free the place + ok := tb.convertLocked(10) + tb.mu.Unlock() + if !ok { + t.Fatal("conversion refused although the only tombstone belongs to a finished flow") + } +} + +// The validated rule: an entry is converted only when releasing it lets the +// new reservation succeed. When the freed budget is too small, nothing is +// converted and the new request goes the capacity path. +func TestSnapshotConversionFreedBudgetInsufficient(t *testing.T) { + clk := newFakeClock() + tb := newSnapTable() + tb.limits(10, 1000, maxSnapTombs, clk.now) + old, _ := liveEntry(100) + tb.reserve(key(1), old) + clk.add(snapTTL + time.Minute) + young, _ := liveEntry(850) + tb.reserve(key(2), young) + e, _ := liveEntry(200) // 950 - 100 + 200 > 1000 + if tb.reserve(key(3), e) { + t.Fatal("reservation succeeded") + } + if s := tb.slot(key(1)); s == nil || s.tomb { + t.Fatal("the old entry was converted although its release frees too little") + } + if s := tb.slot(key(3)); s == nil || s.reason != reasonSnapshotCapacity || tb.conversions.Load() != 0 { + t.Fatalf("new slot %+v, conversions %d", s, tb.conversions.Load()) + } + // With enough freed, the same old entry is converted. + e, _ = liveEntry(140) // 950 - 100 + 140 <= 1000 + tb.mu.Lock() + tb.lastSweep = time.Time{} + tb.mu.Unlock() + if !tb.reserve(key(4), e) || tb.conversions.Load() != 1 { + t.Fatal("conversion refused although the release frees enough") + } +} + +// A failed reservation right after a sweep is refused by the debounce: no +// sweep, no conversion. +func TestSnapshotReservationDebounced(t *testing.T) { + clk := newFakeClock() + tb := newSnapTable() + tb.limits(1, maxSnapBytes, maxSnapTombs, clk.now) + old, _ := liveEntry(10) + tb.reserve(key(1), old) + clk.add(snapTTL + time.Minute) + tb.mu.Lock() + tb.lastSweep = clk.now() + tb.mu.Unlock() + e, _ := liveEntry(10) + if tb.reserve(key(2), e) || tb.conversions.Load() != 0 || tb.sweeps.Load() != 0 { + t.Fatal("a debounced reservation swept or converted") + } +} + +// End to end through the proxy: a live flow whose entry is converted under +// pressure is recorded "original request unavailable", with the client's +// method, URL and Host from Raw(), never the rewritten request. +func TestRewriteOriginalUnavailable(t *testing.T) { + arrived, release := make(chan struct{}), make(chan struct{}) + var got received + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got.add(r.URL.Path + " X-Rule=" + r.Header.Get("X-Rule")) + if r.URL.Path == "/slow" { + close(arrived) + <-release + } + io.WriteString(w, "ok") + })) + t.Cleanup(target.Close) + tr, _, proxyURL := startProxy(t) + clk := newFakeClock() + tr.snaps.limits(1, maxSnapBytes, maxSnapTombs, clk.now) + tr.SetRewriteRules(rules(t, nil, nil, rewrite.Rule{ID: 1, Phase: rewrite.PhaseRequest, Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-Rule", Value: "v"}}})) + + slowDone := make(chan string) + go func() { + resp, err := proxyClient(t, tr, proxyURL).Get(target.URL + "/slow?a=1") + if err != nil { + slowDone <- err.Error() + return + } + b, _ := io.ReadAll(resp.Body) + resp.Body.Close() + slowDone <- string(b) + }() + <-arrived + clk.add(snapTTL + time.Minute) + if _, body := sendThrough(t, tr, proxyURL, "GET", target.URL+"/fast", "", nil); string(body) != "ok" { + t.Fatalf("fast body %q", body) + } + close(release) + if b := <-slowDone; b != "ok" { + t.Fatalf("slow: %s", b) + } + if all := got.all(); !slices.Contains(all, "/slow X-Rule=v") || !slices.Contains(all, "/fast X-Rule=v") { + t.Fatalf("target received %q", all) + } + waitFor(t, "both exchanges", func() bool { return countRows(t, tr, "exchanges") == 2 }) + + id := exchangeIDByPath(t, tr, "/slow?a=1") + var method, u, host, reqHead string + var reqLen int + if err := tr.DB().QueryRow(`SELECT e.method,e.url,e.host,e.req_len,b.req_head FROM exchanges e JOIN exchange_bodies b ON b.id=e.id WHERE e.id=?`, id).Scan(&method, &u, &host, &reqLen, &reqHead); err != nil { + t.Fatal(err) + } + if method != "GET" || u != target.URL+"/slow?a=1" || host != "127.0.0.1" || reqLen != 0 || + reqHead != "GET /slow?a=1 HTTP/1.1\nHost: "+strings.TrimPrefix(target.URL, "http://")+"\n"+unavailableMarker { + t.Fatalf("exchange %s %s %s %d %q", method, u, host, reqLen, reqHead) + } + lines := traceRows(t, tr, id) + if len(lines) != 2 || lines[0].outcome != "error" || lines[0].reason != reasonOriginalUnavailable || lines[0].ruleID.Valid || + lines[1].outcome != "applied" || lines[1].ruleID.Int64 != 1 || lines[1].field.Valid { + t.Fatalf("trace %+v", lines) + } + if ok, ids, _ := tr.RewriteInfo(id); !ok || !slices.Equal(ids, []int64{1}) { + t.Fatal("RewriteInfo of the converted flow") + } + if fast := traceRows(t, tr, exchangeIDByPath(t, tr, "/fast")); len(fast) != 1 || fast[0].outcome != "applied" { + t.Fatalf("fast trace %+v", fast) + } + if tr.snaps.conversions.Load() != 1 { + t.Fatal("no conversion counted") + } +} + +// Without Raw(), the unavailable request records an empty method, URL and host. +func TestRewriteOriginalUnavailableWithoutRaw(t *testing.T) { + tr, _ := openTraffic(t) + s := tr.proxy.Addons[0].(*sink) + f := withID(newFlow("api.example.com", "PUT", "/rewritten", []byte("rewritten"), []byte("ok")), 1) + tr.snaps.mu.Lock() + tr.snaps.slots[keyOf(f)] = &reqSnap{tomb: true, reason: reasonOriginalUnavailable, ruleIDs: []int64{5}, host: "api.example.com"} + tr.snaps.tombs++ + tr.snaps.mu.Unlock() + s.Response(f) + var method, u, host, reqHead string + if err := tr.DB().QueryRow(`SELECT e.method,e.url,e.host,b.req_head FROM exchanges e JOIN exchange_bodies b ON b.id=e.id`).Scan(&method, &u, &host, &reqHead); err != nil { + t.Fatal(err) + } + if method != "" || u != "" || host != "" || strings.Contains(reqHead, "rewritten") || !strings.HasSuffix(reqHead, unavailableMarker) { + t.Fatalf("exchange %q %q %q %q", method, u, host, reqHead) + } +} + +// --- fail-open --------------------------------------------------------------- + +// A rule that panics is an error line; the flow goes on with the request and +// response untouched, and the exchange is recorded. +func TestRewriteRulePanicKeepsFlow(t *testing.T) { + var got received + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + got.add(string(b)) + io.WriteString(w, "target body") + })) + t.Cleanup(target.Close) + tr, _, proxyURL := startProxy(t) + tr.SetRewriteRules(rules(t, nil, nil, + rewrite.Rule{ID: 1, Phase: rewrite.PhaseRequest, Actions: []rewrite.Action{{Type: rewrite.ActionSetBody, Value: "x"}}}, + rewrite.Rule{ID: 2, Phase: rewrite.PhaseResponse, Actions: []rewrite.Action{{Type: rewrite.ActionSetBody, Value: "y"}}})) + setTestHook(t, func(stage string) { + if stage == "rule" { + panic("rule") + } + }) + resp, body := sendThrough(t, tr, proxyURL, "POST", target.URL+"/", "client body", nil) + if resp.StatusCode != 200 || string(body) != "target body" || !slices.Equal(got.all(), []string{"client body"}) { + t.Fatalf("client %d %q, target %q", resp.StatusCode, body, got.all()) + } + lines := traceRows(t, tr, onlyExchangeID(t, tr)) + if len(lines) != 2 || lines[0].outcome != "error" || lines[0].reason != rewrite.ErrClassPanic || lines[1].outcome != "error" { + t.Fatalf("trace %+v", lines) + } +} + +// A panic in the sink itself, after the engine, is recovered by the sink: the +// client still gets the target's response, not the library's empty 200. +func TestRewriteSinkRecoverKeepsFlow(t *testing.T) { + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusAccepted) + io.WriteString(w, "target body") + })) + t.Cleanup(target.Close) + tr, _, proxyURL := startProxy(t) + tr.SetRewriteRules(rules(t, nil, nil, + rewrite.Rule{ID: 1, Phase: rewrite.PhaseRequest, Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-A", Value: "1"}}}, + rewrite.Rule{ID: 2, Phase: rewrite.PhaseResponse, Actions: []rewrite.Action{{Type: rewrite.ActionSetStatus, Value: "500"}}})) + logs := captureLog(t) + setTestHook(t, func(stage string) { + if stage == "request" || stage == "response" { + panic(stage) + } + }) + resp, body := sendThrough(t, tr, proxyURL, "GET", target.URL+"/p", "", nil) + if resp.StatusCode != http.StatusAccepted || string(body) != "target body" { + t.Fatalf("client %d %q", resp.StatusCode, body) + } + if n := countRows(t, tr, "exchanges"); n != 1 { + t.Fatalf("%d exchanges", n) + } + if out := logs.String(); !strings.Contains(out, "class panic") || strings.Contains(out, "/p") { + t.Fatalf("log %q", out) + } +} + +// --- 5 MiB passthrough ------------------------------------------------------- + +// StreamResponseModifier hands back the reader it got, nil included, and is +// nil-safe. +func TestStreamResponseModifierPassThrough(t *testing.T) { + tr, _ := openTraffic(t) + s := tr.proxy.Addons[0].(*sink) + f := withID(newFlow("a.example", "GET", "/", nil, nil), 1) + f.Stream = true + e, _ := liveEntry(1) + tr.snaps.reserve(keyOf(f), e) + if out := s.StreamResponseModifier(f, nil); out != nil { + t.Fatalf("nil reader became %v", out) + } + if n, _ := tr.snaps.counts(); n != 0 || tr.RewriteStats().LargeResponses != 1 { + t.Fatal("the slot of a streamed response was not released and counted") + } + r := strings.NewReader("x") + if out := s.StreamResponseModifier(f, r); out != io.Reader(r) { + t.Fatal("the reader was replaced") + } + if out := (&sink{}).StreamResponseModifier(nil, r); out != io.Reader(r) { + t.Fatal("nil flow / nil traffic") + } + if out := (*sink)(nil).StreamResponseModifier(f, r); out != io.Reader(r) { + t.Fatal("nil sink") + } +} + +// A rewritten request whose response reaches 5 MiB: the response reaches the +// client intact, nothing is recorded, the slot is released and counted. +func TestRewriteLargeResponsePassedThrough(t *testing.T) { + big := bytes.Repeat([]byte("0123456789abcdef"), (libraryStreamBody/16)+1024) + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("X-A") != "1" { + w.WriteHeader(400) + return + } + w.Write(big) + })) + t.Cleanup(target.Close) + tr, _, proxyURL := startProxy(t) + tr.SetRewriteRules(rules(t, nil, nil, rewrite.Rule{ID: 1, Phase: rewrite.PhaseRequest, Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-A", Value: "1"}}})) + resp, body := sendThrough(t, tr, proxyURL, "GET", target.URL+"/big", "", nil) + if resp.StatusCode != 200 || !bytes.Equal(body, big) { + t.Fatalf("status %d, %d bytes, intact=%v", resp.StatusCode, len(body), bytes.Equal(body, big)) + } + waitFor(t, "large response counted", func() bool { return tr.RewriteStats().LargeResponses == 1 }) + if n, tb := tr.snaps.counts(); n != 0 || tb != 0 || countRows(t, tr, "exchanges") != 0 { + t.Fatal("slot kept or exchange recorded") + } +} + +// --- purge, collisions, sensitive values, skip reasons ------------------------ + +// Deleting a host takes the trace with it, and gcBlobs frees the trace's blobs. +func TestRewriteDeleteHostTakesTrace(t *testing.T) { + tr, dir := openTraffic(t) + s := tr.proxy.Addons[0].(*sink) + large := strings.Repeat("L", maxInlineBody+1) + tr.SetRewriteRules(rules(t, nil, nil, rewrite.Rule{ID: 1, Phase: rewrite.PhaseResponse, Actions: []rewrite.Action{{Type: rewrite.ActionSetBody, Value: large}}})) + f := withID(newFlow("gone.example", "GET", "/", nil, []byte("small")), 1) + s.Request(f) + s.Response(f) + if string(f.Response.Body) != large { + t.Fatal("response not rewritten") + } + lines := traceRows(t, tr, onlyExchangeID(t, tr)) + if len(lines) != 1 || !lines[0].blob.Valid || len(lines[0].inline) == 0 { + t.Fatalf("trace %+v", lines) + } + blob := lines[0].blob.String + blobFile := dir + "/_blobs/sha256/" + blob[:2] + "/" + blob + ".bin" + if _, err := os.Stat(blobFile); err != nil { + t.Fatalf("trace blob: %v", err) + } + if n, err := tr.DeleteHost("gone.example"); err != nil || n != 1 { + t.Fatalf("DeleteHost = %d %v", n, err) + } + for _, table := range []string{"exchange_rewrites", "blob_refs", "exchanges"} { + if n := countRows(t, tr, table); n != 0 { + t.Fatalf("%s holds %d rows", table, n) + } + } + if _, err := os.Stat(blobFile); !os.IsNotExist(err) { + t.Fatalf("trace blob survived gcBlobs: %v", err) + } +} + +// An exchange id can repeat. Each of the three INSERT OR REPLACE paths clears +// the trace left under the id first, so the new exchange is never flagged. +func TestRewriteIDCollisionClearsTrace(t *testing.T) { + paths := map[string]func(tr *Traffic){ + "record": func(tr *Traffic) { tr.record(newFlow("c.example", "GET", "/", nil, nil)) }, + "RecordExternal": func(tr *Traffic) { + if err := tr.RecordExternal(ExternalExchange{Method: "GET", URL: "http://c.example/", Status: 200}); err != nil { + t.Fatal(err) + } + }, + "recordStreamStart": func(tr *Traffic) { + u, _ := url.Parse("https://c.example/ws") + f := &mproxy.Flow{Request: &mproxy.Request{Method: "GET", URL: u, Proto: "HTTP/1.1", Header: http.Header{}}, WebScoket: &mproxy.WebSocketData{}} + tr.proxy.Addons[0].(*sink).WebSocketStart(f) + }, + } + for name, write := range paths { + tr, _ := openTraffic(t) + var id string + for attempt := 0; ; attempt++ { + id = fmt.Sprintf("%d-%04d", time.Now().Unix(), (tr.seq.Load()+1)%10000) + if _, err := tr.DB().Exec(`INSERT INTO exchange_rewrites(exchange_id,seq,phase,rule_id,outcome) VALUES(?,0,'request',1,'applied')`, id); err != nil { + t.Fatal(err) + } + write(tr) + var got string + if err := tr.DB().QueryRow(`SELECT id FROM exchanges ORDER BY rowid DESC LIMIT 1`).Scan(&got); err != nil { + t.Fatal(err) + } + if got == id { + break + } + if attempt == 5 { + t.Fatalf("%s: could not land on the planted id", name) + } + if _, err := tr.DB().Exec(`DELETE FROM exchange_rewrites`); err != nil { + t.Fatal(err) + } + } + if ok, _, err := tr.RewriteInfo(id); err != nil || ok { + t.Fatalf("%s: the new exchange inherited the old trace", name) + } + } +} + +// A sensitive value is never in the trace: the line keeps rule id, field and +// a 128-bit HMAC under the trace sub-key, or no fingerprint without a key. +func TestRewriteSensitiveValueNotInTrace(t *testing.T) { + const secret = "Bearer s3cr3t-token-value" + master := []byte("instance key for the test") + m := hmac.New(sha256.New, master) + m.Write([]byte("artex-traffic-trace")) + sub := m.Sum(nil) + m = hmac.New(sha256.New, sub) + m.Write([]byte(secret)) + wantHash := hex.EncodeToString(m.Sum(nil)[:16]) + + for _, withKey := range []bool{true, false} { + tr, _ := openTraffic(t) + logs := captureLog(t) + s := tr.proxy.Addons[0].(*sink) + var k []byte + if withKey { + k = sub + } + tr.SetRewriteRules(rules(t, k, nil, rewrite.Rule{ID: 6, Phase: rewrite.PhaseRequest, + Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "Authorization", Value: secret, Sensitive: true}}})) + f := withID(newFlow("api.example.com", "GET", "/", nil, []byte("ok")), 1) + s.Request(f) + if f.Request.Header.Get("Authorization") != secret { + t.Fatal("the sensitive value did not reach the request") + } + s.Response(f) + id := onlyExchangeID(t, tr) + var dump string + if err := tr.DB().QueryRow(`SELECT group_concat(coalesce(rule_snapshot,'')||'|'||coalesce(after_inline,'')||'|'||coalesce(after_hash,'')||'|'||coalesce(before_hash,'')||'|'||reason, ';') FROM exchange_rewrites`).Scan(&dump); err != nil { + t.Fatal(err) + } + req, _, _ := tr.Get(id) + if strings.Contains(dump, "s3cr3t") || strings.Contains(req, "s3cr3t") || strings.Contains(logs.String(), "s3cr3t") { + t.Fatalf("secret leaked: trace %q request %q log %q", dump, req, logs.String()) + } + lines := traceRows(t, tr, id) + if len(lines) != 1 || lines[0].ruleID.Int64 != 6 || lines[0].field.String != "header:Authorization" || lines[0].inline != nil || lines[0].blob.Valid { + t.Fatalf("trace %+v", lines) + } + if withKey && lines[0].afterHash.String != wantHash { + t.Fatalf("fingerprint %q, want %q", lines[0].afterHash.String, wantHash) + } + if !withKey && lines[0].afterHash.Valid { + t.Fatal("a fingerprint without a key") + } + } +} + +// Body rules are skipped, with their reason, for a compressed, binary or too +// large body; rules without a body component still apply. +func TestRewriteBodySkipReasons(t *testing.T) { + tr, _ := openTraffic(t) + s := tr.proxy.Addons[0].(*sink) + tr.SetRewriteRules(rules(t, nil, nil, + rewrite.Rule{ID: 1, Phase: rewrite.PhaseRequest, Actions: []rewrite.Action{{Type: rewrite.ActionReplaceBody, Regex: "a", Value: "b"}}}, + rewrite.Rule{ID: 2, Phase: rewrite.PhaseResponse, Actions: []rewrite.Action{{Type: rewrite.ActionReplaceBody, Regex: "a", Value: "b"}}}, + rewrite.Rule{ID: 3, Phase: rewrite.PhaseResponse, Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-H", Value: "1"}}})) + cases := []struct { + name string + f *mproxy.Flow + want []string + }{ + {"content-encoding", withID(newFlow("e.example", "GET", "/ce", nil, []byte("aaa"), func(f *mproxy.Flow) { f.Response.Header.Set("Content-Encoding", "gzip") }), 1), + []string{"response/2/skipped/not rewritten: content-encoding", "response/3/applied/"}}, + {"binary", withID(newFlow("e.example", "GET", "/bin", nil, []byte("aaa"), withRespType("image/png")), 2), + []string{"response/2/skipped/not rewritten: binary", "response/3/applied/"}}, + {"too large", withID(newFlow("e.example", "POST", "/big", bytes.Repeat([]byte("a"), rewrite.MaxBodySize+1), []byte("x")), 3), + []string{"request/1/skipped/not rewritten: body too large", "response/3/applied/"}}, + } + for _, c := range cases { + s.Request(c.f) + s.Response(c.f) + var got []string + for _, l := range traceRows(t, tr, exchangeIDByPath(t, tr, c.f.Request.URL.Path)) { + got = append(got, fmt.Sprintf("%s/%d/%s/%s", l.phase, l.ruleID.Int64, l.outcome, l.reason)) + } + if !slices.Equal(got, c.want) { + t.Fatalf("%s: trace %q, want %q", c.name, got, c.want) + } + if c.f.Response.Header.Get("X-H") != "1" { + t.Fatalf("%s: the header rule did not apply", c.name) + } + } + if string(cases[0].f.Response.Body) != "aaa" || cases[0].f.Response.Header.Get("Content-Encoding") != "gzip" { + t.Fatal("a compressed body was rewritten") + } +} + +// --- streams ----------------------------------------------------------------- + +// SSE: the request is rewritten; at SSEStart the status and header actions +// reach the client, the body rule is traced "not rewritten: stream"; the +// exchange records the client's request and the target's status and headers, +// with the trace of both phases, and the events. +func TestRewriteSSEHeadersAndStatus(t *testing.T) { + var got received + srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got.add(r.URL.RequestURI() + " X-Client=" + r.Header.Get("X-Client")) + w.Header().Set("Content-Type", "text/event-stream") + w.Header().Set("X-Target", "t") + w.WriteHeader(200) + fl := w.(http.Flusher) + for _, ev := range []string{"id: 1\ndata: hello\n\n", "data: bye\n\n"} { + _, _ = io.WriteString(w, ev) + fl.Flush() + } + })) + t.Cleanup(srv.Close) + tr, _, proxyURL := startProxy(t) + tr.SetRewriteRules(rules(t, nil, nil, + rewrite.Rule{ID: 1, Phase: rewrite.PhaseRequest, Actions: []rewrite.Action{ + {Type: rewrite.ActionSetPathQuery, Value: "/rewritten-events?x=1"}, + {Type: rewrite.ActionSetHeader, Name: "X-Client", Value: "rewritten"}}}, + rewrite.Rule{ID: 2, Phase: rewrite.PhaseResponse, Actions: []rewrite.Action{ + {Type: rewrite.ActionSetStatus, Value: "203"}, + {Type: rewrite.ActionSetHeader, Name: "X-Rewritten", Value: "1"}}}, + rewrite.Rule{ID: 3, Phase: rewrite.PhaseResponse, Actions: []rewrite.Action{{Type: rewrite.ActionReplaceBody, Regex: "hello", Value: "HIJACKED"}}})) + + resp, body := sendThrough(t, tr, proxyURL, "GET", localhostURL(srv.URL)+"/events", "", http.Header{"X-Client": {"c"}}) + if resp.StatusCode != 203 || resp.Header.Get("X-Rewritten") != "1" || resp.Header.Get("X-Target") != "t" || + !strings.Contains(string(body), "data: hello") || strings.Contains(string(body), "HIJACKED") { + t.Fatalf("client got %d %v %q", resp.StatusCode, resp.Header, body) + } + if all := got.all(); !slices.Equal(all, []string{"/rewritten-events?x=1 X-Client=rewritten"}) { + t.Fatalf("target received %q", all) + } + var ex streamExchange + waitFor(t, "sse end", func() bool { + exs := streamExchanges(t, tr, "%event-stream%") + if len(exs) != 1 || exs[0].respLen == 0 { + return false + } + ex = exs[0] + return true + }) + if ex.status != 200 || ex.url != localhostURL(srv.URL)+"/events" { + t.Fatalf("exchange %+v", ex) + } + reqText, respText, err := tr.Get(ex.id) + if err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(reqText, "GET /events HTTP/1.1\n") || !strings.Contains(reqText, "X-Client: c\n") || strings.Contains(reqText, "rewritten") || + !strings.HasPrefix(respText, "HTTP 200\n") || strings.Contains(respText, "X-Rewritten") { + t.Fatalf("recorded:\n%s\n%s", reqText, respText) + } + var summary []string + for _, l := range traceRows(t, tr, ex.id) { + summary = append(summary, fmt.Sprintf("%s/%d/%s/%s/%s", l.phase, l.ruleID.Int64, l.field.String, l.outcome, l.reason)) + } + want := []string{"request/1/url/applied/", "request/1/header:X-Client/applied/", + "response/2/header:X-Rewritten/applied/", "response/2/status/applied/", "response/3//skipped/not rewritten: stream"} + if !slices.Equal(summary, want) { + t.Fatalf("trace %q, want %q", summary, want) + } + if msgs := allMessages(t, tr, ex.id); len(msgs) != 2 || msgs[0].Payload != "hello" { + t.Fatalf("events %+v", msgs) + } + if n, tb := tr.snaps.counts(); n != 0 || tb != 0 { + t.Fatal("slot kept after SSEStart") + } +} + +// WebSocket: go-mitmproxy calls WebSocketStart after it has upgraded the +// client, with no response on the flow, so the handshake cannot be rewritten: +// every candidate rule is traced "not rewritten: stream", the handshake is +// recorded as the client sent it, and the frames pass unchanged. +func TestRewriteWebSocketTracedNotRewritten(t *testing.T) { + var gotRule atomic.Value + up := websocket.Upgrader{CheckOrigin: func(*http.Request) bool { return true }} + srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotRule.Store(r.Header.Get("X-Rule")) + c, err := up.Upgrade(w, r, nil) + if err != nil { + return + } + defer c.Close() + for { + mt, msg, err := c.ReadMessage() + if err != nil { + return + } + if err := c.WriteMessage(mt, append([]byte("echo:"), msg...)); err != nil { + return + } + } + })) + t.Cleanup(srv.Close) + tr, _, proxyURL := startProxy(t) + tr.SetRewriteRules(rules(t, nil, nil, + rewrite.Rule{ID: 1, Phase: rewrite.PhaseRequest, Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-Rule", Value: "1"}}}, + rewrite.Rule{ID: 2, Phase: rewrite.PhaseResponse, Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-Resp", Value: "1"}}}, + rewrite.Rule{ID: 3, Phase: rewrite.PhaseResponse, Matchers: rewrite.Matchers{Host: "elsewhere.example"}, Actions: []rewrite.Action{{Type: rewrite.ActionSetHeader, Name: "X-Resp", Value: "2"}}})) + target := "wss://" + strings.TrimPrefix(localhostURL(srv.URL), "https://") + "/ws" + c, hs, err := wsDialer(t, tr, proxyURL).Dial(target, http.Header{"X-Client": {"c"}}) + if err != nil { + t.Fatal(err) + } + if hs.Header.Get("X-Resp") != "" { + t.Fatal("the handshake response was rewritten") + } + if err := c.WriteMessage(websocket.TextMessage, []byte("hi")); err != nil { + t.Fatal(err) + } + if _, msg, err := c.ReadMessage(); err != nil || string(msg) != "echo:hi" { + t.Fatalf("echo %q %v", msg, err) + } + closeWS(c) + if v, _ := gotRule.Load().(string); v != "" { + t.Fatal("the handshake request was rewritten") + } + var ex streamExchange + waitFor(t, "websocket end", func() bool { + exs := streamExchanges(t, tr, "websocket") + if len(exs) != 1 || exs[0].reqLen == 0 { + return false + } + ex = exs[0] + return true + }) + var summary []string + for _, l := range traceRows(t, tr, ex.id) { + summary = append(summary, fmt.Sprintf("%s/%d/%s/%s", l.phase, l.ruleID.Int64, l.outcome, l.reason)) + } + if want := []string{"request/1/skipped/not rewritten: stream", "response/2/skipped/not rewritten: stream"}; !slices.Equal(summary, want) { + t.Fatalf("trace %q, want %q", summary, want) + } + if ok, _, _ := tr.RewriteInfo(ex.id); ok { + t.Fatal("a websocket exchange is flagged rewritten") + } + reqText, _, _ := tr.Get(ex.id) + if !strings.Contains(reqText, "X-Client: c\n") || strings.Contains(reqText, "X-Rule") { + t.Fatalf("handshake recorded:\n%s", reqText) + } + if msgs := allMessages(t, tr, ex.id); len(msgs) != 2 || msgs[0].Payload != "hi" || msgs[1].Payload != "echo:hi" { + t.Fatalf("messages %+v", msgs) + } +} diff --git a/traffic/rewrite_rules_test.go b/traffic/rewrite_rules_test.go deleted file mode 100644 index 11ca1f4..0000000 --- a/traffic/rewrite_rules_test.go +++ /dev/null @@ -1,102 +0,0 @@ -package traffic - -import ( - "io" - "net/http" - "net/http/httptest" - "slices" - "sort" - "strings" - "sync" - "testing" - - "github.com/Autumn-27/artex/traffic/rewrite" -) - -// The proxy stores the compiled rewrite rules and does not apply them. With a -// snapshot whose rules match every exchange, what the target receives, what the -// client receives and what is recorded are the same as with no rule at all. -func TestRewriteRulesStoredNotApplied(t *testing.T) { - var mu sync.Mutex - var seen []string - target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - body, _ := io.ReadAll(r.Body) - mu.Lock() - seen = append(seen, r.Method+" "+r.RequestURI+" X-Client="+r.Header.Get("X-Client")+" "+string(body)) - mu.Unlock() - w.Header().Set("X-Target", "t") - w.WriteHeader(http.StatusTeapot) - io.WriteString(w, "target body") - })) - t.Cleanup(target.Close) - - set := rewrite.Compile([]rewrite.Rule{ - {ID: 1, Name: "request", Enabled: true, Phase: rewrite.PhaseRequest, Actions: []rewrite.Action{ - {Type: rewrite.ActionSetMethod, Value: "PUT"}, - {Type: rewrite.ActionSetPathQuery, Value: "/rewritten"}, - {Type: rewrite.ActionSetHeader, Name: "X-Client", Value: "rewritten"}, - {Type: rewrite.ActionSetBody, Value: "rewritten"}}}, - {ID: 2, Name: "response", Enabled: true, Phase: rewrite.PhaseResponse, Actions: []rewrite.Action{ - {Type: rewrite.ActionSetStatus, Value: "200"}, - {Type: rewrite.ActionSetHeader, Name: "X-Rewritten", Value: "1"}, - {Type: rewrite.ActionSetBody, Value: "rewritten"}}}, - }, rewrite.Options{}) - if set.Len(rewrite.PhaseRequest) != 1 || set.Len(rewrite.PhaseResponse) != 1 || len(set.LoadErrors()) != 0 { - t.Fatalf("the snapshot must hold both rules: %v", set.LoadErrors()) - } - - // canon makes a recorded message comparable: the recorder writes headers in - // map order, and the target dates its response. - canon := func(raw string) string { - head, body, _ := strings.Cut(raw, "\n\n") - lines := strings.Split(head, "\n") - sort.Strings(lines[1:]) - lines = slices.DeleteFunc(lines, func(l string) bool { return strings.HasPrefix(l, "Date: ") }) - return strings.Join(lines, "\n") + "\n\n" + body - } - // exchange sends one request through a fresh proxy holding set and returns - // what the client received and what the proxy recorded. - exchange := func(set *rewrite.Set) (client, recorded string) { - tr, _, proxyURL := startProxy(t) - if tr.RewriteRules() != nil { - t.Fatal("Open published rewrite rules") - } - tr.SetRewriteRules(set) - if tr.RewriteRules() != set { - t.Fatal("RewriteRules must return the snapshot last published") - } - req, err := http.NewRequest("POST", target.URL+"/path?q=1", strings.NewReader("client body")) - if err != nil { - t.Fatal(err) - } - req.Header.Set("X-Client", "c") - resp, err := proxyClient(t, tr, proxyURL).Do(req) - if err != nil { - t.Fatal(err) - } - defer resp.Body.Close() - body, _ := io.ReadAll(resp.Body) - client = resp.Status + " X-Target=" + resp.Header.Get("X-Target") + " X-Rewritten=" + resp.Header.Get("X-Rewritten") + " " + string(body) - reqText, respText, err := tr.Get(onlyExchangeID(t, tr)) - if err != nil { - t.Fatal(err) - } - return client, canon(reqText) + "\n--\n" + canon(respText) - } - - baseClient, baseRecorded := exchange(nil) - client, recorded := exchange(set) - if want := "418 I'm a teapot X-Target=t X-Rewritten= target body"; baseClient != want || client != want { - t.Fatalf("client received %q without rules and %q with rules, want %q", baseClient, client, want) - } - if recorded != baseRecorded { - t.Fatalf("the recorded exchange differs with rules stored:\n without: %s\n with: %s", baseRecorded, recorded) - } - if !strings.Contains(recorded, "client body") || !strings.Contains(recorded, "target body") || strings.Contains(recorded, "rewritten") { - t.Fatalf("recorded exchange: %s", recorded) - } - want := "POST /path?q=1 X-Client=c client body" - if len(seen) != 2 || seen[0] != want || seen[1] != want { - t.Fatalf("the target received %q, want %q twice", seen, want) - } -} diff --git a/traffic/stream.go b/traffic/stream.go index c502ebc..d58c5a0 100644 --- a/traffic/stream.go +++ b/traffic/stream.go @@ -16,6 +16,7 @@ import ( "unsafe" "github.com/Autumn-27/artex/db" + "github.com/Autumn-27/artex/traffic/rewrite" mproxy "github.com/lqqyt2423/go-mitmproxy/proxy" ) @@ -144,17 +145,35 @@ func (s *sink) startStream(f *mproxy.Flow, kind string) { if f.Request == nil || f.Request.URL == nil || (kind == "sse" && f.Response == nil) { return } - id, err := s.t.recordStreamStart(f, kind) + rq, rs, lines, res := s.rewriteStream(f, kind) + id, err := s.t.recordStreamStart(rq, rs, kind, lines) + s.applyStreamResponse(f, res) if err != nil { + if rewritten, ruleIDs := appliedRules(lines); rewritten { + var re *recordError + errors.As(err, &re) + log.Printf("[traffic] failed to record a rewritten %s stream of %s (%s; rules %v)", kind, f.Request.URL.Hostname(), re.step, ruleIDs) + return + } log.Printf("[traffic] failed to record %s stream %s: %v", kind, f.Request.URL, err) return } st.id = id if kind == "sse" { - st.reqBytes = int64(len(f.Request.Body)) + st.reqBytes = int64(len(rq.body)) } } +// recordError names the step of a recording that failed. Its text is the step +// and the cause, as recordStreamStart always reported. +type recordError struct { + step string + err error +} + +func (e *recordError) Error() string { return e.step + ": " + e.err.Error() } +func (e *recordError) Unwrap() error { return e.err } + // libMutex returns the unexported mu sync.Mutex that go-mitmproxy holds while it // appends to WebSocketData.Messages / SSEData.Events. The library exposes no // accessor, and the two WebSocket goroutines append concurrently, so reading the @@ -252,9 +271,9 @@ func sseEventsSince(sse *mproxy.SSEData, st *streamState) []streamRow { // streamDisplayURL is the URL shown for a WebSocket exchange: http→ws and // https→wss. A plain ws:// connection reaches the WebSocket hooks through its // CONNECT tunnel, where only the target authority is known, not the path. -func streamDisplayURL(f *mproxy.Flow, kind string) (method, display, tmpl string) { - u := *f.Request.URL - method = f.Request.Method +func streamDisplayURL(reqMethod string, reqURL *url.URL, kind string) (method, display, tmpl string) { + u := *reqURL + method = reqMethod if kind == "websocket" && strings.EqualFold(method, "CONNECT") { host := u.Host if host == "" { @@ -270,63 +289,77 @@ func streamDisplayURL(f *mproxy.Flow, kind string) (method, display, tmpl string u.Scheme = "wss" } } - return method, u.String(), db.TemplatePath(f.Request.URL.EscapedPath()) + return method, u.String(), db.TemplatePath(reqURL.EscapedPath()) } // recordStreamStart writes the stream's exchange: metadata, request head and -// body, the response head (synthetic 101 for WebSocket), and the header text in -// the full-text index. Messages are never indexed. -func (t *Traffic) recordStreamStart(f *mproxy.Flow, kind string) (string, error) { +// body, the response head (synthetic 101 for WebSocket), the rewrite trace, +// and the header text in the full-text index. Messages are never indexed. rq +// is the request the client sent and rs (SSE only) the status and headers the +// target sent, whatever the rules changed. +func (t *Traffic) recordStreamStart(rq requestView, rs responseView, kind string, lines []rewrite.TraceLine) (string, error) { now := time.Now() id := t.newExchangeID(now) - method, display, tmpl := streamDisplayURL(f, kind) - host := f.Request.URL.Hostname() - target := f.Request.URL.RequestURI() - if strings.EqualFold(f.Request.Method, "CONNECT") { - target = f.Request.URL.Host // authority-form, as on the wire + method, display, tmpl, host, target := rq.method, "", "", "", "" + if rq.url != nil { + method, display, tmpl = streamDisplayURL(rq.method, rq.url, kind) + host = rq.url.Hostname() + target = rq.url.RequestURI() + if strings.EqualFold(rq.method, "CONNECT") { + target = rq.url.Host // authority-form, as on the wire + } } - reqHead := fmt.Sprintf("%s %s %s\n%s", f.Request.Method, target, f.Request.Proto, requestHeaderLines(f.Request)) + reqHead := requestHead(rq, target) status, ct, respHead := 101, "websocket", "HTTP/1.1 101 Switching Protocols\n" if kind == "sse" { - status = f.Response.StatusCode - ct = f.Response.Header.Get("Content-Type") - respHead = fmt.Sprintf("HTTP %d\n%s", status, headerLines(f.Response.Header)) + status = rs.status + ct = rs.header.Get("Content-Type") + respHead = fmt.Sprintf("HTTP %d\n%s", status, headerLines(rs.header)) } t.wmu.Lock() defer t.wmu.Unlock() - reqB := t.spill(f.Request.Body, f.Request.Header.Get("Content-Type")) + reqB := t.spill(rq.body, rq.header.Get("Content-Type")) + trace := t.spillTrace(lines) tx, err := t.db.Begin() if err != nil { - return "", fmt.Errorf("beginning transaction: %w", err) + return "", &recordError{"beginning transaction", err} } defer tx.Rollback() //nolint:errcheck // no-op once committed + // An id can repeat (seconds plus a rolling counter); the trace of the + // exchange it replaces must not stay attached to this one. + if _, err := tx.Exec(`DELETE FROM exchange_rewrites WHERE exchange_id=?`, id); err != nil { + return "", &recordError{"clearing rewrite trace", err} + } res, err := tx.Exec(`INSERT OR REPLACE INTO exchanges(id,ts,host,method,url_template,url,status,content_type,req_len,resp_len,path) -VALUES(?,?,?,?,?,?,?,?,?,?,'')`, id, now.Unix(), host, method, tmpl, display, status, ct, len(f.Request.Body), 0) +VALUES(?,?,?,?,?,?,?,?,?,?,'')`, id, now.Unix(), host, method, tmpl, display, status, ct, len(rq.body), 0) if err != nil { - return "", fmt.Errorf("writing index: %w", err) + return "", &recordError{"writing index", err} } rowid, err := res.LastInsertId() if err != nil { - return "", fmt.Errorf("fetching rowid: %w", err) + return "", &recordError{"fetching rowid", err} } if _, err := tx.Exec(`INSERT OR REPLACE INTO exchange_bodies(id,req_head,req_body,req_blob,resp_head,resp_body,resp_blob) VALUES(?,?,?,?,?,NULL,NULL)`, id, reqHead, reqB.inline, nullIfEmpty(reqB.hash), respHead); err != nil { - return "", fmt.Errorf("writing body: %w", err) + return "", &recordError{"writing body", err} } if reqB.hash != "" { if _, err := tx.Exec(`INSERT OR IGNORE INTO blob_refs(hash,exchange_id) VALUES(?,?)`, reqB.hash, id); err != nil { - return "", fmt.Errorf("registering blob reference: %w", err) + return "", &recordError{"registering blob reference", err} } } + if err := writeTrace(tx, id, lines, trace); err != nil { + return "", err + } if t.fts { idx := strings.Join([]string{display, reqHead, reqB.index, respHead}, "\n") if _, err := tx.Exec(`INSERT INTO ex_fts(rowid,content) VALUES(?,?)`, rowid, idx); err != nil { - return "", fmt.Errorf("writing full-text index: %w", err) + return "", &recordError{"writing full-text index", err} } } if err := tx.Commit(); err != nil { - return "", fmt.Errorf("committing: %w", err) + return "", &recordError{"committing", err} } return id, nil } diff --git a/traffic/traffic.go b/traffic/traffic.go index 17b1c05..3ace802 100644 --- a/traffic/traffic.go +++ b/traffic/traffic.go @@ -22,6 +22,7 @@ import ( "os" "path/filepath" "regexp" + "slices" "sort" "strconv" "strings" @@ -92,6 +93,26 @@ CREATE TABLE IF NOT EXISTS stream_messages ( PRIMARY KEY (exchange_id, seq) ); CREATE INDEX IF NOT EXISTS idx_stream_messages_ex ON stream_messages(exchange_id, seq); + +-- Rewrite-rule trace of an exchange, in order. The exchange row keeps what the +-- client sent and what the target answered; these rows say what the rules +-- changed on the way. An exchange is rewritten when it has an 'applied' row. +CREATE TABLE IF NOT EXISTS exchange_rewrites ( + exchange_id TEXT NOT NULL, + seq INTEGER NOT NULL, + phase TEXT NOT NULL, -- 'request' | 'response' + rule_id INTEGER, -- NULL on a global line + rule_snapshot TEXT, -- JSON, no sensitive action value + action TEXT, + field TEXT, -- 'method' | 'url' | 'header:' | 'body' | 'status' + outcome TEXT NOT NULL, -- 'applied' | 'skipped' | 'error' + reason TEXT NOT NULL DEFAULT '', + before_hash TEXT, + after_hash TEXT, + after_inline BLOB, + after_blob TEXT, + PRIMARY KEY (exchange_id, seq) +); ` // ftsSchema is applied separately from indexSchema: a driver build without FTS5 @@ -160,6 +181,10 @@ const ( // and the catalog migration cannot drift apart. const TrafficSearchDescription = "Query the target traffic the recording proxy has captured (host is required; accepts a bare host, host:port, or full URL, optionally filtered further by URL substring or body keyword). When a port is specified, only that service's traffic is returned, avoiding cross-talk between different ports on the same IP. body_contains runs a full-text search over the captured request/response headers and bodies, supporting any substring including Chinese (at least 3 characters). Returns only an ultra-light index (id/method/url/status/resp_len) with no response content; once results are non-empty, use traffic_get to verify the request/response of each one, then hand the IDs that actually support the current vulnerability to bind_finding_traffic. Returns 3 by default, at most 10 per page; use page to paginate when there are more results." +// TrafficGetDescription is the traffic_get description, kept here for the same +// reason as TrafficSearchDescription. +const TrafficGetDescription = "Fetch the raw request/response of one captured traffic record by id (truncated when too large). Use together with traffic_search to avoid re-curling. A WebSocket (status 101) or SSE (text/event-stream) record shows only its handshake here; read its messages with traffic_messages. When proxy rewrite rules changed the exchange, the result starts with a REWRITTEN line naming the rule ids, and shows the request as the client sent it and the response as the target sent it. traffic_search searches only these original values: it never finds a value a rewrite rule wrote." + // Traffic runs the recording proxy and owns the file tree + index. type Traffic struct { dir string @@ -199,8 +224,14 @@ type Traffic struct { // so no host escapes it. Hot-swappable at runtime via SetUpstreamProxy. upstream atomic.Pointer[url.URL] // rewriteRules is the compiled snapshot of the rewrite rules, replaced as a - // whole by SetRewriteRules. The proxy does not apply it yet. + // whole by SetRewriteRules and applied by the sink (rewrite_flow.go). rewriteRules atomic.Pointer[rewrite.Set] + // snaps keeps the original of each request a rule rewrote until its + // exchange is recorded (rewrite_flow.go). Nil on the zero value. + snaps *snapTable + // sweeping tracks the snapshot sweeper, which runs until Close. It is kept + // apart from reaping, which only ever holds work that finishes on its own. + sweeping sync.WaitGroup } // Open initializes the traffic tree, blob store and SQLite index under dir. @@ -226,7 +257,7 @@ func Open(dir, addr string) (*Traffic, error) { if err != nil { return nil, err } - t := &Traffic{dir: dir, addr: addr, db: db, closed: make(chan struct{})} + t := &Traffic{dir: dir, addr: addr, db: db, closed: make(chan struct{}), snaps: newSnapTable()} if err := t.initIndex(); err != nil { db.Close() return nil, err @@ -258,6 +289,7 @@ func Open(dir, addr string) (*Traffic, error) { }) p.AddAddon(&sink{t: t}) t.proxy = p + t.sweeping.Go(t.sweepSnapshots) // stops on t.closed; Close waits for it return t, nil } @@ -358,8 +390,8 @@ func (t *Traffic) SetUpstreamProxy(raw string) error { } // SetRewriteRules publishes a compiled snapshot of the rewrite rules. The swap -// is atomic; nil clears the rules. The snapshot is stored only: the proxy does -// not apply rewrite rules yet, so capture is unchanged whatever it holds. +// is atomic; nil clears the rules. A flow already rewritten keeps the snapshot +// its request saw for its response as well. func (t *Traffic) SetRewriteRules(s *rewrite.Set) { t.rewriteRules.Store(s) } // RewriteRules returns the snapshot last published, nil when there is none. @@ -406,6 +438,7 @@ func (t *Traffic) Close() error { t.closeOnce.Do(func() { close(t.closed) }) } t.reaping.Wait() + t.sweeping.Wait() return t.db.Close() } @@ -430,21 +463,6 @@ type sink struct { streams sync.Map // *mproxy.Flow of a live WebSocket/SSE stream -> *streamState (stream.go) } -func (s *sink) Response(f *mproxy.Flow) { - if f.Request == nil || f.Response == nil { - return - } - s.t.record(f) -} - -// RequestError fires when a request through an established MITM tunnel fails. If -// the failure looks proxy/protocol-caused (h2 quirks, HEAD-with-body, protocol -// errors) and not a plain target-unreachable error, the host is tunneled for -// passthroughTTL. This request has already failed. Later requests in the window -// connect straight through; the first request after the window tries -// interception again. -func (s *sink) RequestError(f *mproxy.Flow, err error) { s.t.maybePassthrough(f, err) } - // maybePassthrough marks a host to be tunneled transparently on the next // connection, but only for errors the proxy itself caused — a target that is // simply down/filtered would fail without us too, and must stay MITM'd+recorded. @@ -502,52 +520,121 @@ func proxyCausedErr(err error) bool { // full-text index. Nothing is written to a per-request directory — only bodies // above maxInlineBody spill to the content-addressed blob store. func (t *Traffic) record(f *mproxy.Flow) { + t.recordViews(flowRequestView(f.Request), flowResponseView(f.Response), nil) +} + +// requestView is the request an exchange records: what the client sent, which +// is not the flow's request once a rule rewrote it. +type requestView struct { + method string + url *url.URL // nil when the original request is unavailable and Raw() is nil + proto string + header http.Header + // rawHost is the client's Host header (Raw().Host), "" when unknown. + rawHost string + body []byte + // unavailable marks a request rewritten by a rule whose original was not + // kept: only its method, URL and Host are known. + unavailable bool +} + +// responseView is the response an exchange records: what the target answered. +type responseView struct { + status int + header http.Header + body []byte +} + +func flowRequestView(r *mproxy.Request) requestView { + v := requestView{method: r.Method, url: r.URL, proto: r.Proto, header: r.Header, body: r.Body} + if raw := r.Raw(); raw != nil { + v.rawHost = raw.Host + } + return v +} + +func flowResponseView(r *mproxy.Response) responseView { + return responseView{status: r.StatusCode, header: r.Header, body: r.Body} +} + +// unavailableMarker stands for the headers and body of a request whose +// original was not kept (see rewrite_flow.go). +const unavailableMarker = "[original request unavailable]\n" + +// recordViews writes one exchange from explicit views, with its rewrite trace, +// in one transaction: the exchange row, its bodies, blob references, the trace +// and the full-text index of the views only. A failure rolls everything back, +// so there is never a rewritten exchange without its trace. Without trace +// lines this is exactly what record always wrote. +func (t *Traffic) recordViews(rq requestView, rs responseView, lines []rewrite.TraceLine) { // The write lock covers blob + index writes, so DeleteHost (and its blob GC) // can run under the same lock without racing a concurrent record. t.wmu.Lock() defer t.wmu.Unlock() - host := f.Request.URL.Hostname() - method := f.Request.Method - tmpl := db.TemplatePath(f.Request.URL.EscapedPath()) + var host, tmpl, url, uri string + if rq.url != nil { + host = rq.url.Hostname() + tmpl = db.TemplatePath(rq.url.EscapedPath()) + url = rq.url.String() + uri = rq.url.RequestURI() + } + method := rq.method now := time.Now() id := t.newExchangeID(now) - ct := f.Response.Header.Get("Content-Type") - url := f.Request.URL.String() - reqHead := fmt.Sprintf("%s %s %s\n%s", method, f.Request.URL.RequestURI(), f.Request.Proto, requestHeaderLines(f.Request)) - respHead := fmt.Sprintf("HTTP %d\n%s", f.Response.StatusCode, headerLines(f.Response.Header)) + ct := rs.header.Get("Content-Type") + reqHead := requestHead(rq, uri) + respHead := fmt.Sprintf("HTTP %d\n%s", rs.status, headerLines(rs.header)) // Bodies are spilled before the transaction opens: blob writes are filesystem // work and must not sit inside the SQLite write lock. - reqB := t.spill(f.Request.Body, f.Request.Header.Get("Content-Type")) - respB := t.spill(f.Response.Body, ct) + reqB := t.spill(rq.body, rq.header.Get("Content-Type")) + respB := t.spill(rs.body, ct) + trace := t.spillTrace(lines) + + // A rewritten exchange is logged by host, step and rule ids only: its URL + // and the error text may carry a value a rule wrote. + rewritten, ruleIDs := appliedRules(lines) + fail := func(step string, err error) { + if rewritten { + log.Printf("[traffic] failed to record a rewritten exchange of %s (%s; rules %v)", host, step, ruleIDs) + return + } + log.Printf("[traffic] failed to record %s (%s): %v", url, step, err) + } tx, err := t.db.Begin() if err != nil { - log.Printf("[traffic] failed to record %s (beginning transaction): %v", url, err) + fail("beginning transaction", err) return } defer tx.Rollback() //nolint:errcheck // no-op once committed + // An id can repeat (seconds plus a rolling counter); the trace of the + // exchange it replaces must not stay attached to this one. + if _, err := tx.Exec(`DELETE FROM exchange_rewrites WHERE exchange_id=?`, id); err != nil { + fail("clearing rewrite trace", err) + return + } // path stays empty for db-resident exchanges; a non-empty path marks a legacy // row whose bodies still live in the old on-disk tree (see Get). res, err := tx.Exec(`INSERT OR REPLACE INTO exchanges(id,ts,host,method,url_template,url,status,content_type,req_len,resp_len,path) VALUES(?,?,?,?,?,?,?,?,?,?,'')`, - id, now.Unix(), host, method, tmpl, url, f.Response.StatusCode, ct, - len(f.Request.Body), len(f.Response.Body)) + id, now.Unix(), host, method, tmpl, url, rs.status, ct, + len(rq.body), len(rs.body)) if err != nil { - log.Printf("[traffic] failed to record %s (writing index): %v", url, err) + fail("writing index", err) return } rowid, err := res.LastInsertId() if err != nil { - log.Printf("[traffic] failed to record %s (fetching rowid): %v", url, err) + fail("fetching rowid", err) return } if _, err := tx.Exec(`INSERT OR REPLACE INTO exchange_bodies(id,req_head,req_body,req_blob,resp_head,resp_body,resp_blob) VALUES(?,?,?,?,?,?,?)`, id, reqHead, reqB.inline, nullIfEmpty(reqB.hash), respHead, respB.inline, nullIfEmpty(respB.hash)); err != nil { - log.Printf("[traffic] failed to record %s (writing body): %v", url, err) + fail("writing body", err) return } @@ -556,26 +643,51 @@ VALUES(?,?,?,?,?,?,?)`, continue } if _, err := tx.Exec(`INSERT OR IGNORE INTO blob_refs(hash,exchange_id) VALUES(?,?)`, h, id); err != nil { - log.Printf("[traffic] failed to record %s (registering blob reference): %v", url, err) + fail("registering blob reference", err) return } } + if err := writeTrace(tx, id, lines, trace); err != nil { + var re *recordError + errors.As(err, &re) + fail(re.step, re.err) + return + } + if t.fts { // The index is fed from memory, so a body that spilled to _blobs is still - // fully searchable even though only its preview is stored inline. + // fully searchable even though only its preview is stored inline. Only the + // recorded views are indexed, never a value a rule wrote. idx := strings.Join([]string{url, reqHead, reqB.index, respHead, respB.index}, "\n") if _, err := tx.Exec(`INSERT INTO ex_fts(rowid,content) VALUES(?,?)`, rowid, idx); err != nil { - log.Printf("[traffic] failed to record %s (writing full-text index): %v", url, err) + fail("writing full-text index", err) return } } if err := tx.Commit(); err != nil { - log.Printf("[traffic] failed to record %s (committing): %v", url, err) + fail("committing", err) } } +// appliedRules reports whether a trace rewrote its exchange, with the ids of +// the rules that did, in order of first appearance. +func appliedRules(lines []rewrite.TraceLine) (bool, []int64) { + rewritten := false + var ids []int64 + for _, l := range lines { + if l.Outcome != rewrite.OutcomeApplied { + continue + } + rewritten = true + if l.RuleID != 0 && !slices.Contains(ids, l.RuleID) { + ids = append(ids, l.RuleID) + } + } + return rewritten, ids +} + // newExchangeID names a recorded exchange: unix seconds plus a rolling sequence. func (t *Traffic) newExchangeID(now time.Time) string { n := t.seq.Add(1) @@ -726,15 +838,35 @@ func headerLines(h map[string][]string) string { return b.String() } -// requestHeaderLines restores the HTTP Host header, which net/http stores on -// Request.Host rather than in Header. Keeping it in request.http makes the raw -// capture complete and directly replayable. +// requestHeaderLines is requestHeaderLinesOf for a flow's request. func requestHeaderLines(req *mproxy.Request) string { - headers := req.Header.Clone() + v := flowRequestView(req) + return requestHeaderLinesOf(v.header, req.URL.Host, v.rawHost) +} + +// requestHead is the head an exchange records for a request view: request +// line and headers, or only the Host when the original request is unavailable. +func requestHead(rq requestView, target string) string { + urlHost := "" + if rq.url != nil { + urlHost = rq.url.Host + } + if rq.unavailable { + return fmt.Sprintf("%s %s %s\n%s%s", rq.method, target, rq.proto, requestHeaderLinesOf(nil, urlHost, rq.rawHost), unavailableMarker) + } + return fmt.Sprintf("%s %s %s\n%s", rq.method, target, rq.proto, requestHeaderLinesOf(rq.header, urlHost, rq.rawHost)) +} + +// requestHeaderLinesOf restores the HTTP Host header, which net/http stores on +// Request.Host rather than in Header. Keeping it in request.http makes the raw +// capture complete and directly replayable. rawHost is the client's Host +// header (Raw().Host); urlHost stands in when it is empty. +func requestHeaderLinesOf(header http.Header, urlHost, rawHost string) string { + headers := header.Clone() headers.Del("Host") - host := req.URL.Host - if raw := req.Raw(); raw != nil && strings.TrimSpace(raw.Host) != "" { - host = raw.Host + host := urlHost + if strings.TrimSpace(rawHost) != "" { + host = rawHost } var b strings.Builder if host = strings.TrimSpace(host); host != "" { @@ -1271,6 +1403,9 @@ func (t *Traffic) deleteWhere(tx *sql.Tx, where string, args ...any) (int64, err if _, err := tx.Exec(`DELETE FROM stream_messages WHERE exchange_id IN (SELECT id FROM exchanges WHERE `+where+`)`, args...); err != nil { return 0, fmt.Errorf("delete stream messages: %w", err) } + if _, err := tx.Exec(`DELETE FROM exchange_rewrites WHERE exchange_id IN (SELECT id FROM exchanges WHERE `+where+`)`, args...); err != nil { + return 0, fmt.Errorf("delete rewrite trace: %w", err) + } if _, err := tx.Exec(`DELETE FROM blob_refs WHERE exchange_id IN (SELECT id FROM exchanges WHERE `+where+`)`, args...); err != nil { return 0, fmt.Errorf("delete blob references: %w", err) } @@ -2138,7 +2273,7 @@ func (t *Traffic) Tools() []actool.CoreTool { get := actool.Build(actool.Spec{ Name: "traffic_get", - Description: "Fetch the raw request/response of one captured traffic record by id (truncated when too large). Use together with traffic_search to avoid re-curling. A WebSocket (status 101) or SSE (text/event-stream) record shows only its handshake here; read its messages with traffic_messages.", + Description: TrafficGetDescription, Schema: map[string]any{ "type": "object", "properties": map[string]any{"id": map[string]any{"type": "string", "description": "the id returned by traffic_search"}}, @@ -2153,7 +2288,15 @@ func (t *Traffic) Tools() []actool.CoreTool { if err != nil { return actool.Errorf(err.Error()), nil } - return actool.Text("=== REQUEST ===\n" + clip(req, 2500) + "\n\n=== RESPONSE ===\n" + clip(resp, 4000)), nil + rewritten, ruleIDs, err := t.RewriteInfo(a.ID) + if err != nil { + return actool.Errorf(err.Error()), nil + } + head := "" + if rewritten { + head = fmt.Sprintf("=== REWRITTEN === rewrite rules %v changed this exchange in the proxy. Shown below: the request as the client sent it and the response as the target sent it, not the rewritten values.\n\n", ruleIDs) + } + return actool.Text(head + "=== REQUEST ===\n" + clip(req, 2500) + "\n\n=== RESPONSE ===\n" + clip(resp, 4000)), nil }, })