Compare commits

..
Author SHA1 Message Date
BaaRai eddf03ce61 Merge pull request #304 from Protonmatter/protonmatter/llmpool-health-queue
Coalesce circuit-breaker health writes
2026-11-17 22:52:27 +01:00
BaaRai 9bc5856565 Merge pull request #302 from Protonmatter/protonmatter/budget-restore
Restore task spend from the lanes the live ledger counts
2026-11-17 22:52:23 +01:00
protonmatter c9d81a453f Align the Server literal so gofmt accepts it. 2026-11-17 22:39:53 +01:00
protonmatter b2a2f95e8c Release the circuit-breaker queue lock around the database write. 2026-11-17 22:39:53 +01:00
protonmatter 1fbf2263a1 Keep circuit-breaker writes in state order and flush them before the database closes.
A later update can no longer be queued behind an older one, a failed save is retried, and shutdown waits for the last cooling window.
2026-11-17 22:39:53 +01:00
protonmatter 847e32407a Coalesce circuit-breaker writes onto one queue.
A newer state for the same profile replaces one that has not been written yet, and a failed write is logged instead of discarded on a detached goroutine.
2026-11-17 22:39:50 +01:00
BaaRai 7bd9bf6e3e Merge pull request #309 from BaaRai/feat/traffic-rewrite-p33b1
feat(traffic): apply rewrite rules in the capture proxy (P3.3b1)
2026-11-17 22:38:59 +01:00
BaaRai f8bc3b3b6d Merge pull request #305 from Protonmatter/protonmatter/json-body-cap
Cap JSON request bodies
2026-11-17 22:29:08 +01:00
BaaRai 4fb29fd841 Merge pull request #303 from Protonmatter/protonmatter/coverage-flag-error
Keep coverage tools available when the flag read fails
2026-11-17 22:29:04 +01:00
protonmatter 074962a351 Say that a revived task keeps the spend already recorded. 2026-11-17 16:25:16 -05:00
BaaRai 5fdf8eb075 fix(server): keep the MaxBytesError so a caller's 413 mapping still works
A handler that installs its own MaxBytesReader and then calls decode lost the 413: decode converted the error to a plain one and the middleware's limitedBody had been replaced. Return the original *http.MaxBytesError and update the cap test to errors.As.
2026-11-17 22:19:59 +01:00
BaaRai d646f6999b fix(test): check each coverage tool on the set that carries it
list_untested_assets is a planner tool, not a worker tool; the fail-open assertion can only be made against PlannerTools.
2026-11-17 22:18:54 +01:00
protonmatter 186594ed3a Cap every non-multipart body and answer an oversized one with 413.
Handlers that decode JSON directly were still unbounded, and a body past the cap was reported as a bad request.
2026-11-17 15:57:43 -05:00
protonmatter 133dc68668 Test that a failed coverage read stays enabled, including auto-scope writes.
A missing task and a query error both leave the flag on. The worker keeps the coverage tools and still records auto scope when the stored flag is off.
2026-11-17 15:52:04 -05:00
protonmatter 40de6ba898 Say what a restored task budget still allows, and test the lanes that count.
A task already inside the wrap-up reserve only has that reserve left. The worker estimate starts from the defaults because usage rows have no run boundary.
2026-11-17 15:50:00 -05:00
protonmatter 69bdd94d13 Cap JSON request bodies and bound how long a body may arrive.
decode stops at 32 MiB. ReadTimeout limits header and body arrival without cutting a handler that is already running.
2026-11-17 13:04:50 -05:00
protonmatter ce218c4b16 Leave asset-coverage tools available when the flag read fails.
A missing task stays enabled. A database error is returned to the caller and logged, instead of being treated as a reason to hide the tools.
2026-11-17 13:04:36 -05:00
protonmatter be25f14970 Restore a task spend cap from the lanes the live ledger counts.
Planner, worker, judge, and compactor rows rebuild the cap. Main-agent chat, goal decomposition, and side questions stay out, so a restart does not count turns the running process never metered.
2026-11-17 13:04:36 -05:00
38 changed files with 1069 additions and 117 deletions
+1
View File
@@ -28,6 +28,7 @@ All notable changes to this project are documented in this file. The format is b
### Changed
- **Task budgets survive a restart**: planner, worker, judge, and compactor rows in `llm_usage` are counted back into `ARTEX_TASK_BUDGET_*`. A restored task already inside the wrap-up reserve only has that reserve left, so the run wraps up instead of starting another worker. The per-worker estimate is not restored and starts from the defaults (50,000 tokens and 10 turns) until a worker finishes in this process. Chat, goal decomposition, and side questions stay outside the cap.
- **Untrusted-output boundary is now on by default**: Norma frames tool results for the worker, planner, chat and main agent unless `ARTEX_UNTRUSTED_BOUNDARY` is set to `0`, `false`, `off` or `no`. It was opt-in via `ARTEX_UNTRUSTED_BOUNDARY=1` in 0.6.0 (#166); the default flips here (#203).
- **Versioned migrations**: the `db` package applies numbered `db/migrations/NNNN_slug.sql` files once, in order, under the schema advisory lock, and records a SHA-256 checksum in `schema_migrations`; missing, renamed, changed or out-of-order migrations fail closed (#226).
- **Coverage graph performance**: the coverage graph batches its connector lookups (three queries instead of one per root and per company) and adds the partial index `idx_ts_icp`; on a task of about 3,000 assets the p50 dropped from 126 ms to 27 ms and the queries per call from 221 to 3 (#230).
+11 -2
View File
@@ -255,7 +255,10 @@ worker remains confined to the intent it claimed.
- **Budgets**: `ARTEX_TASK_BUDGET_TOKENS` / `ARTEX_TASK_BUDGET_TURNS` cap a whole task (planner and
workers); `ARTEX_INTENT_BUDGET_TOKENS` / `ARTEX_INTENT_BUDGET_TURNS` cap one worker run. Near the
limit the task enters a forced wrap-up; a worker is not started when the remaining budget cannot
cover it. Budgets are in memory: a restart starts fresh.
cover it. Spend already in `llm_usage` is restored after a restart. A task that is already inside
the wrap-up reserve only has that reserve left, so the revived run wraps up instead of starting
another worker. The per-worker cost estimate starts again from the defaults (50,000 tokens and
10 turns) until a worker finishes in this process.
- **Noise governor**: `ARTEX_NOISE_PROFILE=quiet|balanced|aggressive` paces requests ARTEX sends to
targets (per-host rate, burst, in-flight limit, jitter, backoff on 429/503, short GET caching).
With a profile set, agents also receive low-and-slow flags for the installed scanners.
@@ -455,7 +458,13 @@ workers, else 50,000 tokens and 10 turns). Near the wrap-up reserve (the larger
tokens, and 2 turns) the task enters a forced wrap-up: the planner writes a summary and final goal
judgment, and workers may only write back. The task then ends `done` if its goals are met, else
`timeout`. Practical floor: below 50,000 tokens no worker starts; 50,000-70,000 typically gives one
worker then wrap-up; pick well above 70,000 for real exploration.
worker then wrap-up; pick well above 70,000 for real exploration. Planner, worker, judge, and
compactor rows in `llm_usage` are counted back into the cap when a ledger is created, including
after a restart and after a revived run drops its ledger. A restored task already inside the
wrap-up reserve does not get another worker: only that reserve remains, and the run wraps up.
Chat, goal decomposition, and side questions are stored on the same task and are not counted.
The per-worker average is not restored (usage rows have no run boundary), so the estimate starts
from the defaults until a worker finishes in this process.
**Common flags**: `./start.sh -addr :8787 -proxy :8788` (`-addr` frontend/API, `-proxy` traffic
recording proxy).
+17 -1
View File
@@ -92,7 +92,8 @@ func BudgetsFromEnv() (task, intent Budget) {
NewBudget(env("ARTEX_INTENT_BUDGET_TOKENS"), env("ARTEX_INTENT_BUDGET_TURNS"))
}
// BudgetLedger meters one task, in memory (a process restart starts afresh).
// BudgetLedger meters one task. The total is restored from llm_usage when the
// ledger is created, including after a restart. Per-worker averages are not.
type BudgetLedger struct {
task, intent Budget
mu sync.Mutex
@@ -110,6 +111,21 @@ func NewBudgetLedger(task, intent Budget) *BudgetLedger {
return &BudgetLedger{task: task, intent: intent}
}
// RestoreSpend adds usage that was already recorded before this ledger existed.
// It only adds to the task total. Per-worker averages stay at the defaults
// until a worker finishes against this ledger, because llm_usage rows do not
// mark where one worker run ends and the next begins. A nil ledger (no budget
// configured) is a no-op.
func (l *BudgetLedger) RestoreSpend(spent Spend) {
if l == nil || (spent.Tokens == 0 && spent.Turns == 0) {
return
}
l.mu.Lock()
l.spent.Tokens += spent.Tokens
l.spent.Turns += spent.Turns
l.mu.Unlock()
}
// Record meters one task-level call (the planner's own rounds).
func (l *BudgetLedger) Record(u llm.Usage) {
l.mu.Lock()
+11
View File
@@ -144,3 +144,14 @@ func TestBudgetCountsMatchUsageFixture(t *testing.T) {
t.Fatalf("spent %+v remaining %+v estimate %+v, want {%d 5}, remaining, {10400 3}", st.Spent, st.Remaining, st.Estimate, want)
}
}
func TestRestoreSpendCountsTowardCap(t *testing.T) {
l := NewBudgetLedger(NewBudget(100, 10), NewBudget(0, 0))
l.RestoreSpend(Spend{Tokens: 40, Turns: 4})
st := l.State()
if st.Spent != (Spend{40, 4}) || st.Remaining != (Budget{60, 6}) {
t.Fatalf("spent=%+v remaining=%+v", st.Spent, st.Remaining)
}
var none *BudgetLedger
none.RestoreSpend(Spend{Tokens: 1, Turns: 1})
}
+22
View File
@@ -0,0 +1,22 @@
package agent
import (
"log"
"github.com/Autumn-27/artex/db"
)
// coverageOn reports whether asset-coverage tools stay available for this task.
// A nil store is the historical default (enabled). A database error is logged and
// treated as enabled so a transient read failure does not drop the tools mid-run.
func coverageOn(as *db.AssetStore, taskID int64) bool {
if as == nil {
return true
}
enabled, err := as.CoverageEnabled(taskID)
if err != nil {
log.Printf("[coverage] task %d: reading the coverage flag failed; leaving coverage on: %v", taskID, err)
return true
}
return enabled
}
+113
View File
@@ -0,0 +1,113 @@
package agent
import (
"bytes"
"fmt"
"log"
"strings"
"testing"
"time"
"github.com/Autumn-27/artex/db"
)
// A failed coverage-flag read stays enabled: the log says so, the worker keeps
// record_coverage and list_untested_assets, and AddAutoScope still writes. That
// write happens even when the stored flag is off, which is the outage behaviour.
func TestCoverageReadFailureKeepsToolsAndWritesAutoScope(t *testing.T) {
d := testDB(t)
defer d.Close()
off := false
task, err := d.CreateTaskWithOptions("coverage outage", "goal", db.TaskCreateOptions{CoverageEnabled: &off})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = d.DeleteTask(task.ID) })
live := d.Assets()
stored, err := live.CoverageEnabled(task.ID)
if err != nil || stored {
t.Fatalf("stored flag enabled=%v err=%v, want off", stored, err)
}
if coverageOn(live, task.ID) {
t.Fatal("a readable off flag was treated as on")
}
dsn, _, err := db.DSN()
if err != nil {
t.Fatal(err)
}
broken, err := db.Open(dsn)
if err != nil {
t.Fatal(err)
}
if err := broken.Close(); err != nil {
t.Fatal(err)
}
var logged bytes.Buffer
prev := log.Default().Writer()
log.SetOutput(&logged)
t.Cleanup(func() { log.SetOutput(prev) })
if !coverageOn(broken.Assets(), task.ID) {
t.Fatal("a failed coverage read turned the feature off")
}
if !strings.Contains(logged.String(), "leaving coverage on") {
t.Fatalf("coverage read failure was not logged: %q", logged.String())
}
// record_coverage is the worker's coverage tool; list_untested_assets is the
// planner's, so each is checked on the tool set that actually carries it.
hasPlannerTool := func(ts *ToolSet, name string) bool {
for _, tool := range ts.DropCoverageTools(ts.PlannerTools()) {
if tool.Name() == name {
return true
}
}
return false
}
offWorker := NewToolSet(nil, "worker-off")
offWorker.SetCoverageEnabled(false)
if hasTool(offWorker, "record_coverage") {
t.Fatal("record_coverage stayed available to a worker with coverage off")
}
offPlanner := NewToolSet(nil, "planner-off")
offPlanner.SetCoverageEnabled(false)
if hasPlannerTool(offPlanner, "list_untested_assets") {
t.Fatal("list_untested_assets stayed available to the planner with coverage off")
}
outage := coverageOn(broken.Assets(), task.ID)
outWorker := NewToolSet(nil, "worker-outage")
outWorker.SetCoverageEnabled(outage)
if !hasTool(outWorker, "record_coverage") {
t.Fatal("record_coverage dropped from the worker when the coverage read failed")
}
outPlanner := NewToolSet(nil, "planner-outage")
outPlanner.SetCoverageEnabled(outage)
if !hasPlannerTool(outPlanner, "list_untested_assets") {
t.Fatal("list_untested_assets dropped from the planner when the coverage read failed")
}
keptOff := fmt.Sprintf("off-%d.invalid", time.Now().UnixNano())
writeIntentAutoScope(live, task.ID, []*db.Asset{{Type: "root_domain", Domain: keptOff}}, false)
wrote := fmt.Sprintf("outage-%d.invalid", time.Now().UnixNano())
writeIntentAutoScope(live, task.ID, []*db.Asset{{Type: "root_domain", Domain: wrote}}, coverageOn(broken.Assets(), task.ID))
scope, err := live.ListTaskScope(task.ID)
if err != nil {
t.Fatal(err)
}
var sawKept, sawWrote bool
for _, row := range scope {
if row.Domain == keptOff {
sawKept = true
}
if row.Domain == wrote {
sawWrote = true
}
}
if sawKept {
t.Fatal("auto scope was written while coverage was off")
}
if !sawWrote {
t.Fatal("auto scope was not written when the coverage read failed")
}
}
+1 -1
View File
@@ -118,7 +118,7 @@ func (m *MainAgent) Chat(ctx context.Context, taskID int64, mainSeg int, as *db.
tsx.SetAssetStore(as, as.Companies())
}
tsx.SetTaskID(taskID)
tsx.SetCoverageEnabled(as == nil || as.CoverageEnabled(taskID))
tsx.SetCoverageEnabled(coverageOn(as, taskID))
tsx.SetNotify(notify) // generic wake-up (write ops without a dedicated callback go through it, debounced)
tsx.SetResumeTask(resume) // set_goals adds a goal → pull a completed/paused task back to running
tsx.SetNotifyGoal(notifyGoal) // set_goals adds a goal → record a "human added a goal: …" trigger for the planner
+1 -1
View File
@@ -473,7 +473,7 @@ func (p *Planner) Plan(ctx context.Context, taskID int64, as *db.AssetStore, ts
tsx.SetAssetStore(as, as.Companies())
}
tsx.SetTaskID(taskID)
tsx.SetCoverageEnabled(as == nil || as.CoverageEnabled(taskID))
tsx.SetCoverageEnabled(coverageOn(as, taskID))
tsx.killWork = p.killWork // enable kill_work tool (nil = unavailable)
tsx.steerWork = p.steerWork // enable steer_work tool (nil = unavailable)
if origin, _ := ts.OriginFactID(); origin > 0 {
+19 -7
View File
@@ -387,6 +387,21 @@ func renderWorkerGraphOverview(data map[string]any, env dataEnvelope) string {
sections
}
// writeIntentAutoScope records the intent's target assets as auto scope when
// coverage is on. coverageEnabled is fail-open: a failed flag read is true, so
// an outage still writes, including when the stored flag is off.
func writeIntentAutoScope(as *db.AssetStore, taskID int64, assets []*db.Asset, coverageEnabled bool) {
if as == nil || !coverageEnabled {
return
}
for _, a := range assets {
if a == nil {
continue
}
_ = as.AddAutoScope(taskID, a.Type, a.Domain, a.URL, a.IP)
}
}
// Execute runs one intent. hooks (the per-task Guard) gates every tool call; may
// be nil. emit, if non-nil, receives one ActivityRecord per execution step.
// notifyFinding, if non-nil, is called (intentID, summary) when this worker writes
@@ -412,7 +427,7 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
tsx := NewToolSet(ts, name)
tsx.SetFindingRecorder(w.findingRecorder)
tsx.SetTaskID(taskID)
coverageEnabled := as == nil || as.CoverageEnabled(taskID)
coverageEnabled := coverageOn(as, taskID)
tsx.SetCoverageEnabled(coverageEnabled)
if as != nil {
tsx.SetAssetStore(as, as.Companies())
@@ -473,12 +488,9 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
// toward coverage only under a declared domain or IP root (db.declaredScopeFilter);
// coverage scope, not enforced.
// When the asset-coverage feature is off, the test scope (denominator) is no
// longer accumulated.
if coverageEnabled {
for _, a := range assets {
_ = as.AddAutoScope(taskID, a.Type, a.Domain, a.URL, a.IP)
}
}
// longer accumulated. A failed flag read is treated as on, so an outage
// still writes, including for a task whose stored flag is off.
writeIntentAutoScope(as, taskID, assets, coverageEnabled)
}
}
}
+8
View File
@@ -110,10 +110,15 @@ func run() int {
log.Printf("[config] trusted proxies: %v", trusted)
}
srv := server.New(ctx, mgr, skillDir, *dataDir, keyDir, server.WithSecretScrub(), server.WithTrustedProxies(trusted))
// ReadTimeout bounds how long the request headers and body may take to arrive.
// It does not limit how long a handler runs, so an event stream stays open.
// WriteTimeout is left unset: a write deadline would cut that stream.
httpSrv := &http.Server{
Addr: *addr,
Handler: srv.Handler(),
ReadHeaderTimeout: 10 * time.Second,
ReadTimeout: 30 * time.Minute,
IdleTimeout: 2 * time.Minute,
}
go func() {
@@ -142,6 +147,9 @@ func run() int {
if drainErr != nil {
log.Printf("task drain: %v", drainErr)
}
// The health writer flushes on ctx.Done(). Wait for it before the deferred
// store close, so the last cooling window is not written at a closed database.
srv.FlushLLMHealth()
if httpErr != nil || drainErr != nil {
return 1
}
+18
View File
@@ -33,6 +33,24 @@ func (d *DB) EnsureLLMUsageTable() error {
return d.execEmbeddedMigration(llmUsageMigration)
}
// TaskLLMSpend is the token and turn total the in-memory task ledger would
// already have counted. One row is one turn. The ledger sees planner and worker
// completions, plus judge and compactor calls made on those contexts. Main-agent
// chat, goal decomposition, and side questions write llm_usage rows for the same
// task and never enter the ledger, so they are left out: counting them would
// tighten the turn cap after a restart. Cache read and write are not added:
// input_tokens already includes them. Every status counts, because a failed call
// is metered too.
func (d *DB) TaskLLMSpend(taskID string) (tokens, turns int64, err error) {
if d == nil || taskID == "" {
return 0, 0, nil
}
err = d.QueryRow(`SELECT COALESCE(SUM(input_tokens + output_tokens), 0), COUNT(*)
FROM llm_usage
WHERE task_id=$1 AND worker IN ('planner', 'worker', 'judge', 'compactor')`, taskID).Scan(&tokens, &turns)
return tokens, turns, err
}
// InsertLLMUsage appends one metering row. Best-effort: callers log and continue on
// error (a lost metering row must never break the LLM call).
func (d *DB) InsertLLMUsage(u *LLMUsage) error {
+40
View File
@@ -0,0 +1,40 @@
package db
import (
"fmt"
"testing"
"time"
)
// Judge calls share the planner or worker sink and count toward the task cap.
// Chat rows for the same task do not, or a restart would tighten the cap.
func TestTaskLLMSpendCountsJudgeNotChat(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
}
defer d.Close()
taskID := fmt.Sprintf("spend-%d", time.Now().UnixNano())
t.Cleanup(func() {
_, _ = d.Exec(`DELETE FROM llm_usage WHERE task_id=$1`, taskID)
})
insert := func(worker string, in, out int) {
t.Helper()
if err := d.InsertLLMUsage(&LLMUsage{
TaskID: taskID, Worker: worker, InputTokens: in, OutputTokens: out, Status: "ok",
}); err != nil {
t.Fatal(err)
}
}
insert("judge", 30, 10)
insert("chat", 1000, 1000)
tokens, turns, err := d.TaskLLMSpend(taskID)
if err != nil {
t.Fatal(err)
}
if tokens != 40 || turns != 1 {
t.Fatalf("tokens=%d turns=%d, want the judge row only (40 tokens, 1 turn)", tokens, turns)
}
}
+12 -7
View File
@@ -297,17 +297,22 @@ type Coverage struct {
}
// CoverageEnabled reports whether a task has the asset-coverage feature turned on
// (tasks.coverage_enabled). Missing row / error → true (fail open to the default),
// so unknown/legacy tasks keep the historical behavior. taskID<=0 → true.
func (s *AssetStore) CoverageEnabled(taskID int64) bool {
// (tasks.coverage_enabled). A missing row stays enabled, which is the historical
// default for unknown and legacy tasks. taskID<=0 stays enabled. A query error is
// returned with enabled=true so a database blip does not turn the feature off.
func (s *AssetStore) CoverageEnabled(taskID int64) (bool, error) {
if taskID <= 0 {
return true
return true, nil
}
var enabled bool
if err := s.db.QueryRow(`SELECT COALESCE(coverage_enabled,true) FROM tasks WHERE id=$1`, taskID).Scan(&enabled); err != nil {
return true
err := s.db.QueryRow(`SELECT COALESCE(coverage_enabled,true) FROM tasks WHERE id=$1`, taskID).Scan(&enabled)
if err == sql.ErrNoRows {
return true, nil
}
return enabled
if err != nil {
return true, err
}
return enabled, nil
}
// declaredScopeFilter keeps a task_scope row ts in the coverage match only when it is
+33
View File
@@ -2,6 +2,39 @@ package db
import "testing"
func TestCoverageEnabledWithoutTask(t *testing.T) {
on, err := (&AssetStore{}).CoverageEnabled(0)
if err != nil || !on {
t.Fatalf("enabled=%v err=%v", on, err)
}
}
func TestCoverageEnabledMissingRowStaysOn(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
}
defer d.Close()
on, err := d.Assets().CoverageEnabled(9_000_000_000_001)
if err != nil || !on {
t.Fatalf("missing task: enabled=%v err=%v, want enabled with no error", on, err)
}
}
func TestCoverageEnabledQueryErrorStaysOn(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
}
if err := d.Close(); err != nil {
t.Fatal(err)
}
on, err := d.Assets().CoverageEnabled(1)
if err == nil || !on {
t.Fatalf("failed query: enabled=%v err=%v, want enabled and the error", on, err)
}
}
// TestStripHostPort covers the ip:port / [ipv6]:port stripping used by the
// ip/cidr scope path so a target like "10.0.188.136:3000" no longer 404s.
func TestStripHostPort(t *testing.T) {
+13 -16
View File
@@ -46,7 +46,9 @@ type Registry struct {
m map[int64]*State
// persist / forget mirror state to the DB so a cooling-off window survives a
// restart. Both may be nil (no DB); both are called off the hot path.
// restart. Both may be nil (no DB). Both run while r.mu is held, so a later
// change to the same profile cannot enqueue ahead of this one. They must not
// call back into the registry or wait on database I/O.
persist func(id int64, st State)
forget func(id int64)
@@ -118,16 +120,14 @@ func (r *Registry) IsOpen(id int64) bool { return r.Get(id).Open() }
// recovers is fully trusted again (and drops the persisted row).
func (r *Registry) Pass(id int64) {
r.mu.Lock()
defer r.mu.Unlock()
st := r.m[id]
if st == nil || (st.Fails == 0 && st.Trips == 0 && st.OpenUntil.IsZero()) {
r.mu.Unlock()
return // already clean — nothing to write
return // already clean; nothing to write
}
delete(r.m, id)
forget := r.forget
r.mu.Unlock()
if forget != nil {
forget(id)
if r.forget != nil {
r.forget(id)
}
}
@@ -137,6 +137,7 @@ func (r *Registry) Pass(id int64) {
// true when this call is what opened the breaker, so the caller can log it once.
func (r *Registry) Trip(id int64, errMsg string, hard bool) (tripped bool) {
r.mu.Lock()
defer r.mu.Unlock()
st := r.m[id]
if st == nil {
st = &State{}
@@ -152,11 +153,8 @@ func (r *Registry) Trip(id int64, errMsg string, hard bool) (tripped bool) {
st.Fails = 0 // counted into this trip; start fresh for the half-open probe
tripped = true
}
snap := *st
persist := r.persist
r.mu.Unlock()
if persist != nil {
persist(id, snap)
if r.persist != nil {
r.persist(id, *st)
}
return tripped
}
@@ -164,11 +162,10 @@ func (r *Registry) Trip(id int64, errMsg string, hard bool) (tripped bool) {
// Reset clears one profile's state — the UI's "Recover now" action.
func (r *Registry) Reset(id int64) {
r.mu.Lock()
defer r.mu.Unlock()
delete(r.m, id)
forget := r.forget
r.mu.Unlock()
if forget != nil {
forget(id)
if r.forget != nil {
r.forget(id)
}
}
+51
View File
@@ -0,0 +1,51 @@
package llmpool
import (
"sync/atomic"
"testing"
"time"
)
// persist and forget run under the registry lock. A second update on the same
// profile cannot enqueue until the first callback returns, so the queue cannot
// observe the later state first.
func TestHealthCallbacksStayInStateOrder(t *testing.T) {
var entered atomic.Int32
firstIn := make(chan struct{})
release := make(chan struct{})
block := func() {
if entered.Add(1) == 1 {
close(firstIn)
<-release
}
}
reg := NewRegistry(
func(int64, State) { block() },
func(int64) { block() },
)
go reg.Trip(1, "down", true)
select {
case <-firstIn:
case <-time.After(2 * time.Second):
t.Fatal("Trip did not reach persist")
}
passed := make(chan struct{})
go func() {
reg.Pass(1)
close(passed)
}()
select {
case <-passed:
t.Fatal("Pass enqueued while Trip's persist was still running")
case <-time.After(50 * time.Millisecond):
}
close(release)
select {
case <-passed:
case <-time.After(2 * time.Second):
t.Fatal("Pass did not finish after Trip released the lock")
}
if entered.Load() != 2 {
t.Fatalf("callbacks=%d, want Trip then Pass", entered.Load())
}
}
+3 -3
View File
@@ -36,7 +36,7 @@ func (s *Server) assetInterceptCreateRule(w http.ResponseWriter, r *http.Request
return
}
var req assetInterceptRuleReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -63,7 +63,7 @@ func (s *Server) assetInterceptUpdateRule(w http.ResponseWriter, r *http.Request
return
}
var req assetInterceptRuleReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -109,7 +109,7 @@ func (s *Server) assetInterceptToggleRule(w http.ResponseWriter, r *http.Request
var req struct {
Enabled bool `json:"enabled"`
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
+3 -3
View File
@@ -297,7 +297,7 @@ func (s *Server) authInit(w http.ResponseWriter, r *http.Request) {
Password string `json:"password"`
SetupToken string `json:"setup_token"`
}
if err := decode(r, &req); err != nil || req.Password == "" {
if err := decode(w, r, &req); err != nil || req.Password == "" {
writeErr(w, 400, "Password cannot be empty")
return
}
@@ -354,7 +354,7 @@ func (s *Server) authChangePassword(w http.ResponseWriter, r *http.Request) {
OldPassword string `json:"old_password"`
NewPassword string `json:"new_password"`
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, "Invalid request format")
return
}
@@ -416,7 +416,7 @@ func (s *Server) login(w http.ResponseWriter, r *http.Request, getSetting func(s
Username string `json:"username"`
Password string `json:"password"`
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, "Invalid request format")
return
}
+2 -2
View File
@@ -27,7 +27,7 @@ const (
func decodeConversationRequest(w http.ResponseWriter, r *http.Request, value any) bool {
r.Body = http.MaxBytesReader(w, r.Body, maxConversationRequestBytes)
if err := decode(r, value); err != nil {
if err := decode(w, r, value); err != nil {
var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) {
writeErr(w, http.StatusRequestEntityTooLarge, "Request body too large")
@@ -396,7 +396,7 @@ func (s *Server) pgSendConversationMessage(w http.ResponseWriter, r *http.Reques
Message string `json:"message"`
Attachments []chatAttachment `json:"attachments,omitempty"` // files uploaded via method 1 (paths relative to the conversation working dir)
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
+2 -2
View File
@@ -52,7 +52,7 @@ func (s *Server) pgCreateCustomTool(w http.ResponseWriter, r *http.Request) {
return
}
var req customToolReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -104,7 +104,7 @@ func (s *Server) pgUpdateCustomTool(w http.ResponseWriter, r *http.Request) {
return
}
var req customToolReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
+69
View File
@@ -0,0 +1,69 @@
package server
import (
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestDecodeAcceptsSmallBody(t *testing.T) {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(`{"x":"ok"}`))
var v map[string]string
if err := decode(w, r, &v); err != nil || v["x"] != "ok" {
t.Fatalf("decode: %v %+v", err, v)
}
}
func TestDecodeCapsBody(t *testing.T) {
w := httptest.NewRecorder()
body := strings.NewReader(`{"x":"` + strings.Repeat("a", decodeMaxBytes) + `"}`)
r := httptest.NewRequest(http.MethodPost, "/", body)
var v map[string]string
err := decode(w, r, &v)
var tooLarge *http.MaxBytesError
if !errors.As(err, &tooLarge) {
t.Fatalf("err=%v, want the body cap", err)
}
}
func TestRawJSONBodyOverTheCapIs413(t *testing.T) {
h := limitRequestBody(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var v map[string]string
if err := json.NewDecoder(r.Body).Decode(&v); err != nil {
writeErr(w, 400, err.Error())
return
}
writeJSON(w, 200, v)
}))
small := httptest.NewRecorder()
h.ServeHTTP(small, httptest.NewRequest(http.MethodPost, "/", strings.NewReader(`{"x":"ok"}`)))
if small.Code != 200 {
t.Fatalf("small body status=%d", small.Code)
}
large := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(`{"x":"`+strings.Repeat("a", decodeMaxBytes)+`"}`))
h.ServeHTTP(large, req)
if large.Code != http.StatusRequestEntityTooLarge {
t.Fatalf("status=%d body=%s, want 413", large.Code, large.Body.String())
}
}
func TestMultipartBodyIsNotGivenTheJSONCap(t *testing.T) {
h := limitRequestBody(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if _, ok := r.Body.(*limitedBody); ok {
t.Errorf("multipart body was given the JSON cap")
}
w.WriteHeader(http.StatusNoContent)
}))
r := httptest.NewRequest(http.MethodPost, "/", strings.NewReader("not-read"))
r.Header.Set("Content-Type", "multipart/form-data; boundary=abc")
w := httptest.NewRecorder()
h.ServeHTTP(w, r)
if w.Code != http.StatusNoContent {
t.Fatalf("status=%d", w.Code)
}
}
+30 -2
View File
@@ -17,8 +17,36 @@ func (e *Engine) budgetFor(taskID string) *agent.BudgetLedger {
if v, ok := e.budgets.Load(taskID); ok {
return v.(*agent.BudgetLedger)
}
v, _ := e.budgets.LoadOrStore(taskID, agent.NewBudgetLedger(agent.BudgetsFromEnv()))
return v.(*agent.BudgetLedger)
created := agent.NewBudgetLedger(agent.BudgetsFromEnv())
e.seedBudget(taskID, created)
v, loaded := e.budgets.LoadOrStore(taskID, created)
if loaded {
return v.(*agent.BudgetLedger)
}
return created
}
// seedBudget rebuilds spent tokens and turns from llm_usage for every new
// ledger. budgetFor calls it on first use, and again after resetSettledRun or
// StopTask deletes the ledger in this same process. The in-memory sink runs
// before the row is inserted, so a call that happens after this seed is
// counted once. A nil ledger means no budget is configured and nothing is read.
// The restore does not rebuild per-worker averages, so the next worker estimate
// starts from the defaults until a worker finishes against this ledger.
func (e *Engine) seedBudget(taskID string, ledger *agent.BudgetLedger) {
if ledger == nil || e == nil || e.m == nil || e.m.pg == nil || taskID == "" {
return
}
tokens, turns, err := e.m.pg.TaskLLMSpend(taskID)
if err != nil {
log.Printf("[budget] task %s: cannot restore spend from llm_usage: %v", taskID, err)
return
}
if tokens == 0 && turns == 0 {
return
}
ledger.RestoreSpend(agent.Spend{Tokens: tokens, Turns: turns})
log.Printf("[budget] task %s restored %d tokens / %d turns from llm_usage", taskID, tokens, turns)
}
// budgetStop is the enforcement point of a planner round: it logs the remaining
+64
View File
@@ -2,9 +2,12 @@ package server
import (
"context"
"strconv"
"testing"
"time"
"github.com/Autumn-27/artex/agent"
pgdb "github.com/Autumn-27/artex/db"
"github.com/Autumn-27/norma/llm"
)
@@ -67,3 +70,64 @@ func TestResetSettledRunRestartsBudget(t *testing.T) {
t.Fatalf("revived run: ledger not fresh, spent %d tokens", got)
}
}
// With Postgres, resetSettledRun drops the ledger and the next one is seeded
// again from llm_usage. In-memory spend that was never written does not come
// back. A restore that lands on the wrap-up reserve leaves only that reserve,
// and the worker estimate starts from the defaults.
func TestResetSettledRunReseedsBudgetFromUsage(t *testing.T) {
for _, k := range []string{"ARTEX_TASK_BUDGET_TURNS", "ARTEX_INTENT_BUDGET_TOKENS", "ARTEX_INTENT_BUDGET_TURNS"} {
t.Setenv(k, "")
}
t.Setenv("ARTEX_TASK_BUDGET_TOKENS", "100000")
m, err := NewManager(t.TempDir(), "")
if err != nil {
t.Skipf("postgres unavailable (%v)", err)
}
defer m.Close()
taskID := strconv.FormatInt(time.Now().UnixNano(), 10)
t.Cleanup(func() {
_, _ = m.pg.Exec(`DELETE FROM llm_usage WHERE task_id=$1`, taskID)
})
// 100000 - wrap-up reserve (max(5%, 20000)) = 80000. One planner row.
if err := m.pg.InsertLLMUsage(&pgdb.LLMUsage{
TaskID: taskID, Worker: "planner", InputTokens: 80_000, Status: "ok",
}); err != nil {
t.Fatal(err)
}
e := NewEngine(m)
task := &Task{ID: taskID}
l := e.budgetFor(task.ID)
st := l.State()
if st.Spent.Tokens != 80_000 || st.Spent.Turns != 1 {
t.Fatalf("seed spent=%+v, want 80000 tokens and 1 turn", st.Spent)
}
if st.Remaining.Tokens != st.Margin.Tokens || !st.AtMargin || st.Dispatch {
t.Fatalf("restored task near the cap: remaining=%d margin=%d at=%v dispatch=%v",
st.Remaining.Tokens, st.Margin.Tokens, st.AtMargin, st.Dispatch)
}
if st.Estimate.Tokens != agent.DefaultWorkerEstimateTokens || st.Estimate.Turns != agent.DefaultWorkerEstimateTurns {
t.Fatalf("estimate=%+v, want the defaults (worker history is not restored)", st.Estimate)
}
l.Record(llm.Usage{InputTokens: 1000})
l.BeginWrapup()
e.settling.Store(task.ID, true)
e.resetSettledRun(task.ID)
if e.isSettling(task.ID) {
t.Fatal("revived run is still marked settling")
}
again := e.budgetFor(task.ID)
if again == l {
t.Fatal("revive reused the deleted ledger")
}
got := again.State()
if got.Spent.Tokens != 80_000 || got.Spent.Turns != 1 {
t.Fatalf("reseed spent=%+v, want the usage row and not the in-memory extra", got.Spent)
}
if got.Remaining.Tokens != got.Margin.Tokens || got.Dispatch {
t.Fatalf("reseed still past the wrap-up margin: remaining=%d margin=%d dispatch=%v",
got.Remaining.Tokens, got.Margin.Tokens, got.Dispatch)
}
}
+2 -1
View File
@@ -110,7 +110,8 @@ func (e *Engine) resetTimeoutRevival(taskID string) {
}
// resetSettledRun clears the settle state of a finished run (settling and the
// budget ledger) so a revived task can plan and claim again with a fresh budget.
// budget ledger) so a revived task can plan and claim again. The next ledger
// is seeded from llm_usage, so the spend already recorded still counts.
// It also clears the persisted end_reason of the previous run (ApplyTaskAdmission
// already did in the same UPDATE as the revive; this is idempotent and never
// touches a task that is terminal again).
+8 -8
View File
@@ -141,7 +141,7 @@ func (s *Server) interceptCreateRule(w http.ResponseWriter, r *http.Request) {
return
}
var req interceptRuleReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -169,7 +169,7 @@ func (s *Server) interceptUpdateRule(w http.ResponseWriter, r *http.Request) {
return
}
var req interceptRuleReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -217,7 +217,7 @@ func (s *Server) interceptToggleRule(w http.ResponseWriter, r *http.Request) {
var req struct {
Enabled bool `json:"enabled"`
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -400,7 +400,7 @@ func (s *Server) interceptDecide(w http.ResponseWriter, r *http.Request) {
var req struct {
Decision string `json:"decision"` // "allowed" | "denied"
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -445,7 +445,7 @@ func (s *Server) interceptBatchDecide(w http.ResponseWriter, r *http.Request) {
IDs []int64 `json:"ids"`
Decision string `json:"decision"` // "allowed" | "denied"
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -647,7 +647,7 @@ func (s *Server) targetGateSetExemptions(w http.ResponseWriter, r *http.Request)
var req struct {
Exemptions []db.TargetExemption `json:"exemptions"`
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -697,7 +697,7 @@ func (s *Server) interceptSetToolConfig(w http.ResponseWriter, r *http.Request)
var req struct {
EnabledTools []string `json:"enabled_tools"`
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -722,7 +722,7 @@ func (s *Server) interceptGetJudgeConfig(w http.ResponseWriter, r *http.Request)
// interceptSetJudgeConfig persists the judge configuration.
func (s *Server) interceptSetJudgeConfig(w http.ResponseWriter, r *http.Request) {
var req intercept.JudgeConfig
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
+203 -16
View File
@@ -1,7 +1,9 @@
package server
import (
"context"
"log"
"sync"
"time"
"github.com/Autumn-27/artex/agent"
@@ -17,26 +19,19 @@ import (
// - Circuit-breaker state is process-wide (s.llmHealth); rebuilding the pool does not clear it.
// newLLMHealthRegistry builds the process-wide circuit-breaker registry, mirroring
// state into PG so a cooling-off window survives a restart. Writes are async and
// best-effort — the in-memory copy is authoritative.
func newLLMHealthRegistry(pg *db.DB) *llmpool.Registry {
// state into PG so a cooling-off window survives a restart. Writes are async.
// The in-memory copy is authoritative. The returned syncer is nil when there is
// no database; otherwise it flushes when ctx is cancelled.
func newLLMHealthRegistry(ctx context.Context, pg *db.DB) (*llmpool.Registry, *healthSync) {
if pg == nil {
return llmpool.NewRegistry(nil, nil)
return llmpool.NewRegistry(nil, nil), nil
}
syncer := newHealthSync(ctx, pg.SaveLLMHealth, pg.ClearLLMHealth)
persist := func(id int64, st llmpool.State) {
h := db.LLMHealth{ProfileID: id, Fails: st.Fails, Trips: st.Trips, LastError: st.LastError}
if !st.OpenUntil.IsZero() {
t := st.OpenUntil
h.OpenUntil = &t
}
go func() {
if err := pg.SaveLLMHealth(h); err != nil {
log.Printf("[llmpool] failed to persist circuit-breaker state: %v", err)
}
}()
syncer.enqueue(id, healthOp{st: st})
}
forget := func(id int64) {
go func() { _ = pg.ClearLLMHealth(id) }()
syncer.enqueue(id, healthOp{clear: true})
}
reg := llmpool.NewRegistry(persist, forget)
// Restore only windows that haven't expired (LoadLLMHealth filters), so a
@@ -51,7 +46,199 @@ func newLLMHealthRegistry(pg *db.DB) *llmpool.Registry {
log.Printf("[llmpool] restored circuit-breaker state: profile #%d cooling until %s", h.ProfileID, st.OpenUntil.Format(time.RFC3339))
}
}
return reg
return reg, syncer
}
// healthOp is one circuit-breaker write for a profile. clear drops the row.
// seq orders writes for that profile: a newer op wins over one still in flight.
type healthOp struct {
seq uint64
clear bool
st llmpool.State
}
// healthSync writes circuit-breaker state from one goroutine. A newer update for
// the same profile replaces one that has not been written yet, so a burst of
// state changes becomes one database call. The database call runs without mu,
// so enqueue does not wait on a slow write. writeMu keeps those calls in order.
// Before each call, an op that is already older than the applied or pending
// sequence is dropped. save and clear must not call enqueue.
type healthSync struct {
save func(db.LLMHealth) error
clear func(int64) error
retryAfter time.Duration // tests set a short delay; zero uses one second
mu sync.Mutex
writeMu sync.Mutex // orders database calls; enqueue does not take it
pending map[int64]healthOp
applied map[int64]uint64
seq uint64
stopped bool
wake chan struct{}
done chan struct{}
}
func newHealthSync(ctx context.Context, save func(db.LLMHealth) error, clear func(int64) error) *healthSync {
if ctx == nil {
ctx = context.Background()
}
h := &healthSync{
save: save,
clear: clear,
pending: map[int64]healthOp{},
applied: map[int64]uint64{},
wake: make(chan struct{}, 1),
done: make(chan struct{}),
}
go h.loop(ctx)
return h
}
// FlushLLMHealth waits until the circuit-breaker writer has flushed the ops
// queued before shutdown. Call it after the server context is cancelled and
// before the database closes.
func (s *Server) FlushLLMHealth() {
if s == nil || s.llmHealthSync == nil {
return
}
select {
case <-s.llmHealthSync.done:
case <-time.After(5 * time.Second):
log.Printf("[llmpool] timed out flushing circuit-breaker state")
}
}
func (h *healthSync) enqueue(id int64, op healthOp) {
h.mu.Lock()
h.seq++
op.seq = h.seq
if h.stopped {
h.mu.Unlock()
if err := h.apply(id, op); err != nil {
log.Printf("[llmpool] write circuit-breaker profile #%d: %v", id, err)
}
return
}
if prev, ok := h.pending[id]; ok && prev.seq > op.seq {
h.mu.Unlock()
return
}
h.pending[id] = op
h.mu.Unlock()
select {
case h.wake <- struct{}{}:
default:
}
}
func (h *healthSync) take() map[int64]healthOp {
h.mu.Lock()
defer h.mu.Unlock()
if len(h.pending) == 0 {
return nil
}
out := h.pending
h.pending = map[int64]healthOp{}
return out
}
func (h *healthSync) restore(id int64, op healthOp) {
h.mu.Lock()
defer h.mu.Unlock()
if h.stopped || h.applied[id] >= op.seq {
return
}
if cur, ok := h.pending[id]; ok && cur.seq >= op.seq {
return
}
h.pending[id] = op
}
// superseded reports whether a newer op for id is already applied or waiting.
// The caller holds writeMu, so the decision and the database call stay in order.
func (h *healthSync) superseded(id int64, seq uint64) bool {
h.mu.Lock()
defer h.mu.Unlock()
if seq > 0 && h.applied[id] >= seq {
return true
}
cur, ok := h.pending[id]
return ok && cur.seq > seq
}
func (h *healthSync) apply(id int64, op healthOp) error {
h.writeMu.Lock()
defer h.writeMu.Unlock()
if h.superseded(id, op.seq) {
return nil
}
var err error
if op.clear {
if h.clear != nil {
err = h.clear(id)
}
} else if h.save != nil {
row := db.LLMHealth{ProfileID: id, Fails: op.st.Fails, Trips: op.st.Trips, LastError: op.st.LastError}
if !op.st.OpenUntil.IsZero() {
openUntil := op.st.OpenUntil
row.OpenUntil = &openUntil
}
err = h.save(row)
}
if err != nil {
return err
}
h.mu.Lock()
if h.applied[id] < op.seq {
h.applied[id] = op.seq
}
h.mu.Unlock()
return nil
}
func (h *healthSync) write(batch map[int64]healthOp) (failed bool) {
for id, op := range batch {
if err := h.apply(id, op); err != nil {
log.Printf("[llmpool] write circuit-breaker profile #%d: %v", id, err)
h.restore(id, op)
failed = true
}
}
return failed
}
func (h *healthSync) finish() {
h.mu.Lock()
h.stopped = true
batch := h.pending
h.pending = map[int64]healthOp{}
h.mu.Unlock()
for id, op := range batch {
if err := h.apply(id, op); err != nil {
log.Printf("[llmpool] write circuit-breaker profile #%d: %v", id, err)
}
}
}
func (h *healthSync) loop(ctx context.Context) {
defer close(h.done)
for {
failed := h.write(h.take())
delay := h.retryAfter
if delay <= 0 {
delay = time.Second
}
var retry <-chan time.Time
if failed {
retry = time.After(delay)
}
select {
case <-ctx.Done():
h.finish()
return
case <-h.wake:
case <-retry:
}
}
}
// poolMember builds one chain member from a profile, reusing the per-profile
+183
View File
@@ -0,0 +1,183 @@
package server
import (
"context"
"errors"
"sync"
"testing"
"time"
"github.com/Autumn-27/artex/db"
"github.com/Autumn-27/artex/llmpool"
)
func TestHealthSyncLoopWritesLatest(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
var mu sync.Mutex
saved := map[int64]db.LLMHealth{}
cleared := map[int64]int{}
savedCh := make(chan struct{}, 4)
clearedCh := make(chan struct{}, 4)
h := newHealthSync(ctx,
func(row db.LLMHealth) error {
mu.Lock()
saved[row.ProfileID] = row
mu.Unlock()
savedCh <- struct{}{}
return nil
},
func(id int64) error {
mu.Lock()
cleared[id]++
delete(saved, id)
mu.Unlock()
clearedCh <- struct{}{}
return nil
},
)
openUntil := time.Now().Add(time.Minute).UTC().Truncate(time.Second)
h.enqueue(8, healthOp{st: llmpool.State{Fails: 1, LastError: "first"}})
h.enqueue(8, healthOp{st: llmpool.State{Fails: 4, Trips: 1, OpenUntil: openUntil, LastError: "down"}})
deadline := time.After(2 * time.Second)
var latest db.LLMHealth
for {
mu.Lock()
latest = saved[8]
mu.Unlock()
if latest.Fails == 4 && latest.OpenUntil != nil && latest.OpenUntil.Equal(openUntil) {
break
}
select {
case <-savedCh:
case <-deadline:
t.Fatalf("loop saved=%+v, want the latest trip", latest)
}
}
h.enqueue(7, healthOp{clear: true})
select {
case <-clearedCh:
case <-time.After(2 * time.Second):
t.Fatal("loop did not clear")
}
mu.Lock()
defer mu.Unlock()
got := saved[8]
if got.Fails != 4 || got.Trips != 1 || got.LastError != "down" || got.OpenUntil == nil || !got.OpenUntil.Equal(openUntil) {
t.Fatalf("saved=%+v, want the latest trip including OpenUntil", got)
}
if cleared[7] != 1 {
t.Fatalf("cleared=%v", cleared)
}
}
func TestHealthSyncRetriesFailedSave(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
var mu sync.Mutex
calls := 0
done := make(chan struct{})
h := newHealthSync(ctx,
func(row db.LLMHealth) error {
mu.Lock()
calls++
n := calls
mu.Unlock()
if n == 1 {
return errors.New("database down")
}
if n == 2 {
close(done)
}
return nil
},
func(int64) error { return nil },
)
h.retryAfter = 20 * time.Millisecond
h.enqueue(3, healthOp{st: llmpool.State{Fails: 2, LastError: "blip"}})
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("failed save was not retried")
}
mu.Lock()
defer mu.Unlock()
if calls < 2 {
t.Fatalf("save calls=%d, want a retry", calls)
}
}
func TestHealthSyncEnqueueDoesNotWaitOnSave(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
started := make(chan struct{})
release := make(chan struct{})
later := make(chan struct{})
h := newHealthSync(ctx,
func(row db.LLMHealth) error {
if row.Fails == 1 {
close(started)
<-release
return nil
}
if row.Fails == 9 {
close(later)
}
return nil
},
func(int64) error { return nil },
)
h.enqueue(4, healthOp{st: llmpool.State{Fails: 1}})
select {
case <-started:
case <-time.After(2 * time.Second):
t.Fatal("save did not start")
}
done := make(chan struct{})
go func() {
h.enqueue(4, healthOp{st: llmpool.State{Fails: 9, LastError: "later"}})
close(done)
}()
select {
case <-done:
case <-time.After(200 * time.Millisecond):
t.Fatal("enqueue blocked while a save was in flight")
}
close(release)
select {
case <-later:
case <-time.After(2 * time.Second):
t.Fatal("the later state was not written")
}
}
func TestHealthSyncDropsStaleWrite(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
writes := 0
h := newHealthSync(ctx,
func(db.LLMHealth) error {
writes++
return nil
},
func(int64) error { return nil },
)
h.mu.Lock()
h.seq = 4
h.applied[5] = 2
h.pending[6] = healthOp{seq: 4, clear: true}
h.mu.Unlock()
if err := h.apply(5, healthOp{seq: 1, st: llmpool.State{Fails: 1}}); err != nil {
t.Fatal(err)
}
if err := h.apply(6, healthOp{seq: 3, st: llmpool.State{Fails: 3}}); err != nil {
t.Fatal(err)
}
if writes != 0 {
t.Fatalf("stale writes=%d", writes)
}
}
+7 -4
View File
@@ -117,7 +117,9 @@ type Server struct {
// It deliberately lives OUTSIDE the provider caches: rebuilding the chain
// (saving an unrelated profile, flipping a setting) must not erase what we
// learned about which backends are out of credit / rate-limited.
llmHealth *llmpool.Registry
// llmHealthSync flushes that state when ctx is cancelled, before the database closes.
llmHealth *llmpool.Registry
llmHealthSync *healthSync
// Explicit task chains use one task-scoped dynamic provider shared by goals,
// planner, workers, and the main agent. Bundles are stable; their runtime reads
@@ -163,9 +165,10 @@ func New(ctx context.Context, m *Manager, skillDir string, dataDir string, keyDi
chatCancel: map[string]context.CancelCauseFunc{}, triggerQ: map[string][]triggeredRun{},
triggerActive: map[string]int{}, triggerCfg: map[string]triggerBehavior{},
profChatAgents: map[int64]*agent.ChatAgent{},
provByProfile: map[int64]*provEntry{}, llmHealth: newLLMHealthRegistry(m.pg),
taskAgents: map[string]*taskAgentBundle{}, archiveWake: make(chan struct{}, 1),
provByProfile: map[int64]*provEntry{},
taskAgents: map[string]*taskAgentBundle{}, archiveWake: make(chan struct{}, 1),
loginLimit: newLoginLimiter(time.Now)}
s.llmHealth, s.llmHealthSync = newLLMHealthRegistry(ctx, m.pg)
for _, o := range opts {
o(s)
}
@@ -1024,7 +1027,7 @@ func (s *Server) Handler() http.Handler {
root := http.NewServeMux()
root.Handle("/api/", api)
root.Handle("/", s.webuiHandler())
return root
return limitRequestBody(root)
}
// --- handlers ---
+103 -22
View File
@@ -9,6 +9,7 @@ import (
"fmt"
"io"
"log"
"mime"
"net/http"
"net/url"
"os"
@@ -79,7 +80,87 @@ func pathInt(r *http.Request, name string) (int64, bool) {
return n, err == nil
}
func decode(r *http.Request, v any) error { return json.NewDecoder(r.Body).Decode(v) }
// decodeMaxBytes is the default JSON body cap. Multipart uploads are not capped
// here; those handlers set their own limit. A smaller MaxBytesReader installed
// by a handler still wins, because it sits inside this one.
const decodeMaxBytes = 32 << 20
// limitedBody records a MaxBytesError so a later 400 can be answered as 413.
type limitedBody struct {
io.ReadCloser
tooLarge bool
}
func (b *limitedBody) Read(p []byte) (int, error) {
n, err := b.ReadCloser.Read(p)
var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) {
b.tooLarge = true
}
return n, err
}
type bodyLimitWriter struct {
http.ResponseWriter
tooLarge func() bool
}
func (b *bodyLimitWriter) WriteHeader(code int) {
if code == http.StatusBadRequest && b.tooLarge != nil && b.tooLarge() {
code = http.StatusRequestEntityTooLarge
}
b.ResponseWriter.WriteHeader(code)
}
func (b *bodyLimitWriter) Flush() {
if f, ok := b.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
func (b *bodyLimitWriter) Unwrap() http.ResponseWriter { return b.ResponseWriter }
// limitRequestBody caps a non-multipart body at decodeMaxBytes and passes the
// real ResponseWriter into MaxBytesReader. A handler that then answers 400 for
// that read is rewritten to 413.
func limitRequestBody(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Body == nil || r.Body == http.NoBody || isMultipart(r) {
next.ServeHTTP(w, r)
return
}
body := &limitedBody{ReadCloser: http.MaxBytesReader(w, r.Body, decodeMaxBytes)}
r.Body = body
next.ServeHTTP(&bodyLimitWriter{ResponseWriter: w, tooLarge: func() bool { return body.tooLarge }}, r)
})
}
func isMultipart(r *http.Request) bool {
media, _, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
if err != nil {
return false
}
return strings.HasPrefix(media, "multipart/")
}
func decode(w http.ResponseWriter, r *http.Request, v any) error {
if r.Body != nil {
if _, already := r.Body.(*limitedBody); !already {
r.Body = &limitedBody{ReadCloser: http.MaxBytesReader(w, r.Body, decodeMaxBytes)}
}
}
err := json.NewDecoder(r.Body).Decode(v)
var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) {
if lb, ok := r.Body.(*limitedBody); ok {
lb.tooLarge = true
}
// Return the original error: a caller that maps 413 itself keeps working,
// and the middleware still rewrites a 400 for callers that do not.
return tooLarge
}
return err
}
// ---------- tasks (delete) ----------
@@ -138,7 +219,7 @@ func (s *Server) pgDeleteTask(w http.ResponseWriter, r *http.Request) {
return
}
var opts DeleteTaskOptions
if err := decode(r, &opts); err != nil && err != io.EOF {
if err := decode(w, r, &opts); err != nil && err != io.EOF {
writeErr(w, 400, "invalid JSON: "+err.Error())
return
}
@@ -250,7 +331,7 @@ func (s *Server) pgCreateAgent(w http.ResponseWriter, r *http.Request) {
return
}
var req struct{ Key, Name, Description string }
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -290,7 +371,7 @@ func (s *Server) pgUpdateAgent(w http.ResponseWriter, r *http.Request) {
return
}
var req struct{ Name, Description string }
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -516,7 +597,7 @@ func (s *Server) pgSavePrompt(w http.ResponseWriter, r *http.Request) {
return
}
var body struct{ Template, Note string }
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -565,7 +646,7 @@ func (s *Server) pgSaveWrapup(w http.ResponseWriter, r *http.Request) {
Prompt string `json:"prompt"`
MaxTurns *int `json:"max_turns"`
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -619,7 +700,7 @@ func (s *Server) pgSaveTaskTimeoutWrapup(w http.ResponseWriter, r *http.Request)
Prompt string `json:"prompt"`
MaxTurns *int `json:"max_turns"`
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -689,7 +770,7 @@ func (s *Server) pgPreviewPrompt(w http.ResponseWriter, r *http.Request) {
Template string `json:"template"`
Sample map[string]string `json:"sample"`
}
_ = decode(r, &body)
_ = decode(w, r, &body)
vars, _ := pg.PromptVars(a.ID)
if body.Template == "" {
body.Template, _ = pg.CurrentPrompt(a.ID)
@@ -724,7 +805,7 @@ func (s *Server) pgSetAgentVisibility(w http.ResponseWriter, r *http.Request) {
MCP []int64 `json:"mcp"`
Skill []string `json:"skill"`
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -797,7 +878,7 @@ func (s *Server) pgUpdateTool(w http.ResponseWriter, r *http.Request) {
Agents []string `json:"agents"`
Enabled bool `json:"enabled"`
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -963,7 +1044,7 @@ func (s *Server) pgSaveMCP(w http.ResponseWriter, r *http.Request) {
return
}
var m db.MCPServer
if err := decode(r, &m); err != nil {
if err := decode(w, r, &m); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -1182,7 +1263,7 @@ func (s *Server) fsCreateSkill(w http.ResponseWriter, r *http.Request) {
MCPs []string `json:"mcps"` // optional; MCP servers this skill unlocks
Instructions string `json:"instructions"` // optional; scaffolded if empty
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -1267,7 +1348,7 @@ func (s *Server) fsUpdateSkillMeta(w http.ResponseWriter, r *http.Request) {
License *string `json:"license"`
Compatibility *string `json:"compatibility"`
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -1788,7 +1869,7 @@ func (s *Server) fsWriteFile(w http.ResponseWriter, r *http.Request) {
var body struct {
Content string `json:"content"`
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -1819,7 +1900,7 @@ func (s *Server) fsCreateDir(w http.ResponseWriter, r *http.Request) {
var body struct {
Path string `json:"path"`
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -1888,7 +1969,7 @@ func (s *Server) pgToggleSkillVisibility(w http.ResponseWriter, r *http.Request)
SkillName string `json:"skill_name"`
Visible bool `json:"visible"`
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -1926,7 +2007,7 @@ func (s *Server) pgToggleVisibility(w http.ResponseWriter, r *http.Request) {
ResourceID int64 `json:"resource_id"`
Visible bool `json:"visible"`
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -1967,7 +2048,7 @@ func (s *Server) pgSaveProfile(w http.ResponseWriter, r *http.Request) {
APIKey string `json:"api_key"`
Streaming *bool `json:"streaming"`
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -2036,7 +2117,7 @@ func (s *Server) pgSaveLLMRetryPolicy(w http.ResponseWriter, r *http.Request) {
return
}
var pol db.LLMRetryPolicy
if err := decode(r, &pol); err != nil {
if err := decode(w, r, &pol); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -2108,7 +2189,7 @@ func (s *Server) pgActivateProfile(w http.ResponseWriter, r *http.Request) {
var body struct {
ID int64 `json:"id"`
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -2141,7 +2222,7 @@ func (s *Server) pgLLMPoolReset(w http.ResponseWriter, r *http.Request) {
var body struct {
ID int64 `json:"id"` // 0 / omitted = all
}
if err := decode(r, &body); err != nil {
if err := decode(w, r, &body); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -2168,7 +2249,7 @@ func (s *Server) pgListModels(w http.ResponseWriter, r *http.Request) {
Proxy string `json:"proxy"`
ProfileID *int64 `json:"profile_id"` // fallback: use stored key from this profile
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
+2 -2
View File
@@ -124,7 +124,7 @@ func (s *Server) syncSSDatasource(w http.ResponseWriter, r *http.Request) {
URL string `json:"url"`
APIKey string `json:"api_key"`
}
_ = decode(r, &body) // empty body = just create the placeholder
_ = decode(w, r, &body) // empty body = just create the placeholder
body.URL = strings.TrimSpace(body.URL)
body.APIKey = strings.TrimSpace(body.APIKey)
@@ -266,7 +266,7 @@ func (s *Server) syncSSRun(w http.ResponseWriter, r *http.Request) {
return
}
var req ssSyncReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, "invalid JSON: "+err.Error())
return
}
+2 -2
View File
@@ -512,7 +512,7 @@ func (s *Server) queueTaskArchiveDelete(w http.ResponseWriter, r *http.Request)
func (s *Server) queueTaskArchivesBatch(w http.ResponseWriter, r *http.Request) {
var request archiveBatchRequest
if err := decode(r, &request); err != nil {
if err := decode(w, r, &request); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -552,7 +552,7 @@ func (s *Server) queueTaskArchivesBatch(w http.ResponseWriter, r *http.Request)
func (s *Server) queueTaskArchiveActionBatch(w http.ResponseWriter, r *http.Request, action string) {
var request archiveBatchRequest
if err := decode(r, &request); err != nil {
if err := decode(w, r, &request); err != nil {
writeErr(w, 400, err.Error())
return
}
+1 -1
View File
@@ -33,7 +33,7 @@ func (s *Server) attachTaskAssets(w http.ResponseWriter, r *http.Request) {
SourceSummary string `json:"source_summary"`
Scope companyScopeInputs `json:"scope"`
}
if err := decode(r, &request); err != nil {
if err := decode(w, r, &request); err != nil {
var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) {
writeErr(w, http.StatusRequestEntityTooLarge, "request body too large")
+3 -3
View File
@@ -21,7 +21,7 @@ type taskCategoryRequest struct {
func decodeTaskCategoryRequest(w http.ResponseWriter, r *http.Request) (*taskCategoryRequest, bool) {
r.Body = http.MaxBytesReader(w, r.Body, maxTaskCategoryRequestBytes)
var request taskCategoryRequest
if err := decode(r, &request); err != nil {
if err := decode(w, r, &request); err != nil {
var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) {
writeErr(w, http.StatusRequestEntityTooLarge, "request body too large")
@@ -150,7 +150,7 @@ func (s *Server) updateTaskCategory(w http.ResponseWriter, r *http.Request) {
var request struct {
CategoryID json.RawMessage `json:"category_id"`
}
if err := decode(r, &request); err != nil {
if err := decode(w, r, &request); err != nil {
writeErr(w, http.StatusBadRequest, err.Error())
return
}
@@ -184,7 +184,7 @@ func (s *Server) updateTasksCategoryBatch(w http.ResponseWriter, r *http.Request
TaskIDs []string `json:"task_ids"`
CategoryID json.RawMessage `json:"category_id"`
}
if err := decode(r, &request); err != nil {
if err := decode(w, r, &request); err != nil {
var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) {
writeErr(w, http.StatusRequestEntityTooLarge, "request body too large")
+3 -3
View File
@@ -91,7 +91,7 @@ func (s *Server) taskInterceptCreateRule(w http.ResponseWriter, r *http.Request)
return
}
var req taskInterceptRuleReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -123,7 +123,7 @@ func (s *Server) taskInterceptUpdateRule(w http.ResponseWriter, r *http.Request)
return
}
var req taskInterceptRuleReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -180,7 +180,7 @@ func (s *Server) taskInterceptToggleRule(w http.ResponseWriter, r *http.Request)
var req struct {
Enabled bool `json:"enabled"`
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
+1 -1
View File
@@ -30,7 +30,7 @@ func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) {
PlanQuietMode *bool `json:"plan_quiet_mode"`
TargetGate *bool `json:"target_gate"`
}
if err := decode(r, &request); err != nil {
if err := decode(w, r, &request); err != nil {
var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) {
writeErr(w, http.StatusRequestEntityTooLarge, "request body too large")
+3 -3
View File
@@ -216,7 +216,7 @@ func (s *Server) rewriteCreateRule(w http.ResponseWriter, r *http.Request) {
return
}
var req rewriteRuleReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -249,7 +249,7 @@ func (s *Server) rewriteUpdateRule(w http.ResponseWriter, r *http.Request) {
return
}
var req rewriteRuleReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -308,7 +308,7 @@ func (s *Server) rewriteToggleRule(w http.ResponseWriter, r *http.Request) {
var req struct {
Enabled bool `json:"enabled"`
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
+2 -2
View File
@@ -60,7 +60,7 @@ func (s *Server) pgCreateTrigger(w http.ResponseWriter, r *http.Request) {
return
}
var req triggerReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -96,7 +96,7 @@ func (s *Server) pgUpdateTrigger(w http.ResponseWriter, r *http.Request) {
return
}
var req triggerReq
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
+2 -2
View File
@@ -139,7 +139,7 @@ func (s *Server) wsWrite(w http.ResponseWriter, r *http.Request) {
Path string `json:"path"`
Content string `json:"content"`
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}
@@ -168,7 +168,7 @@ func (s *Server) wsMkdir(w http.ResponseWriter, r *http.Request) {
var req struct {
Path string `json:"path"`
}
if err := decode(r, &req); err != nil {
if err := decode(w, r, &req); err != nil {
writeErr(w, 400, err.Error())
return
}