Compare commits

..
Author SHA1 Message Date
BaaRai 2853f085e5 fix: llm/test stored-key fallback skips the runtime's key normalization
The connection test resolves its API key as: form input, else the stored
profile's key, else the global key. Form keys pass through ConfigFrom,
which trims surrounding whitespace; the stored-key fallback assigned
p.APIKey raw. A key pasted with a trailing space or newline therefore
behaved asymmetrically: conversations worked (the runtime path loads the
profile through ConfigFrom and trims) while the connection test sent the
malformed header and failed auth with 401. SaveProfile had the same gap
one layer down, storing the raw pasted value.

Changes:
- the llm/test fallback now trims the stored key and session header name,
  matching ConfigFrom's normalization so the test sends byte-identical
  headers to what the runtime sends for the same profile
- SaveProfile trims the key before storing (and before deriving the
  hint), so surrounding whitespace never reaches the database
- tests: an integration test saves a profile with a whitespace-padded
  key, runs the real handler against a fake anthropic endpoint, and
  asserts the endpoint received the trimmed key; a second test pins the
  ConfigFrom trim the fallback must match
2026-10-12 07:37:32 +02:00
BaaRai 0e53de78c2 Merge pull request #57 from BaaRai/feat/ask-operator-env-handoff
feat: ask_operator environmental handoff discipline
2026-10-12 06:46:25 +02:00
BaaRai ce8efa5139 Merge pull request #55 from BaaRai/feat/codeaudit-skill
feat: codeaudit skill (white-box source audit pipeline)
2026-10-12 06:46:23 +02:00
BaaRai 4b815d5ee0 feat: ask_operator environmental handoff discipline
When a worker hits an environmental failure (sandboxed or filtered
network path, missing binary or key, TLS or proxy interception, missing
runtime permission), nothing today tells it how to stop: transcripts
show turn after turn of retries with different flags, equivalent tools,
or fallback transports, all burning budget against a wall the agent
cannot move.

Changes:

- the ask_operator tool description now covers environmental blocks
  explicitly: do not retry or work around; include the exact single
  command the operator should run in their own terminal as one
  copy-pastable line, plus what to paste back, then wait. The question
  parameter description gains the same guidance.
- worker and pentest seeds gain an "Operator handoff on environmental
  blocks" block: stop after the first environmental failure, no
  flag-hopping, no bypass hunting, one command per ask_operator request,
  never batch.
- tests: the tool description must keep the handoff markers
  (TestAskOperatorEnvHandoffGuidance) and the executor seeds must keep
  the block (TestPromptSeedsCarryOperatorHandoff).

Pattern adapted from bug-bounty-hunting-prompts (MIT), whose operator
handoff rule produced exactly this single-command format.
2026-10-12 06:45:21 +02:00
8 changed files with 152 additions and 6 deletions
+24
View File
@@ -0,0 +1,24 @@
package agent
import "strings"
import "testing"
// The operator-handoff discipline must stay in the executor agents' default
// seeds: environmental failures stop the retry loop and become one
// copy-pastable ask_operator command.
func TestPromptSeedsCarryOperatorHandoff(t *testing.T) {
for name, tmpl := range BuiltinPromptSeeds() {
switch name {
case "worker", "pentest":
for _, marker := range []string{
"Operator handoff on environmental blocks",
"one line, copy-pastable",
"never batch",
} {
if !strings.Contains(tmpl, marker) {
t.Errorf("%s seed missing operator-handoff marker %q", name, marker)
}
}
}
}
}
+5 -1
View File
@@ -90,7 +90,11 @@ Finding quality gates (proof standard, measured impact, testing bounds):
- **Every reported finding carries a measured impact**: the attacker outcome as a number with its denominator ("3 of 20 payload variants returned other-tenant rows; 17 rejected", "commands executed: 2, output returned: yes") plus what the defense blocked. "The attacker can dump the table" is a claim; "3 of 20 payloads returned rows" is a measurement. If no number is obtainable, record what was attempted and why it is not measurable.
- **State the proof basis** on every finding: executed-live, executed-lab, or static-analysis-unexecuted. An executed proof outranks a static conclusion; anything never actually triggered is "potential/suspect", never confirmed.
- **Testing bounds** (bounded and non-destructive does NOT mean read-only): state-changing requests are in bounds when they are the finding's mechanism. Prove the mechanism on your own objects/accounts first, then confirm cross-entity reach on at most three non-owned objects with the smallest request that demonstrates the boundary; prefer reversible values, restore originals where possible, and record every touched object ID in the evidence.
- **Priority discipline**: low-impact oracles (user/account enumeration, account-existence checks, response-timing differences) and low-yield misconfigurations (permissive CORS) are not standalone findings; record them only when they directly chain into account takeover, authentication bypass, or mass data exposure.`
- **Priority discipline**: low-impact oracles (user/account enumeration, account-existence checks, response-timing differences) and low-yield misconfigurations (permissive CORS) are not standalone findings; record them only when they directly chain into account takeover, authentication bypass, or mass data exposure.
Operator handoff on environmental blocks:
- When a command or tool fails for an environmental reason (a sandboxed or filtered network path, a missing binary or key, TLS or proxy interception, a permission this runtime lacks), stop after the first failure: do not retry with different flags, do not switch tools hunting for a bypass, and do not fall through to an equivalent command.
- Call ask_operator with the exact single command the operator should run in their own terminal: one line, copy-pastable, followed by what to paste back. One command per request; never batch. Wait for the reply instead of burning turns.`
// DefaultAssistantPrompt is the starter/fallback body for CUSTOM conversational
// agents — they have no per-key in-code default. It is seeded into agent_prompts
+5 -1
View File
@@ -262,7 +262,11 @@ Finding quality gates (proof standard, measured impact, testing bounds):
- **Every reported finding carries a measured impact**: the attacker outcome as a number with its denominator ("3 of 20 payload variants returned other-tenant rows; 17 rejected", "commands executed: 2, output returned: yes") plus what the defense blocked. "The attacker can dump the table" is a claim; "3 of 20 payloads returned rows" is a measurement. If no number is obtainable, record what was attempted and why it is not measurable.
- **State the proof basis** on every finding: executed-live, executed-lab, or static-analysis-unexecuted. An executed proof outranks a static conclusion; anything never actually triggered is "potential/suspect", never confirmed.
- **Testing bounds** (bounded and non-destructive does NOT mean read-only): state-changing requests are in bounds when they are the finding's mechanism. Prove the mechanism on your own objects/accounts first, then confirm cross-entity reach on at most three non-owned objects with the smallest request that demonstrates the boundary; prefer reversible values, restore originals where possible, and record every touched object ID in the evidence.
- **Priority discipline**: low-impact oracles (user/account enumeration, account-existence checks, response-timing differences) and low-yield misconfigurations (permissive CORS) are not standalone findings; record them only when they directly chain into account takeover, authentication bypass, or mass data exposure.`
- **Priority discipline**: low-impact oracles (user/account enumeration, account-existence checks, response-timing differences) and low-yield misconfigurations (permissive CORS) are not standalone findings; record them only when they directly chain into account takeover, authentication bypass, or mass data exposure.
Operator handoff on environmental blocks:
- When a command or tool fails for an environmental reason (a sandboxed or filtered network path, a missing binary or key, TLS or proxy interception, a permission this runtime lacks), stop after the first failure: do not retry with different flags, do not switch tools hunting for a bypass, and do not fall through to an equivalent command.
- Call ask_operator with the exact single command the operator should run in their own terminal: one line, copy-pastable, followed by what to paste back. One command per request; never batch. Wait for the reply instead of burning turns.`
// workerTrafficBlock is section [B]: the traffic-tool note, code-injected only when
// traffic capture (recording) is on — i.e. the traffic_* tools actually exist.
+5
View File
@@ -6,6 +6,7 @@ import (
"encoding/json"
"errors"
"fmt"
"strings"
"time"
)
@@ -160,6 +161,10 @@ ORDER BY is_default DESC, priority DESC, id ASC`)
// SaveProfile inserts (id==0) or updates a profile. Empty apiKey on update keeps existing.
func (d *DB) SaveProfile(p *LLMProfile) (int64, error) {
// Pasted keys can carry surrounding whitespace; every reader that goes
// through ConfigFrom trims, so store the trimmed form or the raw value
// leaks into paths that assign it directly (the llm/test fallback).
p.APIKey = strings.TrimSpace(p.APIKey)
hint := p.APIKeyHint
if len(p.APIKey) >= 4 {
hint = "…" + p.APIKey[len(p.APIKey)-4:]
+2 -2
View File
@@ -25,13 +25,13 @@ var askOperatorImageExt = map[string]bool{".png": true, ".jpg": true, ".jpeg": t
// resumes the run with the answer as the next input (#41).
func (s *Server) toolAskOperator() actool.CoreTool {
return wrTool("ask_operator",
"Ask the human operator for input you cannot obtain yourself (a captcha solution, an email/SMS OTP code, a 2FA answer, a decision). The question appears in this intent's chat for the operator, optionally with a screenshot, and this intent then pauses; the operator's reply resumes this run with the answer as your next input. Ask precisely, and only when you are genuinely blocked.",
"Ask the human operator for input you cannot obtain yourself (a captcha solution, an email/SMS OTP code, a 2FA answer, a decision, or an action only they can perform). The question appears in this intent's chat for the operator, optionally with a screenshot, and this intent then pauses; the operator's reply resumes this run with the answer as your next input. Ask precisely, and only when you are genuinely blocked. When the block is environmental (a sandboxed or filtered network path, a missing binary or key, a proxy or TLS failure this runtime cannot fix), do not retry or work around it: include in the question the exact single command the operator should run in their own terminal, as one copy-pastable line, plus what to paste back, then wait.",
map[string]any{
"type": "object",
"properties": map[string]any{
"question": map[string]any{
"type": "string",
"description": "What you need from the operator, precisely (e.g. 'enter the 6-digit code sent to x@y', 'solve the captcha in the attached screenshot').",
"description": "What you need from the operator, precisely (e.g. 'enter the 6-digit code sent to x@y', 'solve the captcha in the attached screenshot'). For environmental blocks: the exact single command they should run in their own terminal (one line, copy-pastable) and what to paste back.",
},
"image_path": map[string]any{
"type": "string",
+12
View File
@@ -68,3 +68,15 @@ func TestAskOperatorActivityShape(t *testing.T) {
t.Fatalf("unexpected detail payload: %+v", parsed)
}
}
// The tool description must tell agents what to do on environmental blocks:
// hand the operator one copy-pastable command instead of retrying or hunting
// for bypasses. Pattern adapted from bug-bounty-hunting-prompts (MIT).
func TestAskOperatorEnvHandoffGuidance(t *testing.T) {
desc := (&Server{}).toolAskOperator().Description()
for _, marker := range []string{"copy-pastable", "do not retry or work around"} {
if !strings.Contains(desc, marker) {
t.Errorf("ask_operator description missing env-handoff marker %q", marker)
}
}
}
+94
View File
@@ -0,0 +1,94 @@
package server
import (
"bytes"
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"github.com/Autumn-27/artex/agent"
"github.com/Autumn-27/artex/db"
)
// The llm/test stored-key fallback must send the same header bytes the runtime
// sends: form keys are trimmed inside ConfigFrom, the DB fallback used to be
// assigned raw, so a key saved with surrounding whitespace ran fine in
// conversations but 401'd the connection test.
func TestLLMTestStoredKeyFallbackIsTrimmed(t *testing.T) {
m, err := NewManager(t.TempDir(), "")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { m.Close() })
if m.pg == nil {
t.Skip("no postgres for this run")
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
s := New(ctx, m, t.TempDir(), t.TempDir(), t.TempDir())
var gotKey atomic.Value
fake := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotKey.Store(r.Header.Get("x-api-key"))
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"id":"m","type":"message","role":"assistant","model":"m","content":[{"type":"text","text":"OK"}],"stop_reason":"end_turn","usage":{"input_tokens":1,"output_tokens":1}}`))
}))
t.Cleanup(fake.Close)
pid, err := m.pg.SaveProfile(&db.LLMProfile{
Name: "ws-key", Format: "anthropic", Model: "claude-x",
BaseURL: fake.URL, APIKey: "k1 \t",
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _, _ = m.pg.Exec("DELETE FROM llm_profiles WHERE id=$1", pid) })
// The handler trims the fallback key: the profile was saved through
// SaveProfile (which now trims), so also prove the storage trim itself.
stored, err := m.pg.ProfileByID(pid)
if err != nil || stored == nil {
t.Fatalf("ProfileByID: %v %v", stored, err)
}
if stored.APIKey != "k1" {
t.Fatalf("SaveProfile must trim surrounding whitespace, got %q", stored.APIKey)
}
body, _ := json.Marshal(map[string]any{
"provider": "anthropic", "model": "claude-x", "base_url": fake.URL,
"api_key": "", "profile_id": pid, "streaming": false,
})
req := httptest.NewRequest(http.MethodPost, "/api/llm/test", bytes.NewReader(body))
req.Header.Set("Content-Type", "application/json")
rec := httptest.NewRecorder()
s.testLLM(rec, req)
if k, _ := gotKey.Load().(string); k != "k1" {
t.Fatalf("test request must send the trimmed stored key, header was %q", k)
}
// The reply body only proves the call completed; the header is the assertion.
if rec.Code != http.StatusOK {
t.Fatalf("handler status %d", rec.Code)
}
var resp struct {
OK bool `json:"ok"`
Error string `json:"error"`
}
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
if !resp.OK {
t.Logf("note: provider round-trip returned ok=false (%s); header assertion stands", resp.Error)
}
}
// Sanity: the runtime path and the test path must agree byte-for-byte on the
// key they would send for the same stored profile.
func TestLLMTestFallbackMatchesRuntimeNormalization(t *testing.T) {
const raw = " k1 \n"
runtime := agent.ConfigFrom("anthropic", "m", "https://x.example", raw, "")
if runtime.APIKey != "k1" {
t.Fatalf("ConfigFrom trims: %q", runtime.APIKey)
}
}
+5 -2
View File
@@ -1404,11 +1404,14 @@ func (s *Server) testLLM(w http.ResponseWriter, r *http.Request) {
// Same for the session header name: when the form omits it, the stored profile's value is the fallback.
if req.ProfileID != nil && (cfg.APIKey == "" || cfg.SessionHeaderKey == "") {
if p, err := s.m.pg.ProfileByID(*req.ProfileID); err == nil && p != nil {
// Trim like ConfigFrom does for form values: a key stored with
// surrounding whitespace runs fine in conversations (the runtime
// path re-trims) but would be sent raw here and fail auth.
if cfg.APIKey == "" {
cfg.APIKey = p.APIKey
cfg.APIKey = strings.TrimSpace(p.APIKey)
}
if cfg.SessionHeaderKey == "" {
cfg.SessionHeaderKey = p.SessionHeaderKey
cfg.SessionHeaderKey = strings.TrimSpace(p.SessionHeaderKey)
}
}
}