mirror of
https://github.com/BaaRai/ARTEX.git
synced 2026-11-19 22:50:55 +08:00
feat(engine): intent claim heartbeat, TTL and fencing token (S3c)
An intent claim was atomic but had no heartbeat, no expiry and no owner token: startup reset every running intent to open unconditionally, so a second instance on the same database stole live claims, and the worker that lost its intent could still write the result. - schema: last_heartbeat_at and unclaim_token on exploration_nodes (idempotent ADD COLUMN IF NOT EXISTS, nullable, no CHECK, no backfill). - db: every claim (open->running and the paused->running message path) gets a fresh crypto/rand token and starts its lease in the same UPDATE. HeartbeatIntentClaim renews it for that token only. - TTL: db.IntentClaimTTL = 90s, heartbeat every 15s (db.IntentClaimHeartbeatInterval), both on the database clock. - takeover: ResetRunningIntents now reopens only claims whose heartbeat is older than the TTL, or NULL (rows claimed before the columns existed). The engine runs it per task every 15s as well as at startup. - fencing: a heartbeat or result write whose token is not the current one returns db.ErrIntentClaimLost and writes nothing. All intent state writes in runIntent go through the fenced SettleIntentClaim / SettleIntentClaimBlocked; a rejected heartbeat cancels the stale run with the new work_claim_lost cause. Behaviour change: after a restart, intents that were in flight are resumed once their claim expires (at most 105s) instead of immediately. Not in this change: fencing of the facts, findings and activity a worker writes during its run; they are not guarded by the token.
This commit is contained in:
@@ -201,6 +201,13 @@ ALTER TABLE exploration_nodes ADD COLUMN IF NOT EXISTS delete_reason TEXT;
|
||||
-- a node most recently went from "has a live downstream branch" to none (NULL = hot).
|
||||
ALTER TABLE exploration_nodes ADD COLUMN IF NOT EXISTS content_version INT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE exploration_nodes ADD COLUMN IF NOT EXISTS cold_since_round BIGINT;
|
||||
-- Intent claim lease: unclaim_token is the fencing token of the current claim (fresh
|
||||
-- per claim, cleared when the claim is settled or taken over); last_heartbeat_at is
|
||||
-- the claimer's latest heartbeat. A running intent whose heartbeat is older than
|
||||
-- db.IntentClaimTTL may be taken over. Both are NULL on rows claimed before these
|
||||
-- columns existed, which counts as expired.
|
||||
ALTER TABLE exploration_nodes ADD COLUMN IF NOT EXISTS last_heartbeat_at TIMESTAMPTZ;
|
||||
ALTER TABLE exploration_nodes ADD COLUMN IF NOT EXISTS unclaim_token TEXT;
|
||||
-- ck_node_kind: existing installs predate the 'digest' kind — recreate to allow it.
|
||||
DO $$
|
||||
BEGIN
|
||||
|
||||
Reference in New Issue
Block a user