Retrying failed recomputations locks their rows in no set order, so it can deadlock with a drain on the same rows #306

New Issue

2026-10-31T00:27:01Z

cgalo5758 commented

2026-10-31 00:27:01 +00:00

What happens

When a rule of an entitlement set changes, the console owes each affected pool a recomputation, and a drain works through them pool by pool. A recomputation that fails five times stops, and the Retry control on the entitlement set's page returns the set's failed recomputations to pending.

Retry does that with one UPDATE (RequeueFailedObligations), which locks the rows in whatever order the query plan visits them. A plan that walks the set's changes newest first locks a batch's later rule before its earlier one. A drain, and the record of a drain's failure, lock one pool's rows in the batch's order, earliest rule first.

A drain can hold those rows while they are failed. It lists a pool while its rows are pending, and if another drain records their last allowed failure before it reaches the pool, it still settles them, because it passes over settled rows only. An operator who presses Retry at that moment can hold the later rule's row and wait for the earlier one while the drain holds the earlier one and waits for the later. Postgres aborts one of them: Retry shows its failure message, or the drain records the deadlock as one more failed attempt.

docs/database-locks.md lists only core.settle_obligation among the paths that lock these rows; Retry's UPDATE locks them too.

What should happen

Retry locks the rows it returns to pending in an order that agrees with the drain's, by pool and then by the batch's rule order, as the plan acts' Retry locks its rows in ascending pool order. docs/database-locks.md lists Retry among the paths that lock these rows.

Where

internal/entitlements/queries/entitlement_set_changes.sql (RequeueFailedObligations), internal/entitlements/rule_change.go (drainPoolUnderActs, owedActs), docs/database-locks.md

Why it matters

Retry is the operator's only way out of a failed recomputation. It can fail, or cost the pool an attempt, for a reason that has nothing to do with the pool.

Related: #186 (the lock order written once), #196 (the record of a failed drain locks the obligation before the pool).

### What happens When a rule of an entitlement set changes, the console owes each affected pool a recomputation, and a drain works through them pool by pool. A recomputation that fails five times stops, and the Retry control on the entitlement set's page returns the set's failed recomputations to pending. Retry does that with one `UPDATE` (`RequeueFailedObligations`), which locks the rows in whatever order the query plan visits them. A plan that walks the set's changes newest first locks a batch's later rule before its earlier one. A drain, and the record of a drain's failure, lock one pool's rows in the batch's order, earliest rule first. A drain can hold those rows while they are failed. It lists a pool while its rows are pending, and if another drain records their last allowed failure before it reaches the pool, it still settles them, because it passes over settled rows only. An operator who presses Retry at that moment can hold the later rule's row and wait for the earlier one while the drain holds the earlier one and waits for the later. Postgres aborts one of them: Retry shows its failure message, or the drain records the deadlock as one more failed attempt. `docs/database-locks.md` lists only `core.settle_obligation` among the paths that lock these rows; Retry's `UPDATE` locks them too. ### What should happen Retry locks the rows it returns to pending in an order that agrees with the drain's, by pool and then by the batch's rule order, as the plan acts' Retry locks its rows in ascending pool order. `docs/database-locks.md` lists Retry among the paths that lock these rows. ### Where `internal/entitlements/queries/entitlement_set_changes.sql` (`RequeueFailedObligations`), `internal/entitlements/rule_change.go` (`drainPoolUnderActs`, `owedActs`), `docs/database-locks.md` ### Why it matters Retry is the operator's only way out of a failed recomputation. It can fail, or cost the pool an attempt, for a reason that has nothing to do with the pool. Related: #186 (the lock order written once), #196 (the record of a failed drain locks the obligation before the pool).

cgalo5758 added the

kind

bug

area/entitlements labels 2026-10-31 00:27:01 +00:00

cgalo5758 referenced this issue from a commit

2026-10-31 00:42:30 +00:00

Lock the pool before the obligation when settling a rule change's obligation

cgalo5758 referenced this issue

2026-10-31 00:42:39 +00:00

Recording a failed drain locks the obligation before the pool, while the drain locks the pool first #196

Sign in to join this conversation.