Recording a failed drain locks the obligation before the pool, while the drain locks the pool first #196

New Issue

2026-10-09T03:50:02Z

cgalo5758 commented

2026-10-09 03:50:02 +00:00

What happens

When a rule change's drain fails on a pool, recordObligationFailure records the attempt in a second transaction through MarkObligationFailed, which calls core.settle_obligation. That function locks the obligation row, then the pool row. The drain itself (settlePool) locks the pool row through LockPools first, then each obligation row it settles, and docs/database-locks.md states pool before obligation as the order.

Both transactions run as materializers, which share the rendezvous lock, so they can run at the same time. When the commit's own drain and the poller's drain overlap on one change, one can be recording a failure on an obligation while the other holds that obligation's pool and waits for the same obligation row. Each waits on the other, and Postgres aborts one with a deadlock error. No row is left half written:

If the failure record is aborted, the attempt goes unrecorded and the obligation stays pending for the next pass.
If the drain is aborted, it records the deadlock as a failed attempt. Five failed attempts make the obligation a dead letter that only the operator's Retry clears.

docs/database-locks.md also says the failure record "takes the rendezvous alone"; it takes the obligation and pool rows as well.

What should happen

The failure record locks the pool row before any obligation row, as the drain does, for example by calling LockPools before MarkObligationFailed. docs/database-locks.md names the locks it takes.

Where

internal/entitlements/rule_change.go (recordObligationFailure, settlePool), core.settle_obligation, docs/database-locks.md

Why it matters

A deadlock between two drains counts against an obligation's attempt budget as if the pool's own work had failed, and the operator sees a failure they cannot act on.

#186 is about writing the lock order once; #194 is another path that breaks it.

### What happens When a rule change's drain fails on a pool, `recordObligationFailure` records the attempt in a second transaction through `MarkObligationFailed`, which calls `core.settle_obligation`. That function locks the obligation row, then the pool row. The drain itself (`settlePool`) locks the pool row through `LockPools` first, then each obligation row it settles, and `docs/database-locks.md` states pool before obligation as the order. Both transactions run as materializers, which share the rendezvous lock, so they can run at the same time. When the commit's own drain and the poller's drain overlap on one change, one can be recording a failure on an obligation while the other holds that obligation's pool and waits for the same obligation row. Each waits on the other, and Postgres aborts one with a deadlock error. No row is left half written: - If the failure record is aborted, the attempt goes unrecorded and the obligation stays pending for the next pass. - If the drain is aborted, it records the deadlock as a failed attempt. Five failed attempts make the obligation a dead letter that only the operator's Retry clears. `docs/database-locks.md` also says the failure record "takes the rendezvous alone"; it takes the obligation and pool rows as well. ### What should happen The failure record locks the pool row before any obligation row, as the drain does, for example by calling `LockPools` before `MarkObligationFailed`. `docs/database-locks.md` names the locks it takes. ### Where `internal/entitlements/rule_change.go` (`recordObligationFailure`, `settlePool`), `core.settle_obligation`, `docs/database-locks.md` ### Why it matters A deadlock between two drains counts against an obligation's attempt budget as if the pool's own work had failed, and the operator sees a failure they cannot act on. #186 is about writing the lock order once; #194 is another path that breaks it.

cgalo5758 added the

kind

bug

area/entitlements labels 2026-10-09 03:50:02 +00:00

cgalo5758 referenced this issue

2026-10-09 03:50:18 +00:00

Locks are written out at each call site, with four key conventions and no written order #186

cgalo5758 referenced this issue

2026-10-10 23:59:47 +00:00

A rule change on a large entitlement set is shown as not applied when its first recomputation fails on one pool, although it was applied #217

Sign in to join this conversation.