Recording a failed drain locks the obligation before the pool, while the drain locks the pool first #196
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: wiki-cafe/member-console#196
Reference in New Issue
Block a user
What happens
When a rule change's drain fails on a pool,
recordObligationFailurerecords the attempt in a second transaction throughMarkObligationFailed, which callscore.settle_obligation. That function locks the obligation row, then the pool row. The drain itself (settlePool) locks the pool row throughLockPoolsfirst, then each obligation row it settles, anddocs/database-locks.mdstates pool before obligation as the order.Both transactions run as materializers, which share the rendezvous lock, so they can run at the same time. When the commit's own drain and the poller's drain overlap on one change, one can be recording a failure on an obligation while the other holds that obligation's pool and waits for the same obligation row. Each waits on the other, and Postgres aborts one with a deadlock error. No row is left half written:
docs/database-locks.mdalso says the failure record "takes the rendezvous alone"; it takes the obligation and pool rows as well.What should happen
The failure record locks the pool row before any obligation row, as the drain does, for example by calling
LockPoolsbeforeMarkObligationFailed.docs/database-locks.mdnames the locks it takes.Where
internal/entitlements/rule_change.go(recordObligationFailure,settlePool),core.settle_obligation,docs/database-locks.mdWhy it matters
A deadlock between two drains counts against an obligation's attempt budget as if the pool's own work had failed, and the operator sees a failure they cannot act on.
#186 is about writing the lock order once; #194 is another path that breaks it.
Recording a failed drain now locks the pool row before the obligation row, the same order as the drain and the rule change's commit. The order lives in the database function both of them call,
core.settle_obligation, which a new migration replaces: it takes the pool row right after checking the rendezvous lock, for a settlement and a recorded failure alike, and is otherwise unchanged (same owner, same grant, same writes, same attempt count). A caller that already holds the pool row takes it again without waiting. A failure recorded on an obligation another drain has just settled now waits for that drain to commit, then returns without writing. The migration's down step restores the previous function, and with it the old order.docs/database-locks.mddescribes the new order for the commit and for the drain's failure record, and no longer lists the inversion among the places where paths differ. The plan acts' own failure record needed no change: it updates one obligation row and takes no pool row. Attempt counting is unchanged. The Retry control's own lock order is a separate problem, #306.Commit:
7d8e123c