A rule change on a large entitlement set is shown as not applied when its first recomputation fails on one pool, although it was applied #217

New Issue

2026-10-10T23:59:47Z

cgalo5758 commented

2026-10-10 23:59:47 +00:00

What happens

A rule change on an entitlement set carried by more than 250 pools is applied in two parts. The commit stores the new rules, one change record per rule, and an obligation to recompute each pool under each of those changes. Those are durable from that point. The request then recomputes up to 50 pools within 2 seconds (its drain) and leaves the rest to the poller, which picks up outstanding obligations every 5 seconds. (A pool is the record of what one organization is allowed.)

If that drain fails on any pool, RuleChangeBatchCommit returns its result together with the drain error. The commit stands either way. ApplyEntitlementSetRules treats any error that is not a refusal as a failed apply:

It answers 422 with "Failed to apply the rule change. Details are in the server logs." and shows no toast.
The batch stays staged in the tray, so Apply changes offers to apply rules the database already holds.
The Rules section is built from the rules read before the commit, so it shows the old values, while History, read afterwards, lists the change as applied.
It does not call startEntitlementRecomputeDrain, so the remaining pools wait for the poller instead of the drain workflow, which the commit normally starts at once.

A pool can fail the drain because another transaction holds its row past the lock timeout, or because of the deadlock described in #196. The failure is recorded as an attempt, and after five attempts the obligation needs the operator's Retry.

What should happen

The spec (entitlement-set-management, "A rule change commit reaches every carrying pool") says the deferred path drains what it can before returning and leaves the rest to be settled without an operator, and that both paths report their outcome in one toast. A commit that returns a result with a drain error is answered as applied. The error is logged, the tray clears, the toast reports how far the request got ("Rule change applied. of pools recomputed."), and startEntitlementRecomputeDrain runs; it already starts the drain when Failed is above zero. RuleChangeBatchCommit returns a result only when the commit stands, so the handler can tell this case from a real failure. A handler test runs a deferred commit with one pool that fails.

Where

internal/server/operator_entitlement_set_rule_change.go: ApplyEntitlementSetRules. internal/entitlements/rule_change.go: the deferred branch at the end of RuleChangeBatchCommit and DrainChange. TestDrainObligationAttemptBudgetAndDeadLetter in internal/entitlements/rule_change_test.go covers the engine returning both a result and an error. No test covers the handler's answer.

Steps

An entitlement set is carried by more than 250 pools.
An operator stages a rule edit and clicks Apply changes while one of the first pools the request recomputes cannot be recomputed, for example because its row is held by another transaction.
The page answers "Failed to apply the rule change. Details are in the server logs." with the edit still staged, and History lists the edit as applied.

Why it matters

An operator is told a rule change failed when it is live, sees old values in the rules table, and may apply the same batch again.

### What happens A rule change on an entitlement set carried by more than 250 pools is applied in two parts. The commit stores the new rules, one change record per rule, and an obligation to recompute each pool under each of those changes. Those are durable from that point. The request then recomputes up to 50 pools within 2 seconds (its drain) and leaves the rest to the poller, which picks up outstanding obligations every 5 seconds. (A pool is the record of what one organization is allowed.) If that drain fails on any pool, `RuleChangeBatchCommit` returns its result together with the drain error. The commit stands either way. `ApplyEntitlementSetRules` treats any error that is not a refusal as a failed apply: - It answers 422 with "Failed to apply the rule change. Details are in the server logs." and shows no toast. - The batch stays staged in the tray, so Apply changes offers to apply rules the database already holds. - The Rules section is built from the rules read before the commit, so it shows the old values, while History, read afterwards, lists the change as applied. - It does not call `startEntitlementRecomputeDrain`, so the remaining pools wait for the poller instead of the drain workflow, which the commit normally starts at once. A pool can fail the drain because another transaction holds its row past the lock timeout, or because of the deadlock described in #196. The failure is recorded as an attempt, and after five attempts the obligation needs the operator's Retry. ### What should happen The spec (entitlement-set-management, "A rule change commit reaches every carrying pool") says the deferred path drains what it can before returning and leaves the rest to be settled without an operator, and that both paths report their outcome in one toast. A commit that returns a result with a drain error is answered as applied. The error is logged, the tray clears, the toast reports how far the request got ("Rule change applied. <n> of <m> pools recomputed."), and `startEntitlementRecomputeDrain` runs; it already starts the drain when `Failed` is above zero. `RuleChangeBatchCommit` returns a result only when the commit stands, so the handler can tell this case from a real failure. A handler test runs a deferred commit with one pool that fails. ### Where `internal/server/operator_entitlement_set_rule_change.go`: `ApplyEntitlementSetRules`. `internal/entitlements/rule_change.go`: the deferred branch at the end of `RuleChangeBatchCommit` and `DrainChange`. `TestDrainObligationAttemptBudgetAndDeadLetter` in `internal/entitlements/rule_change_test.go` covers the engine returning both a result and an error. No test covers the handler's answer. ### Steps 1. An entitlement set is carried by more than 250 pools. 2. An operator stages a rule edit and clicks Apply changes while one of the first pools the request recomputes cannot be recomputed, for example because its row is held by another transaction. 3. The page answers "Failed to apply the rule change. Details are in the server logs." with the edit still staged, and History lists the edit as applied. ### Why it matters An operator is told a rule change failed when it is live, sees old values in the rules table, and may apply the same batch again.

cgalo5758 added the

kind

bug

area/entitlements area/operator-ui labels 2026-10-10 23:59:47 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: wiki-cafe/member-console#217