A tier reorder that commits during a conferral records the wrong kind of tier change #213

New Issue

2026-10-10T23:03:45Z

cgalo5758 commented

2026-10-10 23:03:45 +00:00

What happens

A conferral can record a tier change with the wrong type when an operator reorders the same ladder's tiers at the same moment. In the case seen, an organization grandfathered on its product by a default change is recorded as upgraded from rank 0 to rank 1 instead of transferred at the same rank. Its entitlements are right: one legacy grant and one live position, on the product it already had. The ledger row is wrong, and it is permanent. Because history rows name tiers by the ladder's current order (#211), Tier changes and Recent activity then show an upgrade from a product the organization never held.

core.confer reads ranks in separate statements: the incumbent's rank (from_rank), the same rank again for the end row, and the new product's rank through core.product_conferral_shapes (to_rank). It classifies the change by comparing them. Materializing transactions run at READ COMMITTED, so each statement sees what has committed by the time it starts.

The reorder's renumbering takes only the shared materialization rendezvous and no pool lock, so nothing orders it against a conferral running in another transaction. When it commits between the first and the last rank read, the same product reads as rank 0 and then as rank 1, and the conferral records upgrade. The pool lock that TestTierReorder_CommitRacesDefaultChangeCommit's comment relies on orders the two organization enactments against each other; the renumbering never takes it.

That test catches the race intermittently with "transfer transitions = 0, want exactly 1": about 1 run in 25 under load, and now and then on an idle machine. A probe that pauses the conferral between its rank reads makes it fail every time.

The same window is open for any conferral that runs while a tier delete or a tier removal renumbers the ladder: a subscription reconcile, a grant, or the default floor. That follows from the code; it has not been reproduced.

What should happen

A tier change is classified against one order of the ladder, the order before the reorder or the order after it, never both. A grandfather on the same product records transfer.

Opening the renumbering transactions with BeginRuleChange (the exclusive rendezvous, which orders a rule change against every materializing transaction) does this. With it on the reorder, the paused probe records transfer, and the natural race failed 0 of 600 runs against 17 of 600 without it. The cost is that a reorder can wait up to the rendezvous lock timeout (5s) behind conferrals in flight. The tier entries in docs/database-locks.md and the test's comment change with it. Reading every rank once inside core.confer is the larger alternative: a migration replacing the function, and it would not cover the default floor's own rank-0 read.

Where

internal/server/operator_plan_ladders.go: ReorderPlanLadderTiers, DeletePlanLadderTier and CommitPlanLadderTierRemoval, which open their transactions with entitlements.BeginMaterializing; core.confer in internal/db/migrations/00023_grant_resumption.sql.

Steps

An organization type's default is a ladder's rank-0 tier, and an organization holds it.
An operator changes the default and grandfathers the organization. At the same moment another operator reorders that ladder so the old default moves to rank 1.
If the reorder commits while the grandfather's conferral is between its rank reads, Tier changes shows an upgrade from rank 0 to rank 1 where a transfer belongs.

Why it matters

The tier-change ledger is permanent, and a wrong type in it shows operators an upgrade that never happened, naming a product the organization never held.

## What happens A conferral can record a tier change with the wrong type when an operator reorders the same ladder's tiers at the same moment. In the case seen, an organization grandfathered on its product by a default change is recorded as upgraded from rank 0 to rank 1 instead of transferred at the same rank. Its entitlements are right: one legacy grant and one live position, on the product it already had. The ledger row is wrong, and it is permanent. Because history rows name tiers by the ladder's current order (#211), Tier changes and Recent activity then show an upgrade from a product the organization never held. `core.confer` reads ranks in separate statements: the incumbent's rank (`from_rank`), the same rank again for the `end` row, and the new product's rank through `core.product_conferral_shapes` (`to_rank`). It classifies the change by comparing them. Materializing transactions run at READ COMMITTED, so each statement sees what has committed by the time it starts. The reorder's renumbering takes only the shared materialization rendezvous and no pool lock, so nothing orders it against a conferral running in another transaction. When it commits between the first and the last rank read, the same product reads as rank 0 and then as rank 1, and the conferral records `upgrade`. The pool lock that `TestTierReorder_CommitRacesDefaultChangeCommit`'s comment relies on orders the two organization enactments against each other; the renumbering never takes it. That test catches the race intermittently with "transfer transitions = 0, want exactly 1": about 1 run in 25 under load, and now and then on an idle machine. A probe that pauses the conferral between its rank reads makes it fail every time. The same window is open for any conferral that runs while a tier delete or a tier removal renumbers the ladder: a subscription reconcile, a grant, or the default floor. That follows from the code; it has not been reproduced. ## What should happen A tier change is classified against one order of the ladder, the order before the reorder or the order after it, never both. A grandfather on the same product records `transfer`. Opening the renumbering transactions with `BeginRuleChange` (the exclusive rendezvous, which orders a rule change against every materializing transaction) does this. With it on the reorder, the paused probe records `transfer`, and the natural race failed 0 of 600 runs against 17 of 600 without it. The cost is that a reorder can wait up to the rendezvous lock timeout (5s) behind conferrals in flight. The tier entries in `docs/database-locks.md` and the test's comment change with it. Reading every rank once inside `core.confer` is the larger alternative: a migration replacing the function, and it would not cover the default floor's own rank-0 read. ## Where `internal/server/operator_plan_ladders.go`: `ReorderPlanLadderTiers`, `DeletePlanLadderTier` and `CommitPlanLadderTierRemoval`, which open their transactions with `entitlements.BeginMaterializing`; `core.confer` in `internal/db/migrations/00023_grant_resumption.sql`. ## Steps 1. An organization type's default is a ladder's rank-0 tier, and an organization holds it. 2. An operator changes the default and grandfathers the organization. At the same moment another operator reorders that ladder so the old default moves to rank 1. 3. If the reorder commits while the grandfather's conferral is between its rank reads, Tier changes shows an upgrade from rank 0 to rank 1 where a transfer belongs. ## Why it matters The tier-change ledger is permanent, and a wrong type in it shows operators an upgrade that never happened, naming a product the organization never held.

cgalo5758 added the

kind

bug

area/entitlements area/catalog labels 2026-10-10 23:03:45 +00:00

cgalo5758 referenced this issue

2026-10-10 23:21:52 +00:00

Plan-wide changes to ladders and defaults run inside HTTP handlers, each with its own transaction shape #187

cgalo5758 referenced this issue

2026-10-10 23:59:47 +00:00

Adding, deleting or reordering tiers while the entitlement lock is held answers "see server logs" instead of the refusal that says to try again #216

cgalo5758 referenced this issue from a commit

2026-10-11 00:39:00 +00:00

Order an operator's tier changes against every conferral

cgalo5758 commented

2026-10-11 00:39:06 +00:00

Fixed in a3041242. The tier add, the tier delete, the tier removal and the reorder's renumbering now open their transactions with the exclusive materialization lock (BeginRuleChange). A conferral reads a ladder's ranks and a product's shape in several statements; it now finishes before a tier change or starts after it, so it classifies every tier change against one order of the ladder. The tier add is included because it grows the added product's shape, which the conferral also reads more than once.

TestTierChangesWaitForAConferral holds a conferral open and checks that each of the four tier changes waits for it, and the race test passed 500 runs idle and 500 under load. The entitlements spec now names the four tier changes among the exclusive holders.

The cost: a tier change waits for every conferral, grant, reconcile and recomputation in flight, up to the 5 s lock timeout, and new ones wait behind it while it runs. When that wait times out, the add, the delete and the reorder still answer with their generic failure copy; #216 tracks that.

Fixed in a3041242. The tier add, the tier delete, the tier removal and the reorder's renumbering now open their transactions with the exclusive materialization lock (`BeginRuleChange`). A conferral reads a ladder's ranks and a product's shape in several statements; it now finishes before a tier change or starts after it, so it classifies every tier change against one order of the ladder. The tier add is included because it grows the added product's shape, which the conferral also reads more than once. `TestTierChangesWaitForAConferral` holds a conferral open and checks that each of the four tier changes waits for it, and the race test passed 500 runs idle and 500 under load. The entitlements spec now names the four tier changes among the exclusive holders. The cost: a tier change waits for every conferral, grant, reconcile and recomputation in flight, up to the 5 s lock timeout, and new ones wait behind it while it runs. When that wait times out, the add, the delete and the reorder still answer with their generic failure copy; #216 tracks that.

cgalo5758 closed this issue

2026-10-11 00:39:07 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: wiki-cafe/member-console#213