Files
cgalo5758 6f65c744fe Pin how the plan-wide acts interleave with each other, with a rule change and with a conferral
The plan-wide acts (tier removal, default change, reorder, tier add and
delete) are now sent against each other and against a conferral, each
interleaving chosen by a lock a side transaction holds and releases once
the requests wait on it. dbtest.AwaitLockWait proves a request is queued,
and PostgreSQL grants a row lock to its waiters in the order they queued,
so no test sleeps. The kit gains Backend.Waiters, how many backends must
wait at once, with a test of its own; its first caller is these tests.

TestTierRemovalAgainstDefaultChange runs a removal and a default change
on one pool in each order: queued on the pool's row, and one after the
other. Every case ends with no live position on a deleted tier.
TestDefaultChangeTwiceSerializesOnThePool sends one default change twice
and shows one request grandfathering the pool and the other finding the
legacy grant the first wrote. TestPlanActsRefuseWhileARuleChangeHoldsTheRendezvous
holds the exclusive rendezvous while all five acts run, and pins each
act's contention answer: the removal's and the organization's "Another
entitlement set change is in progress. Try again." and the generic
sentence of the reorder, the add and the delete. It waits out the 5 s
lock timeout once. TestRenumberingCommitsDuringAConferral commits a
reorder, a tier delete and a tier removal while a default change's
conferral waits between its rank reads; the files pin the transition as
it is recorded today (upgrade 0 to 1 for the reorder where a transfer
belongs; downgrade 1 to 0 for the delete and the removal), because a
renumbering takes the rendezvous in shared mode only. Pinned with #187:
TestReorderEnactsAgainstTheRankZeroItSaved sends two reorders whose
enactments queue on one pool; the first enacts against the rank 0 it
saved after the second has put the other product back at rank 0, so it
grandfathers the holder onto a legacy grant of the product that is rank 0
again, and the second finds the holder on that grant and leaves it.

No output changed: the commit adds tests, their golden files and one field
of the test kit, and touches no production file. The names and results of
./internal/server and ./internal/testkit/..., run after a database reset at
the parent and at this commit, differ by the 13 new tests and subtests, all
passing; the new tests passed three runs in a row and 20 runs of the four
that do not wait out the timeout. The edit to the plan-wide acts' golden
test is its JSON encoder, split so the new tests encode their files the same
way; its 44 files are unchanged. The kit's wait.go and wait_test.go and the
AwaitLockWait bullet of docs/testing.md change beside it. 14 breaks, each
caught by its new test (new side only): 12 edit production code unchanged
from the parent and 2 edit this commit's wait.go. The section's tests take
about 6 s, 5 s of it the one wait.

Unprobed:
- Interleavings the tests do not choose, such as a removal that arrives
  between the default change's save and its pool lock; the cases cover the
  two queue orders on the pool and the two serial orders.
- A conferral other than the default change's during a renumbering (a
  reconcile, a grant act, the floor); it is the same database function,
  which the reorder case reaches.
- The default change's own stale candidate: a renumbering between its save
  and its enactment. It uses the same fixed product the reorder case
  reaches; no case pauses the default change there.
- The two new failure messages of AwaitLockWait for a count above one;
  they are text, read only when a wait fails.
2026-10-10 16:42:02 -05:00

98 lines
3.5 KiB
Go

// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package dbtest
import (
"database/sql"
"fmt"
"testing"
"time"
)
// lockWaitDeadline bounds how long AwaitLockWait polls, well over any wait a
// test provokes.
const lockWaitDeadline = 10 * time.Second
// Backend selects the server backend a test waits on. Exactly one of PID,
// BlockedBy and Query is set.
type Backend struct {
PID int // pg_backend_pid() of the connection that is expected to wait
BlockedBy int // pg_backend_pid() of the connection whose lock some backend waits on
Query string // text the waiting backend's current statement contains, within this database
// Waiters is how many selected backends must wait at once; zero means
// one. Two requests queued on one row lock are Waiters 2 with Query;
// BlockedBy counts only the first of them, because the second waits on
// the first's tuple lock.
Waiters int
// Abort is called between polls. A non-nil error fails the test with its
// text, so a test whose waiting call can return without ever waiting
// reports that instead of polling to the deadline.
Abort func() error
}
// waiting is the statement that counts the backends b selects that wait on a
// lock, and its argument.
func (b Backend) waiting() (string, any) {
switch {
case b.BlockedBy != 0:
return `SELECT count(*) FROM pg_stat_activity WHERE $1 = ANY(pg_blocking_pids(pid))`, b.BlockedBy
case b.Query != "":
return `SELECT count(*) FROM pg_stat_activity
WHERE datname = current_database() AND pid <> pg_backend_pid()
AND wait_event_type = 'Lock' AND position($1 in query) > 0`, b.Query
default:
return `SELECT count(*) FROM pg_stat_activity WHERE pid = $1 AND wait_event_type = 'Lock'`, b.PID
}
}
// AwaitLockWait polls pg_stat_activity until b waits on a lock (b.Waiters
// backends at once, when it is above one), so a test can assert what the
// waiting backends hold without sleeping for a fixed time. It fails the test
// when b has not waited after ten seconds or when b.Abort reports an error.
func AwaitLockWait(tb testing.TB, d *sql.DB, b Backend) {
tb.Helper()
query, arg := b.waiting()
deadline := time.Now().Add(lockWaitDeadline)
for {
var waiting int
if err := d.QueryRow(query, arg).Scan(&waiting); err != nil {
tb.Fatalf("read pg_stat_activity: %v", err)
}
if waiting >= b.waiters() {
return
}
if b.Abort != nil {
if err := b.Abort(); err != nil {
tb.Fatalf("%v", err)
}
}
if time.Now().After(deadline) {
tb.Fatal(b.neverWaited())
}
time.Sleep(5 * time.Millisecond)
}
}
// waiters is how many selected backends AwaitLockWait waits for.
func (b Backend) waiters() int {
return max(b.Waiters, 1)
}
// neverWaited is the failure message for a selection that not enough
// backends matched before the deadline.
func (b Backend) neverWaited() string {
switch {
case b.Waiters > 1 && b.BlockedBy != 0:
return fmt.Sprintf("fewer than %d backends waited on a lock held by backend %d", b.Waiters, b.BlockedBy)
case b.Waiters > 1:
return fmt.Sprintf("fewer than %d backends running a statement containing %q waited on a lock", b.Waiters, b.Query)
case b.BlockedBy != 0:
return fmt.Sprintf("no backend waited on a lock held by backend %d", b.BlockedBy)
case b.Query != "":
return fmt.Sprintf("no backend running a statement containing %q waited on a lock", b.Query)
default:
return fmt.Sprintf("backend %d never waited on a lock", b.PID)
}
}