Bind each FedWiki request's steps to the site it was made for

#284. The record steps of a status change and of a swap found the site
row by its name. When the row had been deleted under an open request,
the lock found nothing and the step failed with a retryable error, so it
retried for ten years: the request stayed Pending and held a slot of the
organization's site limit. If a new site was later created with the same
name, the step wrote the old request's outcome onto it.

Each step now locks and updates the site by the id the request already
records, and checks the site's workspace and domain under the lock; no
schema, workflow input or request parameter changes. A status request
whose site row is gone, found at the slot gate, the farm authorization,
the reconcile load or the record step, ends declined with site_missing
and is not retried; the transaction that finishes it removes any hold
and recounts usage. When either site of a swap is gone, neither local
status is written, the surviving original site is sent back through the
existing restoration activity (which keeps its retries while the farm
cannot answer), and both requests end declined with site_missing.

The intake refuses a new status or swap request whose site the retention
purge deleted before the intake took the organization's instance lock:
it reads the site by id under that existing lock, in a READ COMMITTED
transaction, with no new row lock. Repeated requests, permanent delete
and the API's answers are unchanged.

Strings (approved): "<domain> was deleted." for a request whose site, or
a swap's activated site, was deleted; "<domain> wasn't kept active.
<parked domain> was deleted." when only the site a swap would make
read-only was deleted. A refused intake answers the existing "Site not
found", or for a swap's parked site the existing "This site changed
since the list loaded.".

swapActiveSite's complexity fell to the limit, so it left the
complexity exemption list that #174 tracks.

Upgrading a deployment with requests already stuck:
1. Before deploying, count read-only the accepted status and swap
   requests whose site row no longer exists.
2. Terminate their stuck record-step runs, then deploy. Boot's settle
   pass and its follow-up two minutes later finish them through the
   tested path: a held activation reaches the status reconcile and
   declines with site_missing; other closed requests, swaps included,
   end failed with retries_exhausted.
3. Check that none of them remains accepted, and inspect by hand any
   replacement row an old request already wrote to and the farm status
   of each surviving swap site.

Full suite on reset databases, on this commit's tree: 5562 passed, 16
skipped, none failed. The two new member
notices were captured by hand at 1280 and 390 wide: both in the
warning tone, the parked site's domain shown, and Dismiss clears them;
no other page changes.

Breaks, all CAUGHT (64):
- status record follows reused domain
- missing record retries
- status record missing fails
- omit record WorkspaceID guard
- omit record Domain guard
- database failure becomes missing
- committed success reapplies
- missing request classified missing site
- record mismatch fails SetFedWikiSiteStatusRequestWorkflow
- record mismatch fails ReconcileFedWikiSiteStatusWorkflow
- finish retains holds
- finish omits recount
- finish commits guard failure
- swap record follows reused names
- swap missing record retries
- swap finishes only activation
- swap skips record restoration
- swap record mismatch skips restoration
- swap finish omits recount
- swap skips farm-missing restoration
- swap record missing skips restoration retry
- restoration retries bounded
- swap never finishes after restoration
- restoration calls missing or mismatched originals
- 403 retries restoration
- 404 retries restoration
- activation with no row treated as create
- held activation skips site check
- missing failure branch gateActivation
- missing failure branch statusChangeFailed
- missing failure branch compensateStatusChange
- missing reconcile load maps to failure
- missing reconcile record maps to failure
- missing reconcile load continues
- missing authorization continues
- gate missing mapped to generic error
- intake bypass status
- intake bypass rotation
- swap skips protected validation
- rotation stamps refused cooldown
- intake checks deleted row before repeat
- intake read error becomes missing
- intake changed repeat accepted
- park missing primary response
- primary missing refreshes
- park missing fails to refresh
- intake reads before instance lock
- purge ignores accepted request
- single intake default isolation
- handler intake default isolation
- missing reason uses action notice
- missing peer uses activation domain
- replacement mistaken for original
- swap shows peer notice
- swap dismiss leaves peer visible
- missing notice danger
- missing outcome marks replacement pending
- handler checks site before recorded repeat
- handler accepts changed repeat
- database failure becomes another non-retryable refusal
- restoration sends empty status to deleted originals
- missing rotation logs an error
- missing swap logs an error
- status record database failure loses context
This commit is contained in:
2026-10-30 17:00:38 -05:00
parent 5c454adc91
commit 9767f9ac27
34 changed files with 2313 additions and 108 deletions
-3
View File
@@ -47,9 +47,6 @@ linters:
- path: '^internal/integrations/fedwiki/web/api\.go$'
linters: [gocognit]
text: 'cognitive complexity 48 of func `\(\*FedWikiHandler\)\.CreateSite`' # #174
- path: '^internal/integrations/fedwiki/web/site_actions\.go$'
linters: [gocognit]
text: 'cognitive complexity 33 of func `\(\*FedWikiPartialsHandler\)\.swapActiveSite`' # #174
# #205: each rule pins the calls on one line of source text; the ladder_id text is on two lines, one in each of PostKeep and PostCancel.
- path: '^internal/server/billing\.go$'
linters: [forbidigo]
+14 -4
View File
@@ -53,7 +53,7 @@ flag-only writes change neither claims nor allocation counts.
| Org type row | `core.org_types` row of one type | `FOR NO KEY UPDATE` | the `LockOrgType` query, run by the default change's T0 | The T0 against the demonstration seed and an edit of the type's other columns, which write the row without the rendezvous. Every other writer of the default already waits on the exclusive rendezvous. `FOR NO KEY UPDATE` does not conflict with the `FOR KEY SHARE` an organization insert takes on its type's row. |
| Grant lineage head | the `core.grants` row no other grant extends | `FOR UPDATE OF g` | the `GetGrantLineageHeadForUpdate` query, run by `RevokeGrantTx`, `ExtendWaitingGrantTx` and `ManageGrantTx` | Acts on one lineage, so an extension that committed while another act waited is visible to it. |
| Usage row | `core.numeric_entitlement_usage` row of one (pool, resource key) | `FOR UPDATE` | the `LockNumericUsage` query, run by `usage.Recount` and by the slot gate, after the site rows the transaction writes and before the request rows it finishes (the hold rows have no place in the order, see rule 5); the count is a statement of its own after the lock | Two recounts of one pool, so each count reads every change committed before its lock was granted; two slot gates for one pool, so the second counts the first's hold and two creates cannot both take the last slot. |
| FedWiki site row | `fedwiki.sites` row of one domain or one site id | `FOR UPDATE`; `FOR SHARE` by `GetSiteForPlacement` | the `LockSiteStatusByDomain` query, run by `sweepLocally`, `RecordSiteStatusActivity`, `RecordSwapActivity` and the sync's status write (`writeStatus`), and the `LockSiteStatusByID` query, run by the slot gate for an activation, and `LockArchivedSiteForPurge`, run by retention; `GetSiteForPlacement`, run by `reconcileSite` and `syncPlacement` inside `Registry.WithLock` immediately before the first claim or placement write | Paths that move one site (a member's request, the downgrade sweep and the farm sync), so each writes the status it finds under the lock and the recount that follows counts the row as it ended; the sync's reading of the site's requests, which therefore reflects every request that committed before the lock was granted; the gate's reading of an activation whose row the sweep already made active; retention's eligibility check through deletion; and a status write or deletion against creation of that site's placement, so the writer's trigger sees the placement. |
| FedWiki site row | `fedwiki.sites` row of one domain or one site id | `FOR UPDATE`; `FOR SHARE` by `GetSiteForPlacement` | the `LockSiteStatusByDomain` query, run by `sweepLocally` and the sync's status write (`writeStatus`), and the `LockSiteStatusByID` query, run by the slot gate and by `RecordSiteStatusActivity` and `RecordSwapActivity` (which check the recorded workspace and domain under it), and `LockArchivedSiteForPurge`, run by retention; `GetSiteForPlacement`, run by `reconcileSite` and `syncPlacement` inside `Registry.WithLock` immediately before the first claim or placement write | Paths that move one site (a member's request, the downgrade sweep and the farm sync), so each writes the status it finds under the lock and the recount that follows counts the row as it ended; the sync's reading of the site's requests, which therefore reflects every request that committed before the lock was granted; the gate's reading of an activation whose row the sweep already made active; retention's eligibility check through deletion; and a status write or deletion against creation of that site's placement, so the writer's trigger sees the placement. |
| FedWiki placement row | `domains.placements` rows with provider `fedwiki` and the site's id | Row lock of the trigger's UPDATE, only where the flag changes | `fedwiki.set_site_placement_servability`, after the site row and before usage | Nothing beyond the site row; flag-only, outside the registry lock. |
| Desired-state record | `core.desired_state_records` row | `FOR UPDATE` | the `LockDesiredStateRecord` query (after the address lock); `StampDesiredStateRecordsDispatched`, which locks the records in identifier order | The record's content and generation within one recompute; the dispatch stamp against another stamp. |
| Desired-state notice | `core.desired_state_outbox` rows | `FOR UPDATE SKIP LOCKED` | the `ClaimDesiredStateOutbox` query | Two relay passes, so each claims different rows; a pass skips rows another still holds. Triggers write the rows through `core.insert_desired_state_notice` inside the transaction that changed the source, and they take no lock beyond the insert. |
@@ -386,6 +386,11 @@ parked site, which takes that instance's lock after the first. The swap policy
row is upserted last. The handler takes no usage-row lock and no site-row
lock. A status change without a cooldown skips the workspace lock and records
its request in a transaction of its own, taking only the instance lock.
Both intake transaction openers use READ COMMITTED. Status intake reads the
original site by ID after acquiring its instance lock, so it sees a purge
that committed before that lock was granted. A swap takes both instance
locks before validating either site and computes the park generation under
its lock. These reads take no site-row lock.
### At the slot gate
@@ -404,15 +409,20 @@ transactions wait on one hold row. The gate takes no workspace advisory lock.
The workflow activities that finish a lifecycle request, and settle, run
`finishAllWith`, one transaction at READ COMMITTED: the lifecycle instance
locks, in the order the handler recorded the requests (the activated site
first, then the parked site); then the activity's own site-row writes; then
first, then the parked site); then the activity's own site-row writes
(status and swap records lock and validate the original IDs, park then
activate for a swap); then
the requests' holds are deleted (`DeleteSlotHolds`); then the usage row
(`usage.Recount`: `LockNumericUsage`, the count, the write); then the swap's
guard; then the request rows (`FinishLifecycleRequest`).
- `RecordCreatedSiteActivity`: the site row insert.
- `RecordSiteStatusActivity`: the site row (`LockSiteStatusByDomain`).
- `RecordSiteStatusActivity`: the site row by id (`LockSiteStatusByID`),
checked against the recorded workspace and domain.
- `RecordSwapActivity`: the site row of the parked site, then the site row of
the activated site; the guard reads the pool's limit after the recount.
the activated site, both by id (`LockSiteStatusByID`) and checked against
the recorded workspace and domain; the guard reads the pool's limit after
the recount.
- `RecordSiteDeletedActivity` and `FinishLifecycleRequestsActivity` (every
declined or failed outcome): no site row.
- `DeleteFedWikiSiteActivity` deletes the site row and recounts in a
@@ -153,6 +153,19 @@ func (f *Farm) LastPatch(domain string) string {
return last
}
// DomainAttempts counts every status PATCH to a domain, including empty statuses.
func (f *Farm) DomainAttempts(domain string) int {
f.mu.Lock()
defer f.mu.Unlock()
total := 0
for key, count := range f.attempts {
if strings.HasPrefix(key, domain+"=") {
total += count
}
}
return total
}
// Attempts counts the requests to set domain to status that reached the farm,
// refused ones included.
func (f *Farm) Attempts(domain, status string) int {
@@ -0,0 +1,7 @@
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package fedwiki
// RecordSiteRequestOn exposes protected-read failure injection only to tests.
var RecordSiteRequestOn = recordSiteRequestInTx
@@ -14,7 +14,7 @@ import (
)
type Querier interface {
ClearForceReducedByDomain(ctx context.Context, domain string) error
ClearForceReducedByID(ctx context.Context, siteID string) error
// The active sites of every workspace whose active primary assignment is the
// pool, and the holds that count beside them, in one statement. Usage is
// pool-grain: an organization's workspaces draw down one `fedwiki_sites`
@@ -147,20 +147,24 @@ type Querier interface {
// The status of the site a request's instance names, its row locked until the
// transaction ends. The slot gate reads it for an activation, so a site the
// sweep or the sync already made active is stamped without a comparison with
// the limit. No row (a create's instance) is sql.ErrNoRows.
LockSiteStatusByID(ctx context.Context, siteID string) (string, error)
// the limit. Status and swap record steps check the recorded workspace and
// domain under this lock. No row is sql.ErrNoRows.
LockSiteStatusByID(ctx context.Context, siteID string) (LockSiteStatusByIDRow, error)
SetForceReducedByDomain(ctx context.Context, domain string) error
SetForceReducedByID(ctx context.Context, siteID string) error
// The farm's storage and last-modified time for a site whose listed status
// equals the row's. It writes no status, so no count changes; its trigger
// corrects the placement flag. No row is sql.ErrNoRows.
SetSiteObservedByDomain(ctx context.Context, arg SetSiteObservedByDomainParams) (Site, error)
// Project maintenance status changes from the sync and quota sweep.
SetSiteStatusByDomain(ctx context.Context, arg SetSiteStatusByDomainParams) (Site, error)
// Project a lifecycle status change locally. archived_at is maintained by
// the trigger function fedwiki.set_site_archived_at on every site write.
// force_reduced_at marks a site a downgrade left read-only, so it is kept
// only while the site stays
// `readonly`: a site that becomes active or archived is no longer parked,
// and the re-upgrade sweep must not bring it back.
SetSiteStatusByDomain(ctx context.Context, arg SetSiteStatusByDomainParams) (Site, error)
SetSiteStatusByID(ctx context.Context, arg SetSiteStatusByIDParams) (Site, error)
// Whether a lifecycle request on the site is still accepted. The downgrade
// sweep leaves such a site alone, as the sync leaves an open create's name:
// the member's request decides its status.
@@ -104,12 +104,7 @@ ON CONFLICT(domain) DO NOTHING
RETURNING *;
-- name: SetSiteStatusByDomain :one
-- Project a lifecycle status change locally. archived_at is maintained by
-- the trigger function fedwiki.set_site_archived_at on every site write.
-- force_reduced_at marks a site a downgrade left read-only, so it is kept
-- only while the site stays
-- `readonly`: a site that becomes active or archived is no longer parked,
-- and the re-upgrade sweep must not bring it back.
-- Project maintenance status changes from the sync and quota sweep.
UPDATE fedwiki.sites
SET status = $2,
force_reduced_at = CASE WHEN $2 = 'readonly' THEN force_reduced_at END,
@@ -193,11 +188,6 @@ UPDATE fedwiki.sites
SET force_reduced_at = NOW(), updated_at = NOW()
WHERE domain = $1 AND force_reduced_at IS NULL;
-- name: ClearForceReducedByDomain :exec
UPDATE fedwiki.sites
SET force_reduced_at = NULL, updated_at = NOW()
WHERE domain = $1;
-- name: ListActiveSitesByWorkspace :many
-- Active sites ranked by recorded modification or creation time, newest first.
SELECT * FROM fedwiki.sites
@@ -272,8 +262,9 @@ SELECT
-- The status of the site a request's instance names, its row locked until the
-- transaction ends. The slot gate reads it for an activation, so a site the
-- sweep or the sync already made active is stamped without a comparison with
-- the limit. No row (a create's instance) is sql.ErrNoRows.
SELECT status FROM fedwiki.sites
-- the limit. Status and swap record steps check the recorded workspace and
-- domain under this lock. No row is sql.ErrNoRows.
SELECT status, force_reduced_at, workspace_id, domain FROM fedwiki.sites
WHERE site_id = sqlc.arg(site_id)::uuid
FOR UPDATE;
@@ -407,3 +398,27 @@ WHERE (sqlc.narg(status)::text IS NULL OR s.status = sqlc.narg(status)::text)
OR o.name ILIKE '%' || sqlc.narg(q)::text || '%')
ORDER BY s.domain ASC
LIMIT sqlc.arg(page_limit)::int OFFSET sqlc.arg(page_offset)::int;
-- name: SetSiteStatusByID :one
-- Project a lifecycle status change locally. archived_at is maintained by
-- the trigger function fedwiki.set_site_archived_at on every site write.
-- force_reduced_at marks a site a downgrade left read-only, so it is kept
-- only while the site stays
-- `readonly`: a site that becomes active or archived is no longer parked,
-- and the re-upgrade sweep must not bring it back.
UPDATE fedwiki.sites
SET status = $2,
force_reduced_at = CASE WHEN $2 = 'readonly' THEN force_reduced_at END,
updated_at = NOW()
WHERE site_id = $1
RETURNING *;
-- name: SetForceReducedByID :exec
UPDATE fedwiki.sites
SET force_reduced_at = NOW(), updated_at = NOW()
WHERE site_id = $1 AND force_reduced_at IS NULL;
-- name: ClearForceReducedByID :exec
UPDATE fedwiki.sites
SET force_reduced_at = NULL, updated_at = NOW()
WHERE site_id = $1;
@@ -0,0 +1,83 @@
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package fedwiki
import (
"context"
"database/sql"
"errors"
"fmt"
"git.coopcloud.tech/wiki-cafe/member-console/internal/db"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integration"
)
// MissingSiteError identifies a status target absent from its recorded identity.
type MissingSiteError struct{ InstanceID string }
func (err *MissingSiteError) Error() string { return "request site not found: " + err.InstanceID }
// RecordSiteRequest serializes intake with purge. READ COMMITTED makes the
// protected read see the commit preceding the instance-lock grant.
func RecordSiteRequest(ctx context.Context, database db.Beginner, spec integration.LifecycleRequestSpec, domain string) (integration.LifecycleAnswer, error) {
var answer integration.LifecycleAnswer
err := db.InTx(ctx, database, db.ReadCommitted, func(tx *sql.Tx) error {
var err error
answer, err = RecordSiteRequestInTx(ctx, tx, spec, domain)
return err
})
return answer, err
}
// RecordSiteRequestInTx validates new status intent under the instance lock.
// The caller supplies a READ COMMITTED transaction and rolls back on refusal.
func RecordSiteRequestInTx(ctx context.Context, tx *sql.Tx, spec integration.LifecycleRequestSpec, domain string) (integration.LifecycleAnswer, error) {
return recordSiteRequestInTx(ctx, tx, spec, domain)
}
func recordSiteRequestInTx(ctx context.Context, tx DBTX, spec integration.LifecycleRequestSpec, domain string) (integration.LifecycleAnswer, error) {
queries := integration.New(tx)
if err := queries.LockLifecycleInstance(ctx, spec.InstanceID); err != nil {
return integration.LifecycleAnswer{}, fmt.Errorf("lock lifecycle instance: %w", err)
}
_, _, found, err := integration.LookupLifecycleRequest(ctx, tx, spec.WorkspaceID, spec.InstanceID, spec.Generation)
if err != nil {
return integration.LifecycleAnswer{}, err
}
if !found {
if err := checkRequestSite(ctx, tx, spec.InstanceID, spec.WorkspaceID, domain); err != nil {
return integration.LifecycleAnswer{}, err
}
}
return integration.RecordLifecycleRequest(ctx, tx, spec)
}
// checkRequestSite reads an original site under an instance lock the caller holds.
func checkRequestSite(ctx context.Context, tx DBTX, instanceID, workspaceID, domain string) error {
row, err := New(tx).GetSiteByID(ctx, instanceID)
if errors.Is(err, sql.ErrNoRows) {
return &MissingSiteError{InstanceID: instanceID}
}
if err != nil {
return fmt.Errorf("read request site: %w", err)
}
if row.WorkspaceID != workspaceID || row.Domain != domain {
return &MissingSiteError{InstanceID: instanceID}
}
return nil
}
// CheckSwapRequestSites takes both existing instance locks before reading either site.
// The caller supplies a READ COMMITTED transaction and records activation then park.
func CheckSwapRequestSites(ctx context.Context, tx *sql.Tx, workspaceID, activateID, activateDomain, parkID, parkDomain string) error {
for _, instanceID := range []string{activateID, parkID} {
if err := integration.New(tx).LockLifecycleInstance(ctx, instanceID); err != nil {
return err
}
}
if err := checkRequestSite(ctx, tx, activateID, workspaceID, activateDomain); err != nil {
return err
}
return checkRequestSite(ctx, tx, parkID, workspaceID, parkDomain)
}
@@ -0,0 +1,222 @@
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package fedwiki_test
import (
"context"
"database/sql"
"errors"
"fmt"
"strings"
"testing"
"time"
"git.coopcloud.tech/wiki-cafe/member-console/internal/db"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integration"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/fedwikitest"
fwmod "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/store"
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest"
)
// failedSiteRead makes only the protected site read fail inside a real transaction.
type failedSiteRead struct{ fwmod.DBTX }
func (queries failedSiteRead) QueryRowContext(ctx context.Context, query string, args ...interface{}) *sql.Row {
if strings.Contains(query, "-- name: GetSiteByID") {
return queries.DBTX.QueryRowContext(ctx, "SELECT 1/0")
}
return queries.DBTX.QueryRowContext(ctx, query, args...)
}
// intakeOrderCase states the transaction order and its expected refusal.
type intakeOrderCase struct {
name string
intakeFirst, repeatable, runtime, readError, swap bool
wantMissing bool
}
func TestIntakeAndPurgeOrder_DB(t *testing.T) {
for _, row := range []intakeOrderCase{
{name: "purge first", wantMissing: true}, {name: "intake first", intakeFirst: true},
{name: "repeatable", repeatable: true, wantMissing: true}, {name: "runtime", runtime: true, wantMissing: true},
{name: "protected read error", readError: true},
{name: "swap", swap: true, wantMissing: true},
} {
t.Run(row.name, func(t *testing.T) {
checkIntakeOrder(t, row)
})
}
}
// checkIntakeOrder checks the protected read after the preceding intake or purge commits.
func checkIntakeOrder(t *testing.T, row intakeOrderCase) {
t.Helper()
database := dbtest.Open(t)
env := fedwikitest.New(t, database)
tenant := env.Counted("Intake", 2, 0)
domain := "intake-" + env.Sfx + ".localhost"
instance := fedwikitest.Site(t, database, tenant.Workspace.ID, domain, "archived")
var peerID string
peerDomain := "peer-" + domain
if row.swap {
peerID = fedwikitest.Site(t, database, tenant.Workspace.ID, peerDomain, "active")
}
generations, err := integration.NextGenerations(env.Ctx, database, []string{instance})
if err != nil {
t.Fatal(err)
}
spec := integration.LifecycleRequestSpec{InstanceID: instance, Generation: generations[instance], WorkspaceID: tenant.Workspace.ID,
ConnectionID: env.Conn, ResourceKey: "fedwiki_sites", Operation: integration.LifecycleSetStatus, Parameters: []byte(`{"status":"active"}`), WorkflowID: "intake-" + instance}
t.Cleanup(func() {
dbtest.On(t, database).Exec("DELETE FROM core.lifecycle_requests WHERE instance_id = $1", instance)
})
if row.readError {
checkProtectedReadError(t, database, spec, domain, row.name)
return
}
intakeDB := database
if row.repeatable {
intakeDB = intakeDatabase(t, "default_transaction_isolation=repeatable%20read")
}
if row.runtime {
intakeDB = intakeDatabase(t, "options=-c%20role%3Dconsole_runtime")
}
holder, err := db.Begin(env.Ctx, database, db.ReadCommitted)
if err != nil {
t.Fatal(err)
}
defer holder.Rollback()
pid := int(dbtest.On(t, holder).Int("SELECT pg_backend_pid()"))
if err := integration.New(holder).LockLifecycleInstance(env.Ctx, instance); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(env.Ctx, 10*time.Second)
defer cancel()
done := make(chan error, 1)
if row.intakeFirst {
if _, err := fwmod.RecordSiteRequestInTx(ctx, holder, spec, domain); err != nil {
t.Fatal(err)
}
go func() {
done <- deleteWithoutRequest(ctx, intakeDB, instance)
}()
} else {
if err := fwmod.New(holder).DeleteSite(ctx, instance); err != nil {
t.Fatal(err)
}
go func() { done <- recordIntakePair(ctx, intakeDB, spec, domain, peerID, peerDomain) }()
}
dbtest.AwaitLockWait(t, database, dbtest.Backend{BlockedBy: pid, Abort: func() error {
select {
case err := <-done:
return fmt.Errorf("%s: lock waiter completed before the holder committed: %v", row.name, err)
default:
return nil
}
}})
if err := holder.Commit(); err != nil {
t.Fatal(err)
}
err = <-done
if !row.wantMissing {
if err != nil {
t.Fatal(err)
}
} else {
var missing *fwmod.MissingSiteError
if !errors.As(err, &missing) {
t.Errorf("%s: intake crossed purge without refusing: %v", row.name, err)
if row.repeatable {
t.Errorf("%s: intake did not use READ COMMITTED", row.name)
}
}
}
}
// intakeDatabase opens a handle with a session setting before any transaction.
func intakeDatabase(t *testing.T, setting string) *sql.DB {
t.Helper()
dsn := dbtest.DSN(t)
separator := "?"
if strings.Contains(dsn, "?") {
separator = "&"
}
return dbtest.Connect(t, dsn+separator+setting)
}
// checkProtectedReadError distinguishes database failure from a refused target.
func checkProtectedReadError(t *testing.T, database *sql.DB, spec integration.LifecycleRequestSpec, domain, name string) {
t.Helper()
tx, err := db.Begin(t.Context(), database, db.ReadCommitted)
if err != nil {
t.Fatal(err)
}
defer tx.Rollback()
_, err = fwmod.RecordSiteRequestOn(t.Context(), failedSiteRead{tx}, spec, domain)
var missing *fwmod.MissingSiteError
if err == nil || errors.As(err, &missing) {
t.Errorf("%s: protected read hid a database error as not found", name)
}
}
// TestSiteRequestRepeats_DB answers recorded content before checking a deleted row.
func TestSiteRequestRepeats_DB(t *testing.T) {
database := dbtest.Open(t)
env := fedwikitest.New(t, database)
tenant := env.Counted("Repeat", 2, 0)
domain := "repeat-" + env.Sfx + ".localhost"
instance := fedwikitest.Site(t, database, tenant.Workspace.ID, domain, "active")
gens, err := integration.NextGenerations(env.Ctx, database, []string{instance})
if err != nil {
t.Fatal(err)
}
spec := integration.LifecycleRequestSpec{InstanceID: instance, Generation: gens[instance], WorkspaceID: tenant.Workspace.ID, ConnectionID: env.Conn, ResourceKey: "fedwiki_sites", Operation: integration.LifecycleSetStatus, Parameters: []byte(`{"status":"archived"}`), WorkflowID: "repeat-" + instance}
t.Cleanup(func() {
dbtest.On(t, database).Exec("DELETE FROM core.lifecycle_requests WHERE instance_id=$1", instance)
})
if answer, err := fwmod.RecordSiteRequest(env.Ctx, database, spec, domain); err != nil || answer.Kind != integration.LifecycleRecorded {
t.Fatalf("live intake: %+v %v", answer, err)
}
dbtest.On(t, database).Exec("DELETE FROM fedwiki.sites WHERE site_id=$1", instance)
if answer, err := fwmod.RecordSiteRequest(env.Ctx, database, spec, domain); err != nil || answer.Kind != integration.LifecycleRepeatOpen {
t.Errorf("repeat after deletion changed its answer: %+v %v", answer, err)
}
spec.Parameters = []byte(`{"status":"active"}`)
if answer, err := fwmod.RecordSiteRequest(env.Ctx, database, spec, domain); err != nil || answer.Kind != integration.LifecycleConflict {
t.Errorf("changed repeat was accepted: %+v %v", answer, err)
}
}
// deleteWithoutRequest checks that the locked read sees committed intake.
func deleteWithoutRequest(ctx context.Context, intakeDB *sql.DB, instance string) error {
return db.InTx(ctx, intakeDB, db.ReadCommitted, func(tx *sql.Tx) error {
if err := integration.New(tx).LockLifecycleInstance(ctx, instance); err != nil {
return err
}
open, err := fwmod.New(tx).SiteHasOpenRequest(ctx, instance)
if err != nil {
return err
}
if !open {
return errors.New("open-request read missed committed intake")
}
return nil
})
}
// recordIntakePair uses the same protected batch validation as the swap handler.
func recordIntakePair(ctx context.Context, database *sql.DB, spec integration.LifecycleRequestSpec, domain, peerID, peerDomain string) error {
if peerID == "" {
_, err := fwmod.RecordSiteRequest(ctx, database, spec, domain)
return err
}
return db.InTx(ctx, database, db.ReadCommitted, func(tx *sql.Tx) error {
if err := fwmod.CheckSwapRequestSites(ctx, tx, spec.WorkspaceID, spec.InstanceID, domain, peerID, peerDomain); err != nil {
return err
}
_, err := integration.RecordLifecycleRequest(ctx, tx, spec)
return err
})
}
@@ -16,14 +16,14 @@ import (
"github.com/lib/pq"
)
const clearForceReducedByDomain = `-- name: ClearForceReducedByDomain :exec
const clearForceReducedByID = `-- name: ClearForceReducedByID :exec
UPDATE fedwiki.sites
SET force_reduced_at = NULL, updated_at = NOW()
WHERE domain = $1
WHERE site_id = $1
`
func (q *Queries) ClearForceReducedByDomain(ctx context.Context, domain string) error {
_, err := q.db.ExecContext(ctx, clearForceReducedByDomain, domain)
func (q *Queries) ClearForceReducedByID(ctx context.Context, siteID string) error {
_, err := q.db.ExecContext(ctx, clearForceReducedByID, siteID)
return err
}
@@ -1018,20 +1018,33 @@ func (q *Queries) LockSiteStatusByDomain(ctx context.Context, domain string) (Lo
}
const lockSiteStatusByID = `-- name: LockSiteStatusByID :one
SELECT status FROM fedwiki.sites
SELECT status, force_reduced_at, workspace_id, domain FROM fedwiki.sites
WHERE site_id = $1::uuid
FOR UPDATE
`
type LockSiteStatusByIDRow struct {
Status string `json:"status"`
ForceReducedAt sql.NullTime `json:"force_reduced_at"`
WorkspaceID string `json:"workspace_id"`
Domain string `json:"domain"`
}
// The status of the site a request's instance names, its row locked until the
// transaction ends. The slot gate reads it for an activation, so a site the
// sweep or the sync already made active is stamped without a comparison with
// the limit. No row (a create's instance) is sql.ErrNoRows.
func (q *Queries) LockSiteStatusByID(ctx context.Context, siteID string) (string, error) {
// the limit. Status and swap record steps check the recorded workspace and
// domain under this lock. No row is sql.ErrNoRows.
func (q *Queries) LockSiteStatusByID(ctx context.Context, siteID string) (LockSiteStatusByIDRow, error) {
row := q.db.QueryRowContext(ctx, lockSiteStatusByID, siteID)
var status string
err := row.Scan(&status)
return status, err
var i LockSiteStatusByIDRow
err := row.Scan(
&i.Status,
&i.ForceReducedAt,
&i.WorkspaceID,
&i.Domain,
)
return i, err
}
const setForceReducedByDomain = `-- name: SetForceReducedByDomain :exec
@@ -1045,6 +1058,17 @@ func (q *Queries) SetForceReducedByDomain(ctx context.Context, domain string) er
return err
}
const setForceReducedByID = `-- name: SetForceReducedByID :exec
UPDATE fedwiki.sites
SET force_reduced_at = NOW(), updated_at = NOW()
WHERE site_id = $1 AND force_reduced_at IS NULL
`
func (q *Queries) SetForceReducedByID(ctx context.Context, siteID string) error {
_, err := q.db.ExecContext(ctx, setForceReducedByID, siteID)
return err
}
const setSiteObservedByDomain = `-- name: SetSiteObservedByDomain :one
UPDATE fedwiki.sites
SET storage_bytes = $2,
@@ -1095,14 +1119,47 @@ type SetSiteStatusByDomainParams struct {
Status string `json:"status"`
}
// Project maintenance status changes from the sync and quota sweep.
func (q *Queries) SetSiteStatusByDomain(ctx context.Context, arg SetSiteStatusByDomainParams) (Site, error) {
row := q.db.QueryRowContext(ctx, setSiteStatusByDomain, arg.Domain, arg.Status)
var i Site
err := row.Scan(
&i.SiteID,
&i.WorkspaceID,
&i.Domain,
&i.CreatedAt,
&i.UpdatedAt,
&i.Status,
&i.StorageBytes,
&i.LastModifiedAt,
&i.ForceReducedAt,
&i.ArchivedAt,
)
return i, err
}
const setSiteStatusByID = `-- name: SetSiteStatusByID :one
UPDATE fedwiki.sites
SET status = $2,
force_reduced_at = CASE WHEN $2 = 'readonly' THEN force_reduced_at END,
updated_at = NOW()
WHERE site_id = $1
RETURNING site_id, workspace_id, domain, created_at, updated_at, status, storage_bytes, last_modified_at, force_reduced_at, archived_at
`
type SetSiteStatusByIDParams struct {
SiteID string `json:"site_id"`
Status string `json:"status"`
}
// Project a lifecycle status change locally. archived_at is maintained by
// the trigger function fedwiki.set_site_archived_at on every site write.
// force_reduced_at marks a site a downgrade left read-only, so it is kept
// only while the site stays
// `readonly`: a site that becomes active or archived is no longer parked,
// and the re-upgrade sweep must not bring it back.
func (q *Queries) SetSiteStatusByDomain(ctx context.Context, arg SetSiteStatusByDomainParams) (Site, error) {
row := q.db.QueryRowContext(ctx, setSiteStatusByDomain, arg.Domain, arg.Status)
func (q *Queries) SetSiteStatusByID(ctx context.Context, arg SetSiteStatusByIDParams) (Site, error) {
row := q.db.QueryRowContext(ctx, setSiteStatusByID, arg.SiteID, arg.Status)
var i Site
err := row.Scan(
&i.SiteID,
@@ -0,0 +1,53 @@
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package usage_test
import (
"errors"
"testing"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integration"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/usage"
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest"
)
func TestActivationGateRequiresItsSite_DB(t *testing.T) {
for _, row := range []struct {
name, operation, status string
hold, missing bool
limit int64
wantMissing, wantHeld, wantHold bool
}{
{"missing", integration.LifecycleSetStatus, "readonly", false, true, 2, true, false, false},
{"missing held", integration.LifecycleSetStatus, "readonly", true, true, 2, true, false, true},
{"missing full", integration.LifecycleSetStatus, "readonly", false, true, 0, true, false, false},
{"live held", integration.LifecycleSetStatus, "readonly", true, false, 2, false, true, true},
{"already active", integration.LifecycleSetStatus, "active", false, false, 0, false, true, true},
{"create", integration.LifecycleCreate, "", false, false, 2, false, true, true},
} {
t.Run(row.name, func(t *testing.T) {
fixture := newFixture(t)
tenant := fixture.pool("Identity", row.limit, 0)
var instance string
if row.status != "" {
instance = fixture.site(tenant.Workspace.ID, "identity", row.status)
}
instance, generation := fixture.record(tenant.Workspace.ID, instance, row.operation)
if row.hold {
fixture.mustHold(instance, generation)
}
if row.missing {
dbtest.On(t, fixture.db).Exec("DELETE FROM fedwiki.sites WHERE site_id = $1", instance)
}
held, err := fixture.hold(instance, generation)
if row.wantMissing {
if !errors.Is(err, usage.ErrSiteNotFound) || held != row.wantHeld || fixture.holdExists(instance, generation) != row.wantHold {
t.Errorf("%s: gate did not distinguish a missing activation from a create", row.name)
}
} else if err != nil || held != row.wantHeld || fixture.holdExists(instance, generation) != row.wantHold {
t.Errorf("%s: gate did not distinguish a missing activation from a create: %v", row.name, err)
}
})
}
}
+18 -12
View File
@@ -203,11 +203,11 @@ type HoldRequest struct {
//
// The steps, in the order the locks are taken: the request's instance lock
// and the request (a request that is missing or no longer accepted fails
// with integration.ErrLifecycleRequestNotFound); a hold the request already
// has, which a lost answer left (recounted and held, without a comparison
// with the limit); the site row of the instance, locked, whose status shows
// an activation the sweep or the sync already made active (stamped without a
// comparison, because the hold counts nothing while the row is active); the
// with integration.ErrLifecycleRequestNotFound); the original site row,
// locked, with a missing activation refused as ErrSiteNotFound; then an
// existing hold, which a lost answer left (recounted without a comparison
// with the limit). An already-active row also needs no comparison, because
// the hold counts nothing while the row is active. The remaining steps are the
// pool and its limit (only sql.ErrNoRows means no entitlement; any other
// error is returned); the usage row and the count; the limit comparison; the
// hold insert and a recount.
@@ -228,6 +228,10 @@ func hold(ctx context.Context, tx entitlements.DBTX, req HoldRequest) (held bool
return false, err
}
siteQ := fwmod.New(tx)
alreadyActive, err := activeSite(ctx, siteQ, req.InstanceID, row.Operation)
if err != nil {
return false, err
}
existing, err := siteQ.SlotHoldExists(ctx, row.RequestID)
if err != nil {
return false, fmt.Errorf("read hold of request %s: %w", row.RequestID, err)
@@ -235,10 +239,6 @@ func hold(ctx context.Context, tx entitlements.DBTX, req HoldRequest) (held bool
if existing {
return recountHeld(ctx, tx, row.WorkspaceID)
}
alreadyActive, err := activeSite(ctx, siteQ, req.InstanceID)
if err != nil {
return false, err
}
poolID, err := primaryPool(ctx, tx, row.WorkspaceID)
if err != nil {
return false, err
@@ -310,16 +310,19 @@ func lockRequest(ctx context.Context, tx entitlements.DBTX, req HoldRequest) (in
}
// activeSite locks the instance's site row and reports whether it is active.
// An instance with no row, a create's, is not.
func activeSite(ctx context.Context, siteQ fwmod.Querier, instanceID string) (bool, error) {
// A create needs no row; a missing activation is refused before a hold is read.
func activeSite(ctx context.Context, siteQ fwmod.Querier, instanceID, operation string) (bool, error) {
status, err := siteQ.LockSiteStatusByID(ctx, instanceID)
if errors.Is(err, sql.ErrNoRows) {
if operation != integration.LifecycleCreate {
return false, ErrSiteNotFound
}
return false, nil
}
if err != nil {
return false, fmt.Errorf("lock site %s: %w", instanceID, err)
}
return status == "active", nil
return status.Status == "active", nil
}
// primaryPool resolves the workspace's active primary pool.
@@ -370,3 +373,6 @@ func RecountAll(ctx context.Context, database *sql.DB, logger *slog.Logger) erro
logger.Debug("fedwiki usage recount completed", slog.Int("pools", len(pools)), slog.Int("failed", failed))
return nil
}
// ErrSiteNotFound refuses an activation whose original site was deleted.
var ErrSiteNotFound = errors.New("site not found")
@@ -0,0 +1,281 @@
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package web
import (
"bytes"
"context"
"database/sql"
"fmt"
"log/slog"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
"time"
"git.coopcloud.tech/wiki-cafe/member-console/internal/db"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integration"
fwmod "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/store"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/workflows"
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest"
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/htmltest"
)
// intakeReadHook runs after the handler's first domain read, before intake locks.
type intakeReadHook struct {
fwmod.Querier
after func()
}
func (queries intakeReadHook) GetSiteByDomain(ctx context.Context, domain string) (fwmod.GetSiteByDomainRow, error) {
row, err := queries.Querier.GetSiteByDomain(ctx, domain)
if err == nil {
queries.after()
}
return row, err
}
// intakeIdentityCase pairs an intake race with its response expectations.
type intakeIdentityCase struct {
name, status string
cooldown time.Duration
swap, park, repeatable, replacement bool
wantBody string
wantRefresh bool
}
func TestSiteIntakeRequiresOriginalRows_DB(t *testing.T) {
for _, row := range []intakeIdentityCase{
{name: "archive", status: "active", wantBody: "Site not found"}, {name: "restore", status: "archived", wantBody: "Site not found"},
{name: "rotation zero", status: "readonly", wantBody: "Site not found"}, {name: "rotation positive", status: "readonly", cooldown: time.Hour, wantBody: "Site not found"},
{name: "swap activation", status: "readonly", swap: true, wantBody: "Site not found"}, {name: "swap park", status: "readonly", swap: true, park: true, wantBody: msgSiteChanged, wantRefresh: true},
{name: "rotation repeatable", status: "readonly", cooldown: time.Hour, repeatable: true, wantBody: "Site not found"},
{name: "swap repeatable", status: "readonly", swap: true, repeatable: true, wantBody: "Site not found"},
{name: "replacement archive", status: "active", replacement: true, wantBody: "Site not found"},
{name: "replacement swap", status: "readonly", swap: true, replacement: true, wantBody: "Site not found"},
} {
t.Run(row.name, func(t *testing.T) {
checkIntakeIdentity(t, row)
})
}
t.Run("live and repeats", func(t *testing.T) {
env := newSiteEnv(t, 3)
instance, domain := env.site("repeat", "active")
generation := env.nextGen(instance)
if response := env.rowAction(env.h.ArchiveSite, "archive", domain, instance, generation); !accepted(response) {
t.Fatal(response.Body.String())
}
env.exec("DELETE FROM fedwiki.sites WHERE site_id = $1", instance)
if response := env.rowAction(env.h.ArchiveSite, "archive", domain, instance, generation); !accepted(response) || len(env.temporal.starts()) != 1 {
t.Error("repeat after deletion changed its answer")
}
if response := env.rowAction(env.h.RestoreSite, "restore", domain, instance, generation); !strings.Contains(response.Body.String(), msgSiteChanged) {
t.Error("changed repeat was accepted")
}
})
}
// noticeIdentityCase states which original is absent and the resulting sentence.
type noticeIdentityCase struct {
name, status string
swap, activate, park bool
want string
}
func TestMissingSiteNotices_DB(t *testing.T) {
for _, row := range []noticeIdentityCase{
{"archive", "active", false, true, false, "%[1]s was deleted."}, {"restore", "archived", false, true, false, "%[1]s was deleted."},
{"activation", "readonly", true, true, false, "%[1]s was deleted."}, {"park", "readonly", true, false, true, "%[1]s wasn't kept active. %[2]s was deleted."}, {"both", "readonly", true, true, true, "%[1]s was deleted."},
} {
for _, replacement := range []bool{false, true} {
t.Run(row.name+strconv.FormatBool(replacement), func(t *testing.T) {
checkMissingNotice(t, row, replacement)
})
}
}
}
// checkIntakeIdentity holds a deletion open until the handler waits on its instance lock.
func checkIntakeIdentity(t *testing.T, row intakeIdentityCase) {
t.Helper()
limit := 3
if row.swap {
limit = 1
}
env := newSiteEnv(t, limit)
var logs bytes.Buffer
env.h.Logger = slog.New(slog.NewTextHandler(&logs, &slog.HandlerOptions{Level: slog.LevelError}))
var cancel context.CancelFunc
env.sctx, cancel = context.WithTimeout(env.sctx, 15*time.Second)
defer cancel()
instance, domain := env.site("intake", row.status)
generation := env.nextGen(instance)
if row.status == "readonly" {
env.exec("UPDATE fedwiki.sites SET force_reduced_at = NOW() WHERE site_id = $1", instance)
}
target := instance
if row.swap {
parkID, _ := env.site("park", "active")
if row.park {
target = parkID
}
}
env.h.SwapCooldown = row.cooldown
if row.repeatable {
dsn := dbtest.DSN(t)
separator := "?"
if strings.Contains(dsn, "?") {
separator = "&"
}
env.h.Database = dbtest.Connect(t, dsn+separator+"default_transaction_isolation=repeatable%20read")
}
started := make(chan struct{})
resume := make(chan struct{})
env.h.SiteQ = intakeReadHook{Querier: env.h.SiteQ, after: func() {
close(started)
select {
case <-resume:
case <-env.sctx.Done():
}
}}
handler := env.h.KeepSiteActive
if row.status == "active" {
handler = env.h.ArchiveSite
}
if row.status == "archived" {
handler = env.h.RestoreSite
}
done := make(chan *httptest.ResponseRecorder, 1)
go func() { done <- env.rowAction(handler, "action", domain, instance, generation) }()
select {
case <-started:
case <-env.sctx.Done():
t.Fatal("handler did not reach its initial site read")
}
holder, holderPID := holdIntakeDeletion(t, env, target, domain, row.replacement)
close(resume)
defer holder.Rollback()
dbtest.AwaitLockWait(t, env.db, dbtest.Backend{BlockedBy: holderPID, Abort: func() error {
select {
case response := <-done:
return fmt.Errorf("intake completed before the deletion committed: %s", response.Body.String())
default:
return nil
}
}})
if err := holder.Commit(); err != nil {
t.Fatal(err)
}
response := <-done
if logs.Len() != 0 {
t.Errorf("%s: missing-site refusal logged an error: %s", row.name, logs.String())
}
if response.Code != 200 || !strings.Contains(response.Body.String(), row.wantBody) {
t.Errorf("%s: missing-site response = %d %q", row.name, response.Code, response.Body.String())
}
if accepted(response) || env.requestCount(instance) != 0 || len(env.temporal.starts()) != 0 {
t.Errorf("%s: intake accepted a missing site", row.name)
if row.repeatable {
t.Errorf("%s: intake did not use READ COMMITTED", row.name)
}
}
refresh := response.Header().Get("HX-Trigger") != ""
if refresh != row.wantRefresh {
t.Errorf("%s: missing-site refresh = %v, want %v", row.name, refresh, row.wantRefresh)
}
count := dbtest.On(t, env.db).Int("SELECT count(*) FROM fedwiki.site_swap_policy WHERE workspace_id = $1", env.ws)
if env.requestCount(instance) != 0 || len(env.temporal.starts()) != 0 || count != 0 {
t.Errorf("%s: refused intake recorded work or stamped cooldown", row.name)
}
}
// checkMissingNotice checks the selected warning and acknowledges both swap requests.
func checkMissingNotice(t *testing.T, row noticeIdentityCase, replacement bool) {
t.Helper()
env := newSiteEnv(t, 1)
instance, domain := env.site("notice", row.status)
var parkID, parkDomain string
if row.swap {
parkID, parkDomain = env.site("park", "active")
}
generation := env.nextGen(instance)
handler := env.h.KeepSiteActive
if row.status == "active" {
handler = env.h.ArchiveSite
}
if row.status == "archived" {
handler = env.h.RestoreSite
}
if response := env.rowAction(handler, "action", domain, instance, generation); !accepted(response) {
t.Fatal(response.Body.String())
}
addresses := []requestAddress{{instance, generation}}
if row.swap {
input := env.temporal.inputs[0].(workflows.SwapSiteRequestInput)
addresses = append(addresses, requestAddress{parkID, input.ParkGeneration})
}
for _, address := range addresses {
env.finish(address.Instance, address.Generation, integration.LifecycleDeclined, workflows.ReasonSiteMissing)
}
if row.activate {
env.exec("DELETE FROM fedwiki.sites WHERE site_id = $1", instance)
if replacement {
env.site("notice", row.status)
}
}
if row.park {
env.exec("DELETE FROM fedwiki.sites WHERE site_id = $1", parkID)
if replacement {
env.site("park", "active")
}
}
want := fmt.Sprintf(row.want, domain, parkDomain)
list := env.sitesList()
var got []string
for _, notice := range htmltest.Parse(t, list).All(".alert:has(button[aria-label=Dismiss])") {
got = append(got, notice.One(t, "span").Text())
}
if len(got) > 1 {
t.Errorf("%s: missing swap showed more than one notice: %v", row.name, got)
}
if len(got) != 1 || got[0] != want {
t.Errorf("%s: missing-site notice = %q, want %q", row.name, got, want)
}
if htmltest.Parse(t, list).Count(".alert-warning:has(button[aria-label=Dismiss])") != 1 {
t.Errorf("%s: missing-site notice has the wrong tone", row.name)
}
if strings.Contains(list, "Pending") {
t.Errorf("%s: deleted site remained Pending or replacement changed", row.name)
}
response := env.post(env.h.DismissRequest, "/dismiss", map[string]string{"instance": instance, "generation": strconv.FormatInt(generation, 10)}, nil)
if response.Code != http.StatusOK {
t.Error(response.Body.String())
}
for _, address := range addresses {
request, found := env.request(address.Instance, address.Generation)
if !found || !request.AcknowledgedAt.Valid {
t.Errorf("%s: missing swap dismissal left a request visible", row.name)
}
}
}
// holdIntakeDeletion holds the purge's instance lock until the handler waits.
func holdIntakeDeletion(t *testing.T, env *siteEnv, target, domain string, replacement bool) (*sql.Tx, int) {
t.Helper()
holder, err := db.Begin(env.ctx, env.db, db.ReadCommitted)
if err != nil {
t.Fatal(err)
}
holderPID := int(dbtest.On(t, holder).Int("SELECT pg_backend_pid()"))
if err := integration.New(holder).LockLifecycleInstance(env.ctx, target); err != nil {
t.Fatal(err)
}
dbtest.On(t, holder).Exec("DELETE FROM fedwiki.sites WHERE site_id = $1", target)
if replacement {
dbtest.On(t, holder).Exec("INSERT INTO fedwiki.sites (workspace_id,domain,status) VALUES ($1,$2,'readonly')", env.ws, domain)
}
return holder, holderPID
}
@@ -480,8 +480,8 @@ func (h *FedWikiPartialsHandler) changeStatus(w http.ResponseWriter, r *http.Req
requestParams{Domain: act.site.Domain, Status: target})
var answer integration.LifecycleAnswer
var refusal string
if rotation && h.SwapCooldown > 0 {
var refusal string
err := h.withWorkspaceTx(ctx, ws, func(ctx context.Context, tx *sql.Tx, q fwmod.Querier) error {
a, err := h.lockedRepeat(ctx, tx, act, integration.LifecycleSetStatus, target)
if err != nil {
@@ -495,7 +495,7 @@ func (h *FedWikiPartialsHandler) changeStatus(w http.ResponseWriter, r *http.Req
refusal = msg
return errRollback
}
if answer, err = integration.RecordLifecycleRequest(ctx, tx, spec); err != nil {
if answer, err = fwmod.RecordSiteRequestInTx(ctx, tx, spec, act.site.Domain); err != nil {
return err
}
if answer.Kind != integration.LifecycleRecorded {
@@ -504,23 +504,25 @@ func (h *FedWikiPartialsHandler) changeStatus(w http.ResponseWriter, r *http.Req
return q.UpsertSiteSwapPolicy(ctx, ws)
})
if err != nil && !errors.Is(err, errRollback) {
if h.answerMissingSite(w, err, act.site.SiteID) {
return
}
h.Logger.Error("failed to record a rotation", slog.Any("error", err))
h.siteActionResult(w, "Couldn't change which site is active. Try again later.")
return
}
if refusal != "" {
h.siteActionResult(w, refusal)
return
}
} else {
var err error
if answer, err = integration.RecordLifecycleRequest(ctx, h.Database, spec); err != nil {
if answer, err = fwmod.RecordSiteRequest(ctx, h.Database, spec, act.site.Domain); err != nil {
if h.answerMissingSite(w, err, act.site.SiteID) {
return
}
h.Logger.Error("failed to record a status change", slog.Any("error", err))
h.siteActionResult(w, "Couldn't update the site. Try again.")
return
}
}
if !h.answerRecord(w, answer) {
if !h.answerSiteIntake(w, answer, refusal) {
return
}
@@ -592,11 +594,9 @@ func (h *FedWikiPartialsHandler) swapActiveSite(w http.ResponseWriter, r *http.R
if err != nil {
return err
}
for _, req := range open {
if req.Status == integration.LifecycleAccepted && workflows.IsSwapRequestWorkflowID(req.WorkflowID) {
refusal = msgActiveSiteBusy
return errRollback
}
if hasOpenSwap(open) {
refusal = msgActiveSiteBusy
return errRollback
}
locked, err := q.ListActiveSitesByWorkspace(ctx, ws)
if err != nil {
@@ -611,6 +611,9 @@ func (h *FedWikiPartialsHandler) swapActiveSite(w http.ResponseWriter, r *http.R
}
parked := locked[len(locked)-1] // lowest-ranked by recorded modification or creation
parkDom = parked.Domain
if err := fwmod.CheckSwapRequestSites(ctx, tx, ws, act.site.SiteID, act.site.Domain, parked.SiteID, parked.Domain); err != nil {
return err
}
answer, err = integration.RecordLifecycleRequest(ctx, tx, newRequestSpec(conn, ws, act.session.PersonID,
integration.LifecycleSetStatus, act.addr, workflowID,
@@ -621,14 +624,8 @@ func (h *FedWikiPartialsHandler) swapActiveSite(w http.ResponseWriter, r *http.R
if answer.Kind != integration.LifecycleRecorded {
return errRollback
}
gens, err := integration.NextGenerations(ctx, tx, []string{parked.SiteID})
if err != nil {
return err
}
park = requestAddress{Instance: parked.SiteID, Generation: gens[parked.SiteID]}
parkAnswer, err := integration.RecordLifecycleRequest(ctx, tx, newRequestSpec(conn, ws, act.session.PersonID,
integration.LifecycleSetStatus, park, workflowID,
requestParams{Domain: parked.Domain, Status: "readonly", SwapActivateSiteID: act.site.SiteID}))
var parkAnswer integration.LifecycleAnswer
park, parkAnswer, err = h.recordParkRequest(ctx, tx, act, parked.SiteID, parked.Domain, workflowID)
if err != nil {
return err
}
@@ -640,15 +637,14 @@ func (h *FedWikiPartialsHandler) swapActiveSite(w http.ResponseWriter, r *http.R
return q.UpsertSiteSwapPolicy(ctx, ws)
})
if err != nil && !errors.Is(err, errRollback) {
if h.answerMissingSite(w, err, act.site.SiteID) {
return
}
h.Logger.Error("failed to record a swap", slog.Any("error", err))
h.siteActionResult(w, "Couldn't change the active site. Try again.")
return
}
if refusal != "" {
h.siteActionResult(w, refusal)
return
}
if !h.answerRecord(w, answer) {
if !h.answerSiteIntake(w, answer, refusal) {
return
}
@@ -796,11 +792,12 @@ func (h *FedWikiPartialsHandler) siteActionConflict(w http.ResponseWriter) {
// cannot interleave the cooldown check-and-stamp or the park-target
// selection (audit #51). fn receives the transaction and a querier on it;
// returning an error rolls the transaction back, otherwise it commits.
// READ COMMITTED lets protected site reads see a purge that held the instance lock.
func (h *FedWikiPartialsHandler) withWorkspaceTx(ctx context.Context, workspaceID string, fn func(context.Context, *sql.Tx, fwmod.Querier) error) error {
if h.Database == nil {
return errors.New("no database for the workspace lock")
}
return db.InTx(ctx, h.Database, db.Plain, func(tx *sql.Tx) error {
return db.InTx(ctx, h.Database, db.ReadCommitted, func(tx *sql.Tx) error {
// hashtext maps the workspace UUID text onto the bigint the advisory-lock
// API takes; an occasional hash collision only serializes two unrelated
// workspaces briefly, which is harmless.
@@ -900,8 +897,20 @@ func (v requestsView) notices(siteStatus map[string]string) []RequestNotice {
if row.Status == integration.LifecycleFailed {
tone = "danger"
}
sentence := noticeSentence(p.Domain, requestAction(row.Operation, p, siteStatus[row.InstanceID]), row.ReasonCode.String)
if row.ReasonCode.String == workflows.ReasonSiteMissing {
sentence = p.Domain + " was deleted."
if _, exists := siteStatus[row.InstanceID]; exists && p.SwapParkSiteID != "" {
for _, peer := range v.rows {
if peer.InstanceID == p.SwapParkSiteID && peer.WorkflowID == row.WorkflowID {
sentence = p.Domain + " wasn't kept active. " + decodeParams(peer.Parameters).Domain + " was deleted."
break
}
}
}
}
out = append(out, RequestNotice{
Sentence: noticeSentence(p.Domain, requestAction(row.Operation, p, siteStatus[row.InstanceID]), row.ReasonCode.String),
Sentence: sentence,
Tone: tone,
Instance: row.InstanceID,
Generation: row.Generation,
@@ -980,3 +989,49 @@ func (h *FedWikiPartialsHandler) bannerRows(ctx context.Context, workspaceID str
}
return out
}
// answerMissingSite renders an intake refusal without recording any work.
func (h *FedWikiPartialsHandler) answerMissingSite(w http.ResponseWriter, err error, primaryID string) bool {
var missing *fwmod.MissingSiteError
if !errors.As(err, &missing) {
return false
}
if missing.InstanceID == primaryID {
h.siteActionResult(w, "Site not found")
} else {
h.siteActionConflict(w)
}
return true
}
// hasOpenSwap reports whether a workspace is already changing its active site.
func hasOpenSwap(rows []integration.LifecycleRequest) bool {
for _, row := range rows {
if row.Status == integration.LifecycleAccepted && workflows.IsSwapRequestWorkflowID(row.WorkflowID) {
return true
}
}
return false
}
// answerSiteIntake renders a policy refusal or the lifecycle address's answer.
func (h *FedWikiPartialsHandler) answerSiteIntake(w http.ResponseWriter, answer integration.LifecycleAnswer, refusal string) bool {
if refusal != "" {
h.siteActionResult(w, refusal)
return false
}
return h.answerRecord(w, answer)
}
// recordParkRequest records the outgoing half of a swap in the caller's transaction.
func (h *FedWikiPartialsHandler) recordParkRequest(ctx context.Context, tx *sql.Tx, act rowAction, instance, domain, workflowID string) (requestAddress, integration.LifecycleAnswer, error) {
gens, err := integration.NextGenerations(ctx, tx, []string{instance})
if err != nil {
return requestAddress{}, integration.LifecycleAnswer{}, err
}
park := requestAddress{Instance: instance, Generation: gens[instance]}
answer, err := integration.RecordLifecycleRequest(ctx, tx, newRequestSpec(h.connectionID(), act.session.WorkspaceID, act.session.PersonID,
integration.LifecycleSetStatus, park, workflowID,
requestParams{Domain: domain, Status: "readonly", SwapActivateSiteID: act.site.SiteID}))
return park, answer, err
}
@@ -183,3 +183,9 @@ Delete operations treat HTTP 404 (Not Found) as success - if the site doesn't ex
HTTP handlers do not wait for these workflows. A handler records a lifecycle request in `core.lifecycle_requests`, commits, starts the workflow under the request's workflow id, and answers as soon as the start is confirmed. The workflow ends by recording the request's status and reason code, and the site list shows an open request as Pending on its row and an undismissed declined or failed one as a notice above the list. A request whose workflow never recorded an outcome is settled by the boot pass in `settle.go`.
The four workflow types that predate lifecycle requests (`CreateFedWikiSiteWorkflow`, `DeleteFedWikiSiteWorkflow`, `SetSiteStatusWorkflow`, `SwapActiveSiteWorkflow`) are not registered. After the console connects to Temporal and before its worker starts, `Adapter.CheckBeforeWorker` lists the running executions of those types. While any is open the worker does not start: the console serves, every Temporal-backed action answers that it is not available, the console logs `refused to start the Temporal worker` at Error with the open runs counted per type, and the list is read again every 15 to 22.5 seconds. A list that cannot be read is retried on a new connection. Let an open run finish on the previous release, or terminate it; the worker starts once none is open.
Status and swap record steps lock the site IDs recorded by their requests, check their workspace and domain, and write only those IDs. A missing original row ends the status request declined with `site_missing`; a changed identity ends it declined with `site_changed`. Finishing removes any hold and recounts usage in the outcome transaction. Creates and permanent deletes keep their existing missing-row behavior.
If either swap row is gone at authorization or recording, neither local status is written. The existing restoration activity sends surviving originals back to their recorded statuses before both requests decline. Missing or mismatched originals and farm 404s are skipped. A final refusal stops retries after the other site is tried; unanswered farm or database reads keep retrying with both requests accepted.
Boot settlement hands a closed held activation to status reconciliation. A site row missing at load or at record ends that request declined with `site_missing`. Other closed requests retain the existing failed/`retries_exhausted` path.
@@ -188,7 +188,7 @@ type DeleteFedWikiSiteInput struct {
// SiteAuthorizationError is the non-retryable error type a mutating activity
// returns when the site it was started for is no longer the site the request
// authorized: the row is gone, belongs to another workspace, or carries
// authorized: the row belongs to another workspace or carries
// another name. Listed in FedWikiActivityOptions' NonRetryableErrorTypes so
// no retry re-asks the same question.
const SiteAuthorizationError = "SiteAuthorization"
@@ -194,7 +194,7 @@ func recordArchiveTime(t *testing.T, row recordArchiveCase) {
parked := env.insertSite(parkedDomain, "active", false)
var parkedGeneration int64
generation, parkedGeneration = env.swapRequests(id, parked, "archive-time-"+id)
err = env.acts.RecordSwapActivity(env.ctx, RecordSwapInput{Activate: LifecycleRequestRef{InstanceID: id, Generation: generation}, Park: LifecycleRequestRef{InstanceID: parked, Generation: parkedGeneration}, ActivateDomain: domain, ParkDomain: parkedDomain, FinishedAt: time.Now()})
err = env.acts.RecordSwapActivity(env.ctx, RecordSwapInput{WorkspaceID: env.ws, Activate: LifecycleRequestRef{InstanceID: id, Generation: generation}, Park: LifecycleRequestRef{InstanceID: parked, Generation: parkedGeneration}, ActivateDomain: domain, ParkDomain: parkedDomain, FinishedAt: time.Now()})
if status, _ := env.request(parked, parkedGeneration); status != "succeeded" {
t.Errorf("parked request = %s", status)
}
@@ -100,7 +100,10 @@ func TestApplyFarmSiteStatusActivityRefusesAMissingSite(t *testing.T) {
err := acts.ApplyFarmSiteStatusActivity(context.Background(), SetSiteStatusInput{
SiteID: "site-gone", WorkspaceID: "ws-A", Domain: "a.localtest.me", Status: "archived",
})
isSiteAuthorizationRefusal(t, err)
var app *temporal.ApplicationError
if !errors.As(err, &app) || app.Type() != SiteMissing || !app.NonRetryable() {
t.Errorf("missing error = %v, want non-retryable SiteMissing", err)
}
if calls.Load() != 0 {
t.Fatalf("farm received %d requests for a site that no longer exists", calls.Load())
}
@@ -187,7 +187,7 @@ func checkRecordPlacement(t *testing.T, row placementRecordCase) {
dbtest.On(t, env.db).Exec(`UPDATE domains.placements SET servable=false WHERE resource_ref=$1`, parkedID)
generation, parkedGeneration = env.swapRequests(id, parkedID, "placement-"+id)
run = func() error {
return env.acts.RecordSwapActivity(env.ctx, RecordSwapInput{Activate: LifecycleRequestRef{InstanceID: id, Generation: generation}, Park: LifecycleRequestRef{InstanceID: parkedID, Generation: parkedGeneration}, ActivateDomain: name, ParkDomain: parkedName, FinishedAt: time.Now()})
return env.acts.RecordSwapActivity(env.ctx, RecordSwapInput{WorkspaceID: env.ws, Activate: LifecycleRequestRef{InstanceID: id, Generation: generation}, Park: LifecycleRequestRef{InstanceID: parkedID, Generation: parkedGeneration}, ActivateDomain: name, ParkDomain: parkedName, FinishedAt: time.Now()})
}
} else {
generation = env.record(id, integration.LifecycleSetStatus, "placement-"+id, map[string]any{"domain": name, "status": row.target})
@@ -112,10 +112,10 @@ func retentionWaitsForIntake(t *testing.T, row purgeWaitCase) {
site := archivedPurgeSite(env, "intake")
farm := observePurgeFarm(env, nil)
side := env.beginSide(env.lockWaitContext())
answer, err := integration.RecordLifecycleRequest(side.ctx, side.tx, integration.LifecycleRequestSpec{
answer, err := fwmod.RecordSiteRequestInTx(side.ctx, side.tx, integration.LifecycleRequestSpec{
InstanceID: site.SiteID, Generation: 1, ConnectionID: env.conn, ResourceKey: "fedwiki_sites", WorkspaceID: env.ws,
Operation: integration.LifecycleSetStatus, Parameters: json.RawMessage(`{"status":"active"}`), WorkflowID: "purge-" + site.SiteID, RequestedByPersonID: env.person,
})
}, site.Domain)
if err != nil || answer.Kind != integration.LifecycleRecorded {
t.Fatalf("intake = %+v, %v", answer, err)
}
@@ -133,7 +133,7 @@ func retentionWaitsForIntake(t *testing.T, row purgeWaitCase) {
t.Fatalf("purge error = %v, want failure %t", call.err, row.wantError)
}
if call.deleted != row.wantDeleted || len(farm.calls()) != row.wantCalls {
t.Error("purge deleted a site whose restore committed before its lock")
t.Errorf("%s: purge deleted an accepted request's site", row.name)
}
checkPurgeRows(env, row.name, site.Domain, site.SiteID, true)
}
@@ -338,7 +338,7 @@ func (a *Activities) RecordCreatedSiteActivity(ctx context.Context, in RecordCre
func (a *Activities) ApplyFarmSiteStatusActivity(ctx context.Context, in SetSiteStatusInput) error {
if _, err := a.authorizeSite(ctx, in.SiteID, in.WorkspaceID, in.Domain); err != nil {
if errors.Is(err, sql.ErrNoRows) {
return temporal.NewNonRetryableApplicationError("the site no longer exists", SiteAuthorizationError, err)
return temporal.NewNonRetryableApplicationError("the site no longer exists", SiteMissing, err)
}
return err
}
@@ -376,6 +376,7 @@ type RecordSiteStatusInput struct {
// The site row is locked here, so the sweep that parked the site after the
// member asked and this write order on the row; the figure is a recount, so
// neither releases a slot by arithmetic (Open Questions 66).
// Missing or changed original identities are non-retryable refusals.
func (a *Activities) RecordSiteStatusActivity(ctx context.Context, in RecordSiteStatusInput) error {
_, err := a.finishWith(ctx, integration.LifecycleOutcome{
InstanceID: in.InstanceID,
@@ -384,14 +385,17 @@ func (a *Activities) RecordSiteStatusActivity(ctx context.Context, in RecordSite
FinishedAt: in.FinishedAt,
}, func(tx *sql.Tx) error {
q := fwmod.New(tx)
if _, err := q.LockSiteStatusByDomain(ctx, in.Domain); err != nil {
if _, err := lockRequestSite(ctx, q, in.InstanceID, in.WorkspaceID, in.Domain); err != nil {
if errorType(err) != "" {
return err
}
return fmt.Errorf("record status of %s: %w", in.Domain, err)
}
if _, err := q.SetSiteStatusByDomain(ctx, fwmod.SetSiteStatusByDomainParams{Domain: in.Domain, Status: in.Status}); err != nil {
if _, err := q.SetSiteStatusByID(ctx, fwmod.SetSiteStatusByIDParams{SiteID: in.InstanceID, Status: in.Status}); err != nil {
return fmt.Errorf("record status of %s: %w", in.Domain, err)
}
if in.ClearForceReduced {
if err := q.ClearForceReducedByDomain(ctx, in.Domain); err != nil {
if err := q.ClearForceReducedByID(ctx, in.InstanceID); err != nil {
return fmt.Errorf("clear force_reduced on %s: %w", in.Domain, err)
}
}
@@ -576,7 +580,7 @@ type HoldSiteSlotOutput struct {
// and no usage row fails it with an Error line naming the pool, and the
// workflow ends the request failed after the activity's attempts. Declining a
// request that has no room is an answer, not an error: the figure is written
// and Held is false.
// and Held is false. A missing activation returns non-retryable SiteMissing.
func (a *Activities) HoldSiteSlotActivity(ctx context.Context, in HoldSiteSlotInput) (*HoldSiteSlotOutput, error) {
var held bool
err := db.InTx(ctx, a.Database, db.ReadCommitted, func(tx *sql.Tx) error {
@@ -586,6 +590,9 @@ func (a *Activities) HoldSiteSlotActivity(ctx context.Context, in HoldSiteSlotIn
})
return err
})
if errors.Is(err, siteusage.ErrSiteNotFound) {
return nil, temporal.NewNonRetryableApplicationError("the site no longer exists", SiteMissing, err)
}
if errors.Is(err, siteusage.ErrNoUsageRow) {
a.Logger.Error("slot gate: the pool has a sites entitlement and no usage row", slog.Any("error", err))
}
@@ -0,0 +1,34 @@
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package workflows
import (
"context"
"database/sql"
"errors"
fwmod "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/store"
"go.temporal.io/sdk/temporal"
)
// SiteMissing identifies a request whose original site row was deleted.
const SiteMissing = "SiteMissing"
// lockRequestSite checks the recorded identity under the site's row lock.
func lockRequestSite(ctx context.Context, queries fwmod.Querier, instanceID, workspaceID, domain string) (fwmod.LockSiteStatusByIDRow, error) {
row, err := queries.LockSiteStatusByID(ctx, instanceID)
if errors.Is(err, sql.ErrNoRows) {
return row, temporal.NewNonRetryableApplicationError("the site no longer exists", SiteMissing, err)
}
if err != nil {
return row, err
}
if row.WorkspaceID != workspaceID {
return row, temporal.NewNonRetryableApplicationError("the site is no longer this workspace's", SiteAuthorizationError, nil)
}
if row.Domain != domain {
return row, temporal.NewNonRetryableApplicationError("the site's name changed since the request", SiteAuthorizationError, nil)
}
return row, nil
}
@@ -0,0 +1,342 @@
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package workflows
import (
"context"
"database/sql"
"errors"
"strings"
"testing"
"time"
"github.com/stretchr/testify/mock"
"go.temporal.io/sdk/temporal"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integration"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/fedwikitest"
fwmod "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/store"
)
// identityRecord constructs the record input without changing its recorded identity.
func identityRecord(in SetSiteStatusRequestInput) RecordSiteStatusInput {
return RecordSiteStatusInput{InstanceID: in.InstanceID, Generation: in.Generation,
WorkspaceID: in.WorkspaceID, Domain: in.Domain, Status: in.TargetStatus,
ClearForceReduced: in.ClearForceReduced, FinishedAt: time.Now()}
}
// statusIdentityCase states the original action and its expected record outcome.
type statusIdentityCase struct {
name, current, target, replacement string
clear bool
wantStatus, wantReason string
}
func TestStatusRecordUsesOriginalSite_DB(t *testing.T) {
for _, action := range []statusIdentityCase{
{"archive/none", "active", "archived", "none", false, integration.LifecycleDeclined, ReasonSiteMissing},
{"archive/same workspace", "active", "archived", "same workspace", false, integration.LifecycleDeclined, ReasonSiteMissing},
{"archive/another workspace", "active", "archived", "another workspace", false, integration.LifecycleDeclined, ReasonSiteMissing},
{"archive/live", "active", "archived", "live", false, integration.LifecycleSucceeded, ""},
{"restore/none", "archived", "active", "none", false, integration.LifecycleDeclined, ReasonSiteMissing},
{"restore/same workspace", "archived", "active", "same workspace", false, integration.LifecycleDeclined, ReasonSiteMissing},
{"restore/another workspace", "archived", "active", "another workspace", false, integration.LifecycleDeclined, ReasonSiteMissing},
{"restore/live", "archived", "active", "live", false, integration.LifecycleSucceeded, ""},
{"clear-force-reduced/none", "readonly", "active", "none", true, integration.LifecycleDeclined, ReasonSiteMissing},
{"clear-force-reduced/same workspace", "readonly", "active", "same workspace", true, integration.LifecycleDeclined, ReasonSiteMissing},
{"clear-force-reduced/another workspace", "readonly", "active", "another workspace", true, integration.LifecycleDeclined, ReasonSiteMissing},
{"clear-force-reduced/live", "readonly", "active", "live", true, integration.LifecycleSucceeded, ""},
} {
t.Run(action.name, func(t *testing.T) {
checkStatusIdentity(t, action)
})
}
}
func TestRecordIdentityChecksAndRetry_DB(t *testing.T) {
for _, row := range []struct {
name, mismatch string
reconcile bool
wantStatus, wantReason string
}{
{"workspace/status", "workspace", false, integration.LifecycleDeclined, ReasonSiteChanged},
{"workspace/reconcile", "workspace", true, integration.LifecycleDeclined, ReasonSiteChanged},
{"domain/status", "domain", false, integration.LifecycleDeclined, ReasonSiteChanged},
{"domain/reconcile", "domain", true, integration.LifecycleDeclined, ReasonSiteChanged},
} {
t.Run(row.name, func(t *testing.T) {
checkRecordMismatch(t, row.mismatch, row.reconcile, row.wantStatus, row.wantReason)
})
}
t.Run("committed success and missing request", func(t *testing.T) {
env := newReqEnv(t, 4, 0)
in := env.heldActivation("committed")
input := identityRecord(in)
if err := env.acts.RecordSiteStatusActivity(env.ctx, input); err != nil {
t.Fatal(err)
}
env.exec("DELETE FROM fedwiki.sites WHERE site_id = $1", in.InstanceID)
if err := env.acts.RecordSiteStatusActivity(env.ctx, input); err != nil {
t.Errorf("committed outcome was applied again: %v", err)
}
input.InstanceID = env.newInstance()
if err := env.acts.RecordSiteStatusActivity(env.ctx, input); errorType(err) != LifecycleRequestMissing {
t.Errorf("missing request was treated as a missing site: %v", err)
}
})
t.Run("database error", func(t *testing.T) {
env := newReqEnv(t, 4, 0)
in := env.heldActivation("locked")
tx, err := env.db.BeginTx(env.ctx, nil)
if err != nil {
t.Fatal(err)
}
defer tx.Rollback()
if _, err := fwmod.New(tx).LockSiteStatusByID(env.ctx, in.InstanceID); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(env.ctx, 100*time.Millisecond)
defer cancel()
err = env.acts.RecordSiteStatusActivity(ctx, identityRecord(in))
if err == nil {
t.Error("record did not fail while its site row was locked")
}
if err != nil && !strings.Contains(err.Error(), "record status of "+in.Domain+":") {
t.Errorf("record database error lost its domain context: %v", err)
}
var app *temporal.ApplicationError
if errorType(err) == SiteMissing || (errors.As(err, &app) && app.NonRetryable()) {
t.Errorf("database error was classified as non-retryable: %v", err)
}
})
t.Run("finish rollback", func(t *testing.T) {
env := newReqEnv(t, 4, 0)
in := env.heldActivation("rollback")
out := []integration.LifecycleOutcome{{InstanceID: in.InstanceID, Generation: in.Generation, Status: integration.LifecycleDeclined, ReasonCode: ReasonSiteMissing, FinishedAt: time.Now()}}
_, err := finishAllWith(env.ctx, env.db, out, nil, func(*sql.Tx, recounted) error { return errors.New("guard refuses") })
status, _ := env.request(in.InstanceID, in.Generation)
if err == nil || status != integration.LifecycleAccepted || !env.holds(in.InstanceID, in.Generation) {
t.Error("finish rollback lost the hold or outcome")
}
if _, err := finishAllWith(env.ctx, env.db, out, nil, nil); err != nil {
t.Fatal(err)
}
env.wantFigure("retry")
})
}
// statusStageCase states whether absence is discovered before a farm call.
type statusStageCase struct {
stage string
farmCallsAllowed bool
}
func TestMissingStatusStagesDecline_DB(t *testing.T) {
for _, row := range []statusStageCase{
{"gate", false}, {"farm", false}, {"recorded farm", false},
{"reconcile load", false}, {"reconcile record", true}, {"confirmed readback", true},
} {
t.Run(row.stage, func(t *testing.T) { checkMissingStatusStage(t, row) })
}
}
// checkMissingStatusStage checks refusal and hold release at one workflow stage.
func checkMissingStatusStage(t *testing.T, row statusStageCase) {
t.Helper()
stage := row.stage
env := newReqEnv(t, 4, 0)
in := env.heldActivation("stages")
env.farm.HoldIn(in.Domain, env.subject, "active")
workflowEnv := env.testEnv()
if stage == "recorded farm" {
workflowEnv = env.testEnvRecorded()
}
switch stage {
case "reconcile record":
workflowEnv.OnActivity(env.acts.ReadFarmSiteActivity, mock.Anything, mock.Anything).Return(func(ctx context.Context, input ReadFarmSiteInput) (*ReadFarmSiteOutput, error) {
if _, err := env.db.ExecContext(ctx, "DELETE FROM fedwiki.sites WHERE site_id = $1", in.InstanceID); err != nil {
return nil, err
}
return env.acts.ReadFarmSiteActivity(ctx, input)
})
case "farm", "recorded farm", "confirmed readback":
workflowEnv.OnActivity(env.acts.ApplyFarmSiteStatusActivity, mock.Anything, mock.Anything).Return(func(ctx context.Context, input SetSiteStatusInput) error {
if _, err := env.db.ExecContext(ctx, "DELETE FROM fedwiki.sites WHERE site_id = $1", in.InstanceID); err != nil {
return err
}
if stage == "confirmed readback" {
return temporal.NewNonRetryableApplicationError("ambiguous", "Timeout", nil)
}
return env.acts.ApplyFarmSiteStatusActivity(ctx, input)
})
default:
env.exec("DELETE FROM fedwiki.sites WHERE site_id = $1", in.InstanceID)
}
workflowEnv.RegisterDelayedCallback(workflowEnv.CancelWorkflow, time.Hour)
if stage == "reconcile load" || stage == "reconcile record" {
workflowEnv.ExecuteWorkflow(ReconcileFedWikiSiteStatusWorkflow, reconcileInput(env, in))
} else {
workflowEnv.ExecuteWorkflow(SetFedWikiSiteStatusRequestWorkflow, in)
}
status, reason := env.request(in.InstanceID, in.Generation)
figure, rows := fedwikitest.FigureAndRows(t, env.db, env.ws)
if status != integration.LifecycleDeclined || reason != ReasonSiteMissing || env.holds(in.InstanceID, in.Generation) || figure != rows {
t.Errorf("%s: missing status did not decline and release its hold: %s/%s", stage, status, reason)
}
_, reads, patches := env.farm.Counts()
if !row.farmCallsAllowed && reads+patches != 0 {
t.Errorf("%s: missing status called the farm after discovering absence", stage)
}
}
// TestIntakeAndPurgeOrder_DB intentionally repeats the default-isolation retention row as a break target.
func TestIntakeAndPurgeOrder_DB(t *testing.T) {
retentionWaitsForIntake(t, purgeWaitCase{name: "protected intake", wantDeleted: false, wantError: false, wantCalls: 0})
}
// checkStatusIdentity checks one record outcome without changing a replacement.
func checkStatusIdentity(t *testing.T, action statusIdentityCase) {
t.Helper()
env := newReqEnv(t, 4, 0)
replacement := action.replacement
in := env.startActivation("identity", action.current)
in.TargetStatus, in.ClearForceReduced = action.target, action.clear
var replacementID string
var before fwmod.GetSiteByIDRow
changes := make(chan func())
workflowEnv := env.testEnv()
workflowEnv.OnActivity(env.acts.ApplyFarmSiteStatusActivity, mock.Anything, mock.Anything).Return(func(ctx context.Context, input SetSiteStatusInput) error {
if replacement == "live" {
return nil
}
if _, err := env.db.ExecContext(ctx, "DELETE FROM fedwiki.sites WHERE site_id = $1", in.InstanceID); err != nil {
return err
}
onIdentityTestThread(changes, func() {
if replacement != "none" {
workspace := env.ws
if replacement == "another workspace" {
workspace = fedwikitest.New(t, env.db).Tenant("Replacement").Workspace.ID
}
replacementID = fedwikitest.Site(t, env.db, workspace, in.Domain, "readonly")
fedwikitest.ForceReduce(t, env.db, in.Domain)
replacementEnv := *env
replacementEnv.ws = workspace
replacementEnv.placeServable(in.Domain, replacementID)
before = identitySiteRow(t, env, replacementID)
}
})
return nil
})
attempts := 0
var recordErr error
workflowEnv.OnActivity(env.acts.RecordSiteStatusActivity, mock.Anything, mock.Anything).Return(func(ctx context.Context, input RecordSiteStatusInput) error {
attempts++
recordErr = env.acts.RecordSiteStatusActivity(ctx, input)
return recordErr
})
workflowEnv.RegisterDelayedCallback(workflowEnv.CancelWorkflow, time.Hour)
runIdentityWorkflow(changes, func() { workflowEnv.ExecuteWorkflow(SetFedWikiSiteStatusRequestWorkflow, in) })
if attempts != 1 {
t.Errorf("%s: record attempted %d times, want 1", action.name, attempts)
}
status, reason := env.request(in.InstanceID, in.Generation)
if action.wantStatus == integration.LifecycleSucceeded {
if status != action.wantStatus || reason != action.wantReason {
t.Errorf("live record = %s/%s", status, reason)
}
} else {
var app *temporal.ApplicationError
if !errors.As(recordErr, &app) || app.Type() != SiteMissing || !app.NonRetryable() {
t.Errorf("%s: missing record error = %v, want non-retryable SiteMissing", action.name, recordErr)
}
if status != action.wantStatus || reason != action.wantReason {
t.Errorf("%s: outcome = %s/%s, want declined/site_missing", action.name, status, reason)
}
}
if replacementID != "" {
row, err := fwmod.New(env.db).GetSiteByID(env.ctx, replacementID)
placement := placementFor(t, env.db, in.Domain)
if err != nil || row.Status != "readonly" || !row.ForceReducedAt.Valid || row.ArchivedAt.Valid || !row.UpdatedAt.Equal(before.UpdatedAt) || placement == nil || !placement.Servable {
t.Errorf("%s: replacement changed", action.name)
}
}
figure, rows := fedwikitest.FigureAndRows(t, env.db, env.ws)
if env.holds(in.InstanceID, in.Generation) || figure != rows {
t.Errorf("%s: finished request retained a hold or wrong usage", action.name)
}
}
// checkRecordMismatch exercises the outcome branch after identity changes.
func checkRecordMismatch(t *testing.T, mismatch string, reconcile bool, wantStatus, wantReason string) {
t.Helper()
env := newReqEnv(t, 4, 0)
in := env.heldActivation("mismatch")
env.farm.HoldIn(in.Domain, env.subject, "active")
workflowEnv := env.testEnv()
workflowEnv.OnActivity(env.acts.ApplyFarmSiteStatusActivity, mock.Anything, mock.Anything).Return(nil)
changed := false
changes := make(chan func())
workflowEnv.OnActivity(env.acts.RecordSiteStatusActivity, mock.Anything, mock.Anything).Return(func(ctx context.Context, input RecordSiteStatusInput) error {
onIdentityTestThread(changes, func() {
if !changed {
changed = true
if mismatch == "domain" {
env.exec("UPDATE fedwiki.sites SET domain = $2 WHERE site_id = $1", in.InstanceID, "changed."+env.root)
} else {
other := fedwikitest.New(t, env.db).Tenant("Moved")
env.exec("UPDATE fedwiki.sites SET workspace_id = $2 WHERE site_id = $1", in.InstanceID, other.Workspace.ID)
}
}
})
return env.acts.RecordSiteStatusActivity(ctx, input)
})
workflowEnv.RegisterDelayedCallback(workflowEnv.CancelWorkflow, time.Hour)
runIdentityWorkflow(changes, func() {
if reconcile {
workflowEnv.ExecuteWorkflow(ReconcileFedWikiSiteStatusWorkflow, reconcileInput(env, in))
} else {
workflowEnv.ExecuteWorkflow(SetFedWikiSiteStatusRequestWorkflow, in)
}
})
status, reason := env.request(in.InstanceID, in.Generation)
if status != wantStatus || reason != wantReason {
t.Errorf("%s: record mismatch outcome = %s/%s, want declined/site_changed", mismatch, status, reason)
}
row, err := fwmod.New(env.db).GetSiteByID(env.ctx, in.InstanceID)
if err != nil || row.Status != "readonly" || !row.ForceReducedAt.Valid {
t.Errorf("%s: identity mismatch wrote a site", mismatch)
}
}
// onIdentityTestThread waits for a fixture mutation on the test's goroutine.
func onIdentityTestThread(changes chan<- func(), change func()) {
done := make(chan struct{})
changes <- func() { defer close(done); change() }
<-done
}
// runIdentityWorkflow runs fixture mutations on the test goroutine while activities wait.
func runIdentityWorkflow(changes <-chan func(), run func()) {
done := make(chan struct{})
go func() { defer close(done); run() }()
for {
select {
case change := <-changes:
change()
case <-done:
return
}
}
}
// identitySiteRow reads the replacement after its fixture has been prepared.
func identitySiteRow(t *testing.T, env *reqEnv, instance string) fwmod.GetSiteByIDRow {
t.Helper()
row, err := fwmod.New(env.db).GetSiteByID(env.ctx, instance)
if err != nil {
t.Fatal(err)
}
return row
}
@@ -28,6 +28,7 @@ const (
ReasonNameRefused = "name_refused"
ReasonNameTaken = "name_taken"
ReasonSiteChanged = "site_changed"
ReasonSiteMissing = "site_missing"
ReasonQuotaUnavailable = "quota_unavailable"
ReasonRetriesExhausted = "retries_exhausted"
ReasonNotStarted = "not_started"
@@ -493,6 +494,9 @@ func gateActivation(ctx workflow.Context, in SetSiteStatusRequestInput, version
if err := workflow.ExecuteActivity(local, acts.HoldSiteSlotActivity, HoldSiteSlotInput{
InstanceID: in.InstanceID, Generation: in.Generation, WorkspaceID: in.WorkspaceID,
}).Get(ctx, &gate); err != nil {
if errorType(err) == SiteMissing {
return &requestEnd{integration.LifecycleDeclined, ReasonSiteMissing}
}
logger.Error("slot gate failed", "error", err)
return &requestEnd{integration.LifecycleFailed, ReasonQuotaUnavailable}
}
@@ -521,7 +525,7 @@ const farmRefusalType = "FarmManagerAPIError"
// the target status, so the change goes on to its record step; otherwise it
// has ended the request, and out and err are that outcome.
//
// A site that is gone (SiteAuthorizationError) is declined, and a farm
// A site that is gone (SiteMissing) or changed identity is declined, and a farm
// refusal that did not apply the change fails the request; in both the
// transaction that finishes the request removes its hold. Any other failure of
// an activation that holds a slot may have come after the farm applied the
@@ -533,6 +537,9 @@ const farmRefusalType = "FarmManagerAPIError"
func statusChangeFailed(ctx workflow.Context, in SetSiteStatusRequestInput, holds bool, applyErr error, self []LifecycleRequestRef) (applied bool, out *LifecycleWorkflowOutput, err error) {
kind := errorType(applyErr)
switch {
case kind == SiteMissing:
out, err = finishRequests(ctx, self, integration.LifecycleDeclined, ReasonSiteMissing)
return false, out, err
case kind == SiteAuthorizationError:
out, err = finishRequests(ctx, self, integration.LifecycleDeclined, ReasonSiteChanged)
return false, out, err
@@ -568,6 +575,9 @@ func compensateStatusChange(ctx workflow.Context, in SetSiteStatusRequestInput,
workflow.GetLogger(ctx).Error("failed to compensate usage", "error", err)
}
}
if errorType(applyErr) == SiteMissing {
return finishRequests(ctx, self, integration.LifecycleDeclined, ReasonSiteMissing)
}
if errorType(applyErr) == SiteAuthorizationError {
return finishRequests(ctx, self, integration.LifecycleDeclined, ReasonSiteChanged)
}
@@ -610,7 +620,9 @@ func SetFedWikiSiteStatusRequestWorkflow(ctx workflow.Context, in SetSiteStatusR
if temporal.IsCanceledError(err) {
return nil, err
}
logger.Error("status change failed", "error", err)
if errorType(err) != SiteMissing {
logger.Error("status change failed", "error", err)
}
if version < 1 {
return compensateStatusChange(ctx, in, reserves, err, self)
}
@@ -630,6 +642,12 @@ func SetFedWikiSiteStatusRequestWorkflow(ctx workflow.Context, in SetSiteStatusR
ClearForceReduced: in.ClearForceReduced,
FinishedAt: workflow.Now(ctx),
}).Get(ctx, nil); err != nil {
if errorType(err) == SiteMissing {
return finishRequests(ctx, self, integration.LifecycleDeclined, ReasonSiteMissing)
}
if errorType(err) == SiteAuthorizationError {
return finishRequests(ctx, self, integration.LifecycleDeclined, ReasonSiteChanged)
}
return nil, err
}
return &LifecycleWorkflowOutput{Status: integration.LifecycleSucceeded}, nil
@@ -657,6 +675,11 @@ func SwapFedWikiActiveSiteRequestWorkflow(ctx workflow.Context, in SwapSiteReque
case err == nil:
case temporal.IsCanceledError(err):
return nil, err
case errorType(err) == SiteMissing:
if err := restoreSwapFarm(ctx, sites); err != nil {
return nil, err
}
return finishRequests(ctx, refs, integration.LifecycleDeclined, ReasonSiteMissing)
case errorType(err) == SiteAuthorizationError:
return finishRequests(ctx, refs, integration.LifecycleDeclined, ReasonSiteChanged)
default:
@@ -677,16 +700,22 @@ func SwapFedWikiActiveSiteRequestWorkflow(ctx workflow.Context, in SwapSiteReque
ParkDomain: in.ParkDomain,
FinishedAt: workflow.Now(ctx),
}).Get(ctx, nil); err != nil {
if errorType(err) != "QuotaExceeded" {
reason := ReasonActiveLimitReached
switch errorType(err) {
case SiteMissing:
reason = ReasonSiteMissing
case SiteAuthorizationError:
reason = ReasonSiteChanged
case "QuotaExceeded":
default:
return nil, err
}
// The sweep parked the site the swap parks, and the pool has no
// slot for the activation: the farm goes back to the console's
// statuses and the member reads the limit.
// A missing or changed identity, or a pool with no activation slot,
// sends surviving originals back before both requests decline.
if err := restoreSwapFarm(ctx, sites); err != nil {
return nil, err
}
return finishRequests(ctx, refs, integration.LifecycleDeclined, ReasonActiveLimitReached)
return finishRequests(ctx, refs, integration.LifecycleDeclined, reason)
}
return &LifecycleWorkflowOutput{Status: integration.LifecycleSucceeded}, nil
}
@@ -789,8 +818,9 @@ func ReconcileFedWikiSiteCreateWorkflow(ctx workflow.Context, in ReconcileSiteCr
// (design D9's boot pass). It reads the farm's entry for the site: when the
// farm holds the site in the requested status the workflow records the status
// and the request succeeds; when the farm holds it otherwise, or not at all,
// or the site row is gone, the request ends failed and the transaction that
// records it removes the hold. A farm that cannot answer keeps the read
// the request ends failed and the transaction that records it removes the hold.
// A missing original row at load or record instead ends declined/site_missing.
// A farm that cannot answer keeps the read
// retrying without an attempt limit, with the request accepted and its slot
// held.
func ReconcileFedWikiSiteStatusWorkflow(ctx workflow.Context, in ReconcileSiteStatusInput) (*LifecycleWorkflowOutput, error) {
@@ -810,7 +840,7 @@ func ReconcileFedWikiSiteStatusWorkflow(ctx workflow.Context, in ReconcileSiteSt
return &LifecycleWorkflowOutput{Status: req.Status, ReasonCode: req.ReasonCode}, nil
}
if !req.SiteFound {
return finishRequests(ctx, self, integration.LifecycleFailed, ReasonRetriesExhausted)
return finishRequests(ctx, self, integration.LifecycleDeclined, ReasonSiteMissing)
}
rctx := workflow.WithActivityOptions(ctx, farmReadBackActivityOptions())
@@ -834,6 +864,12 @@ func ReconcileFedWikiSiteStatusWorkflow(ctx workflow.Context, in ReconcileSiteSt
Status: req.TargetStatus,
FinishedAt: workflow.Now(ctx),
}).Get(ctx, nil); err != nil {
if errorType(err) == SiteMissing {
return finishRequests(ctx, self, integration.LifecycleDeclined, ReasonSiteMissing)
}
if errorType(err) == SiteAuthorizationError {
return finishRequests(ctx, self, integration.LifecycleDeclined, ReasonSiteChanged)
}
return nil, err
}
return &LifecycleWorkflowOutput{Status: integration.LifecycleSucceeded}, nil
@@ -811,7 +811,7 @@ func TestRecordSwapWritesStatusesWithItsOutcomes_DB(t *testing.T) {
done := make(chan error, 1)
go func() {
done <- e.acts.RecordSwapActivity(ctx, RecordSwapInput{
done <- e.acts.RecordSwapActivity(ctx, RecordSwapInput{WorkspaceID: e.ws,
Activate: LifecycleRequestRef{InstanceID: activateID, Generation: gen},
Park: LifecycleRequestRef{InstanceID: parkID, Generation: parkGen},
ActivateDomain: activate, ParkDomain: park, FinishedAt: time.Now(),
@@ -150,7 +150,7 @@ func TestStatusReconcile_DB(t *testing.T) {
e.farm.HoldIn(in.Domain, e.subject, "active")
e.exec(`DELETE FROM fedwiki.sites WHERE site_id = $1`, in.InstanceID)
},
status: "failed", reason: ReasonRetriesExhausted,
status: "declined", reason: ReasonSiteMissing,
},
}
for i, tc := range cases {
@@ -42,7 +42,7 @@ func (a *Activities) SwapFarmSitesActivity(ctx context.Context, input SwapActive
if _, err := a.authorizeSite(ctx, site.id, input.WorkspaceID, site.domain); err != nil {
if errors.Is(err, sql.ErrNoRows) {
return temporal.NewNonRetryableApplicationError(
"a site in the swap no longer exists", SiteAuthorizationError, err)
"a site in the swap no longer exists", SiteMissing, err)
}
return err
}
@@ -162,6 +162,7 @@ type RecordSwapInput struct {
// workflow sends the farm back and declines both requests. A swap that
// releases one slot and takes one, or takes none, is never refused, even on a
// pool already over its limit.
// Missing or changed original identities refuse both writes without retrying.
func (a *Activities) RecordSwapActivity(ctx context.Context, in RecordSwapInput) error {
outcome := func(ref LifecycleRequestRef) integration.LifecycleOutcome {
return integration.LifecycleOutcome{InstanceID: ref.InstanceID, Generation: ref.Generation,
@@ -170,25 +171,31 @@ func (a *Activities) RecordSwapActivity(ctx context.Context, in RecordSwapInput)
var tookSlot bool
apply := func(tx *sql.Tx) error {
q := fwmod.New(tx)
parked, err := q.LockSiteStatusByDomain(ctx, in.ParkDomain)
parked, err := lockRequestSite(ctx, q, in.Park.InstanceID, in.WorkspaceID, in.ParkDomain)
if err != nil {
if errorType(err) != "" {
return err
}
return fmt.Errorf("swap: lock %s: %w", in.ParkDomain, err)
}
activated, err := q.LockSiteStatusByDomain(ctx, in.ActivateDomain)
activated, err := lockRequestSite(ctx, q, in.Activate.InstanceID, in.WorkspaceID, in.ActivateDomain)
if err != nil {
if errorType(err) != "" {
return err
}
return fmt.Errorf("swap: lock %s: %w", in.ActivateDomain, err)
}
tookSlot = activated.Status != "active" && parked.Status != "active"
if _, err := q.SetSiteStatusByDomain(ctx, fwmod.SetSiteStatusByDomainParams{Domain: in.ParkDomain, Status: "readonly"}); err != nil {
if _, err := q.SetSiteStatusByID(ctx, fwmod.SetSiteStatusByIDParams{SiteID: in.Park.InstanceID, Status: "readonly"}); err != nil {
return fmt.Errorf("swap: park %s locally: %w", in.ParkDomain, err)
}
if err := q.SetForceReducedByDomain(ctx, in.ParkDomain); err != nil {
if err := q.SetForceReducedByID(ctx, in.Park.InstanceID); err != nil {
return fmt.Errorf("swap: mark %s parked: %w", in.ParkDomain, err)
}
if _, err := q.SetSiteStatusByDomain(ctx, fwmod.SetSiteStatusByDomainParams{Domain: in.ActivateDomain, Status: "active"}); err != nil {
if _, err := q.SetSiteStatusByID(ctx, fwmod.SetSiteStatusByIDParams{SiteID: in.Activate.InstanceID, Status: "active"}); err != nil {
return fmt.Errorf("swap: activate %s locally: %w", in.ActivateDomain, err)
}
if err := q.ClearForceReducedByDomain(ctx, in.ActivateDomain); err != nil {
if err := q.ClearForceReducedByID(ctx, in.Activate.InstanceID); err != nil {
return fmt.Errorf("swap: clear the parked mark of %s: %w", in.ActivateDomain, err)
}
return nil
@@ -0,0 +1,251 @@
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package workflows
import (
"context"
"errors"
"slices"
"testing"
"time"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integration"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/fedwikitest"
fwmod "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/store"
"github.com/stretchr/testify/mock"
)
// identitySwap records both requests and returns their unchanged workflow input.
func identitySwap(env *reqEnv, sites swapSites) SwapSiteRequestInput {
env.t.Helper()
generation, parkGeneration := env.swapRequests(sites.activateID, sites.parkID, "identity-swap")
return SwapSiteRequestInput{ConnectionID: env.conn, InstanceID: sites.activateID, Generation: generation,
ParkInstanceID: sites.parkID, ParkGeneration: parkGeneration, WorkspaceID: env.ws,
ActivateDomain: sites.activate, ParkDomain: sites.park}
}
// swapIdentityCase states the missing or changed original and expected restoration.
type swapIdentityCase struct {
name string
park, activate, mismatch bool
replacement bool
wantReason string
wantCalls func(swapSites) []string
}
func TestSwapWithMissingSite_DB(t *testing.T) {
for _, row := range []swapIdentityCase{
{"park/absent", true, false, false, false, ReasonSiteMissing, func(sites swapSites) []string {
return []string{sites.park + "=readonly", sites.activate + "=active", sites.activate + "=readonly"}
}},
{"park/replacement", true, false, false, true, ReasonSiteMissing, func(sites swapSites) []string {
return []string{sites.park + "=readonly", sites.activate + "=active", sites.activate + "=readonly"}
}},
{"activate/absent", false, true, false, false, ReasonSiteMissing, func(sites swapSites) []string {
return []string{sites.park + "=readonly", sites.activate + "=active", sites.park + "=active"}
}},
{"activate/replacement", false, true, false, true, ReasonSiteMissing, func(sites swapSites) []string {
return []string{sites.park + "=readonly", sites.activate + "=active", sites.park + "=active"}
}},
{"both/absent", true, true, false, false, ReasonSiteMissing, func(sites swapSites) []string { return []string{sites.park + "=readonly", sites.activate + "=active"} }},
{"both/replacement", true, true, false, true, ReasonSiteMissing, func(sites swapSites) []string { return []string{sites.park + "=readonly", sites.activate + "=active"} }},
{"domain mismatch", true, false, true, false, ReasonSiteChanged, func(sites swapSites) []string {
return []string{sites.park + "=readonly", sites.activate + "=active", sites.activate + "=readonly"}
}},
{"workspace mismatch", false, true, true, false, ReasonSiteChanged, func(sites swapSites) []string {
return []string{sites.park + "=readonly", sites.activate + "=active", sites.park + "=active"}
}},
} {
t.Run(row.name, func(t *testing.T) {
checkSwapIdentity(t, row)
})
}
}
// restorationReadFailure fails only original-site authorization, not outcome writes.
type restorationReadFailure struct{ fwmod.Querier }
func (queries restorationReadFailure) GetSiteByID(context.Context, string) (fwmod.GetSiteByIDRow, error) {
return fwmod.GetSiteByIDRow{}, errors.New("database cannot answer")
}
// restoreIdentityCase states how send-back answers and whether both requests stay open.
type restoreIdentityCase struct {
name string
answer int
database, recovery, missing, mismatch, farmEntry bool
wantAccepted bool
wantAttempts int
wantMinimum bool
wantActivationAttempts int
}
func TestMissingSwapUsesExistingRestoration_DB(t *testing.T) {
for _, row := range []restoreIdentityCase{
{name: "success", wantAccepted: false, wantAttempts: 1, wantMinimum: false, wantActivationAttempts: 2}, {name: "503", answer: 503, wantAccepted: true, wantAttempts: 2, wantMinimum: true, wantActivationAttempts: -1}, {name: "timeout", answer: 408, wantAccepted: true, wantAttempts: 2, wantMinimum: true, wantActivationAttempts: -1},
{name: "database", database: true, wantAccepted: true, wantAttempts: 2, wantMinimum: true, wantActivationAttempts: 1}, {name: "recovery", answer: 503, recovery: true, wantAccepted: false, wantAttempts: 3, wantMinimum: false, wantActivationAttempts: 4},
{name: "database recovery", database: true, recovery: true, wantAccepted: false, wantAttempts: 3, wantMinimum: false, wantActivationAttempts: 2},
{name: "403", answer: 403, wantAccepted: false, wantAttempts: 1, wantMinimum: false, wantActivationAttempts: 2}, {name: "404", answer: 404, wantAccepted: false, wantAttempts: 1, wantMinimum: false, wantActivationAttempts: 2},
{name: "survivor deleted", missing: true, wantAccepted: false, wantAttempts: 1, wantMinimum: false, wantActivationAttempts: 1}, {name: "survivor changed", mismatch: true, wantAccepted: false, wantAttempts: 1, wantMinimum: false, wantActivationAttempts: 1},
{name: "farm entry", farmEntry: true, wantAccepted: false, wantAttempts: 1, wantMinimum: false, wantActivationAttempts: 2},
{name: "farm entry retries", farmEntry: true, answer: 503, wantAccepted: true, wantAttempts: 2, wantMinimum: true, wantActivationAttempts: -1},
} {
t.Run(row.name, func(t *testing.T) {
checkMissingRestoration(t, row)
})
}
}
// checkSwapIdentity checks both declined outcomes, original-only restoration and untouched replacements.
func checkSwapIdentity(t *testing.T, row swapIdentityCase) {
t.Helper()
env := newReqEnv(t, 4, 1)
sites := env.swapSites()
in := identitySwap(env, sites)
workflowEnv := env.testEnv()
wantCalls := row.wantCalls(sites)
changes := make(chan func())
workflowEnv.OnActivity(env.acts.SwapFarmSitesActivity, mock.Anything, mock.Anything).Return(func(ctx context.Context, input SwapActiveSiteInput) error {
if err := env.acts.SwapFarmSitesActivity(ctx, input); err != nil {
return err
}
onIdentityTestThread(changes, func() { changeSwapOriginals(env, sites, row, row.replacement) })
return nil
})
attempts := 0
workflowEnv.OnActivity(env.acts.RecordSwapActivity, mock.Anything, mock.Anything).Return(func(ctx context.Context, input RecordSwapInput) error {
attempts++
return env.acts.RecordSwapActivity(ctx, input)
})
workflowEnv.RegisterDelayedCallback(workflowEnv.CancelWorkflow, time.Hour)
runIdentityWorkflow(changes, func() { workflowEnv.ExecuteWorkflow(SwapFedWikiActiveSiteRequestWorkflow, in) })
if attempts != 1 {
t.Errorf("%s: swap record retried a missing site", row.name)
}
for _, ref := range []LifecycleRequestRef{{in.InstanceID, in.Generation}, {in.ParkInstanceID, in.ParkGeneration}} {
status, reason := env.request(ref.InstanceID, ref.Generation)
if status != integration.LifecycleDeclined || reason != row.wantReason {
t.Errorf("%s: swap did not decline both requests: %s/%s", row.name, status, reason)
if row.mismatch {
t.Errorf("%s: swap mismatch did not restore and decline site_changed", row.name)
}
}
}
if !slices.Equal(env.farm.PatchLog(), wantCalls) {
t.Errorf("%s: restoration addressed a replacement or used the wrong status: %v want %v", row.name, env.farm.PatchLog(), wantCalls)
if row.mismatch {
t.Errorf("%s: swap mismatch did not restore and decline site_changed", row.name)
}
}
for _, site := range []struct {
domain, status string
mark bool
}{{sites.park, "active", false}, {sites.activate, "readonly", true}} {
current, err := fwmod.New(env.db).GetSiteByDomain(env.ctx, site.domain)
if err == nil && (current.Status != site.status || current.ForceReducedAt.Valid != site.mark) {
t.Errorf("%s: swap wrote a surviving or replacement row", row.name)
}
}
figure, count := fedwikitest.FigureAndRows(t, env.db, env.ws)
if figure != count || env.holds(in.InstanceID, in.Generation) || env.holds(in.ParkInstanceID, in.ParkGeneration) {
t.Errorf("%s: swap left wrong usage or holds", row.name)
}
}
// checkMissingRestoration checks retry duration and every PATCH addressed to either original.
func checkMissingRestoration(t *testing.T, row restoreIdentityCase) {
t.Helper()
env := newReqEnv(t, 4, 1)
sites := env.swapSites()
in := identitySwap(env, sites)
workflowEnv := env.testEnv()
if row.answer != 0 {
refusal := env.farm.Refuse(sites.activate, "readonly").With(row.answer)
if row.recovery {
refusal.Times(2)
}
}
workflowEnv.OnActivity(env.acts.SwapFarmSitesActivity, mock.Anything, mock.Anything).Return(func(ctx context.Context, input SwapActiveSiteInput) error {
if _, err := env.acts.Client.SetSiteStatus(ctx, sites.activate, "active"); err != nil {
return err
}
if _, err := env.db.ExecContext(ctx, "DELETE FROM fedwiki.sites WHERE site_id = $1", sites.parkID); err != nil {
return err
}
if row.farmEntry {
return env.acts.SwapFarmSitesActivity(ctx, input)
}
return nil
})
changes := make(chan func())
attempts := 0
workflowEnv.OnActivity(env.acts.RestoreSwapFarmStatusesActivity, mock.Anything, mock.Anything).Return(func(ctx context.Context, input SwapActiveSiteInput) error {
attempts++
onIdentityTestThread(changes, func() {
if row.missing {
env.exec("DELETE FROM fedwiki.sites WHERE site_id = $1", sites.activateID)
}
if row.mismatch {
env.exec("UPDATE fedwiki.sites SET domain = $2 WHERE site_id = $1", sites.activateID, "changed."+env.root)
}
})
activities := *env.acts
if row.database && (!row.recovery || attempts <= 2) {
activities.SiteQ = restorationReadFailure{activities.SiteQ}
}
return activities.RestoreSwapFarmStatusesActivity(ctx, input)
})
workflowEnv.RegisterDelayedCallback(workflowEnv.CancelWorkflow, 30*time.Second)
runIdentityWorkflow(changes, func() { workflowEnv.ExecuteWorkflow(SwapFedWikiActiveSiteRequestWorkflow, in) })
checkRestorationOutcomes(t, env, in, row, attempts)
if (row.wantMinimum && attempts < row.wantAttempts) || (!row.wantMinimum && attempts != row.wantAttempts) {
t.Errorf("%s: restoration attempts = %d, want %d (minimum %v)", row.name, attempts, row.wantAttempts, row.wantMinimum)
}
if (row.wantActivationAttempts >= 0 && env.farm.DomainAttempts(sites.activate) != row.wantActivationAttempts) || env.farm.DomainAttempts(sites.park) != 0 {
t.Errorf("%s: restoration called a missing or mismatched original site", row.name)
}
}
// changeSwapOriginals removes or changes the original rows after farm success.
func changeSwapOriginals(env *reqEnv, sites swapSites, row swapIdentityCase, replacement bool) {
env.t.Helper()
for _, site := range []struct {
remove bool
id, domain, status string
}{
{row.park, sites.parkID, sites.park, "active"}, {row.activate, sites.activateID, sites.activate, "readonly"},
} {
if !site.remove {
continue
}
if row.mismatch {
if row.park {
env.exec("UPDATE fedwiki.sites SET domain = $2 WHERE site_id = $1", site.id, "changed."+env.root)
} else {
other := fedwikitest.New(env.t, env.db).Tenant("Moved")
env.exec("UPDATE fedwiki.sites SET workspace_id = $2 WHERE site_id = $1", site.id, other.Workspace.ID)
}
} else {
env.exec("DELETE FROM fedwiki.sites WHERE site_id = $1", site.id)
if replacement {
env.insertSite(site.domain, site.status, site.status == "readonly")
}
}
}
}
// checkRestorationOutcomes checks both requests during retries and after restoration.
func checkRestorationOutcomes(t *testing.T, env *reqEnv, in SwapSiteRequestInput, row restoreIdentityCase, attempts int) {
t.Helper()
for _, ref := range []LifecycleRequestRef{{in.InstanceID, in.Generation}, {in.ParkInstanceID, in.ParkGeneration}} {
status, reason := env.request(ref.InstanceID, ref.Generation)
if row.wantAccepted {
if status != integration.LifecycleAccepted || attempts < row.wantAttempts {
t.Errorf("%s: restoration stopped while it could not get an answer", row.name)
}
} else if status != integration.LifecycleDeclined || reason != ReasonSiteMissing {
t.Errorf("%s: restoration did not finish with both requests declined", row.name)
}
}
}
@@ -46,7 +46,7 @@ func TestRecordSwapCommitsItsOutcomesWithItsStatuses_DB(t *testing.T) {
done := make(chan error, 1)
go func() {
done <- e.acts.RecordSwapActivity(ctx, RecordSwapInput{
done <- e.acts.RecordSwapActivity(ctx, RecordSwapInput{WorkspaceID: e.ws,
Activate: LifecycleRequestRef{InstanceID: activateID, Generation: gen},
Park: LifecycleRequestRef{InstanceID: parkID, Generation: parkGen},
ActivateDomain: activate, ParkDomain: park, FinishedAt: time.Now(),
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-10-30
@@ -0,0 +1,146 @@
## What the current code does
References describe the branch after #283's archive and the rebase on main, before this change. A lifecycle request carries `instance_id`, independently of the site's lifetime: `internal/db/migrations/00024_integration_delivery_core.sql:195`. Its three operations are create, set_status and delete (`internal/integration/lifecycle.go:33`). A swap is two set_status requests, not a fourth operation.
### Request steps and their identities
| Step | Current identity and decision |
|---|---|
| Intake | `beginRowAction` answers a recorded repeat first, then reads by domain and compares the posted ID and workspace (`internal/integrations/fedwiki/web/site_actions.go:351`). API delete does the same, or resolves a name to an ID when the client supplied no address (`internal/integrations/fedwiki/web/api.go:553`). These are reads before the recording transaction. |
| Recording intent | `newRequestSpec` records the site's ID as the instance, and parameters carry its public domain and target status. A swap records each site's own ID and the other site's ID (`internal/integrations/fedwiki/web/requests.go:122`, `internal/integrations/fedwiki/web/site_actions.go:615`). No existing request stores only a name as its local identity. |
| Slot gate | `usage.hold` locks by ID but treats any missing row as a create, even for an activation; its existing-hold shortcut reads no site (`internal/integrations/fedwiki/usage/usage.go:225`, `:314`). |
| Create and create recovery | The farm creates and reads by domain, its API's address; read-back checks the requesting owner. Local creation reads/inserts by the preallocated instance ID (`internal/integrations/fedwiki/workflows/request_activities.go:41`, `:105`, `:301`). A missing row is expected, not a reason to decline a create. Domain reservations and their release use the instance ID. |
| Status farm write | `ApplyFarmSiteStatusActivity` authorizes by ID, workspace and domain before the domain-addressed farm call. Missing means non-retryable `SiteAuthorizationError`, mapped to `site_changed` (`internal/integrations/fedwiki/workflows/request_activities.go:338`, `internal/integrations/fedwiki/workflows/requests.go:536`). |
| Status read-back and reconcile | After an ambiguous activation, `ReadFarmSiteActivity` reads the farm by domain with no local site check. Reconcile initially loads the site by ID, but missing currently means failed/retries_exhausted. Both paths can reach the same name-based record step (`internal/integrations/fedwiki/workflows/requests.go:546`, `:796`, `internal/integrations/fedwiki/workflows/request_activities.go:499`). |
| Status record | Locks, sets status and clears the force-reduced mark by domain despite carrying `InstanceID` (`internal/integrations/fedwiki/workflows/request_activities.go:379`). |
| Swap farm write and restoration | Both sites are authorized by ID before farm writes. Restoration also reads each original ID, skips missing or mismatched rows and uses the surviving row's recorded status. The immediate send-back after a failed activation uses the previously authorized park domain (`internal/integrations/fedwiki/workflows/swap.go:40`, `:53`, `:78`). |
| Swap record | Carries both request references but locks and writes both statuses and marks by domain (`internal/integrations/fedwiki/workflows/swap.go:136`, `:165`). |
| Permanent delete | Authorizes by ID, calls the farm by domain, releases the placement by ID and deletes the local row by ID. A missing original row succeeds without a farm call. Its record step writes no site (`internal/integrations/fedwiki/workflows/activities.go:244`, `:299`, `internal/integrations/fedwiki/workflows/request_activities.go:414`). |
| Finish and boot settle | Request lookup and finishing use instance and generation, never a site's name. Finishing removes holds and recounts. Boot leaves running workflows alone (`internal/integrations/fedwiki/workflows/request_activities.go:207`, `internal/integrations/fedwiki/workflows/requests.go:265`, `internal/integrations/fedwiki/workflows/settle.go:97`). |
### Which deletions can overlap a request
| Deletion | Can it remove the requested row while the request is open? |
|---|---|
| Retention purge | No, once intake commits first. `purgeArchivedSite` takes the instance lock, locks the still-expired row by ID and checks for an open request before its farm DELETE and local delete (`internal/integrations/fedwiki/workflows/reconcile.go:148`). If purge wins first, today's intake can resume after it and accept the stale action; this change refuses it. #283's one maintenance workflow orders sync, purge and quota work, but does not serialize member requests with maintenance (`docs/database-locks.md:429`). |
| Permanent delete from a member or the JSON API | Yes for its own accepted delete, intentionally. No for another accepted request on that same ID: `recordNew` enforces one open request (`internal/integration/lifecycle.go:247`). A stale status action can pass its first read before deletion; the new status intake check refuses it once the delete has finished. Permanent-delete intake and its idempotent missing-row success remain unchanged. |
| System farm-absence deletion | Not under a member request today. `deleteFarmAbsent` deletes only snapshot rows of the System workspace (`internal/integrations/fedwiki/workflows/sync.go:559`). Requests use the session's workspace, and the System tenant's reserved person cannot log in (`internal/systemtenant/systemtenant.go:36`); no query moves sites between workspaces. The deletion has no request guard, but that alone does not make a System request reachable. |
| Parent deletion, placement cleanup, direct SQL | The site's workspace FK has no ON DELETE CASCADE (`internal/integrations/fedwiki/store/migrations/00001_init.sql:36`); deleting a workspace cannot silently cascade its sites. Placement cleanup deletes placements, not sites. Administrative SQL can delete a row under any request and is covered by the record-step guard. Migrations/test resets are not live deletion paths. |
There are no other production local site-delete callers: member delete, retention and farm-absence sync call the two DELETE queries in `internal/integrations/fedwiki/store/queries/sites.sql:74`. After the purge serialization and ordered maintenance, administrative SQL remains the way to delete a row under another open request. Runs stuck before those fixes also remain. Binding the record steps is defense in depth and makes missing rows terminal; the live intake race reaches the gate or farm authorization.
## Decisions
### 1. Use the identity already recorded
Keep `instance_id` as the sole local identity. Extend `LockSiteStatusByID` (`internal/integrations/fedwiki/store/queries/sites.sql:271`) to return the status, force-reduced mark, workspace and domain. Status and swap record steps use it, verify the recorded workspace and domain, and use ID variants of the status and mark writes. Remove a domain query only if this leaves it with no production caller; keep the maintenance queries that still have callers. Preserve the archive-time and placement-servability triggers and the usage recount.
Both swap rows must be locked and checked before either write. A mismatch is still `SiteAuthorizationError`/`site_changed`. A missing row is a distinct top-level non-retryable `SiteMissing` error; unrelated database errors remain ordinary retryable errors. The transaction rolls back on either refusal. `finishAllWith` still detects an already-committed success before running the site callback, so losing the activity's answer does not reinterpret a completed request after its site is deleted.
Rejected: a name lookup followed by an ID comparison. It prevents one wrong write but still makes a reusable name part of local identity. Rejected: a site FK on the request or another ID in parameters. History must survive deletion, and the existing instance plus each swap's peer ID already supplies everything needed. No schema or stored-parameter change is needed.
The farm continues to use domain addresses because its API has no console site-ID address. Keep `ReadFarmSiteActivity`, its inputs and the immediate send-back inside `SwapFarmSitesActivity` unchanged. Their extra checks would guard administrative deletion inside an attempt or end an unanswered farm read; neither is needed to bind the record step. Status read-back still waits while the farm cannot answer. When it proceeds to recording, the ID check applies. Reconcile keeps its initial ID existence read and gains the record-step guard. No requirement governing unanswered farm reads changes.
### 2. Missing sites end requests, not record retries
`SiteMissing` from a status farm write or record step ends the request `declined` with `site_missing` through `finishRequests`. Both `statusChangeFailed` and the existing pre-slot-gate branch `compensateStatusChange` handle it (`internal/integrations/fedwiki/workflows/requests.go:536`, `:571`). Reconcile does the same if the ID is missing at load or record time (`:812`, `:829`). The slot gate distinguishes operations: a create needs no row; an activation without its original row returns `usage.ErrSiteNotFound`, which `HoldSiteSlotActivity` maps to `SiteMissing`. Check the activation's row even when a hold already exists, before the shortcut. `gateActivation` maps that error to declined/site_missing, not quota_unavailable or active_limit_reached. Do not change unrelated gate-error policy.
Finishing a decline deletes the hold and recounts in the same READ COMMITTED transaction as both outcome and finish time. A failed finish retries the database operation; the missing-site check itself is never retried. An archive usually has no hold: its active row counted until deletion, so describe this as removing any hold and recounting, not an unconditional arithmetic decrement.
For a swap, missing either original ID refuses the entire local transaction. At either the farm step or record step, call `restoreSwapFarm(ctx, sites)`, then finish both requests declined/site_missing together. This also covers an earlier farm attempt that changed the surviving site before the next attempt discovers the other site gone. No surviving or replacement local row gets the proposed status or mark.
Keep `RestoreSwapFarmStatusesActivity` unchanged: it authorizes original IDs, skips absent or mismatched rows and farm 404s, and tries the other site before ending on a final refusal. It retries while the farm cannot answer; both requests stay accepted during that wait. A failed DB read in restoration also retries, not masquerades as a missing row. A final refusal is logged and the workflow then declines both requests; cancellation retains its current behavior (`internal/integrations/fedwiki/workflows/swap.go:78`, `internal/integrations/fedwiki/workflows/requests.go:699`). Use this existing rule rather than a second retry policy or a partial local swap.
Permanent delete keeps its existing exception: a missing original row succeeds, with no farm call and no replacement touched. A create still expects no row and inserts its allocated ID. A missing request record is `LifecycleRequestMissing`, not a missing site, and still fails visibly in history.
### 3. Refuse a missing status target under the existing instance lock
A page loaded before deletion is already refused if the row is absent at `beginRowAction`'s read. The live gap is retention deleting between that read and recording. In a READ COMMITTED transaction, take the instance lock retention already takes, answer recorded addresses from core, then read the original site by ID with `GetSiteByID` and check workspace and domain before calling `integration.RecordLifecycleRequest`. Purge-first makes the read see no row; intake-first makes purge see the accepted request and skip. Add no site-row lock.
Keep this provider-specific check in FedWiki store helpers: one opens the single-request transaction; the batch form uses an existing transaction for rotations and swaps. Both use the existing core content/generation decisions, not copies. A swap takes activation then park instance locks before its reads, computes the park generation under its lock, validates both IDs before inserting either request and rolls back both records and the cooldown on refusal. Return a repeat before selecting a new park candidate or checking cooldown. Preserve #283's candidate ranking.
Wire ordinary status, rotation with or without cooldown, and both swap requests. Leave partial and API permanent delete unchanged: a missing original row already completes successfully without a farm call. Creates still need no row.
For a missing or foreign primary target, keep HTTP 200 `Site not found`, without a refresh: `siteActionResult`'s message branch sets no HX-Trigger (`internal/integrations/fedwiki/web/site_actions.go:773`). A vanished park candidate uses `siteActionConflict`: HTTP 200, `This site changed since the list loaded.` and a refresh, with its preserved alert target (`:787`). A refusal records no request, starts no workflow, takes no hold and stamps no cooldown.
A failed protected read returns an error, never a missing-site answer. This rule is scoped to the new protected read. The first domain reads currently report any error as not found (`internal/integrations/fedwiki/web/site_actions.go:368`, `internal/integrations/fedwiki/web/api.go:571`); correcting them is outside this change. No ride-along or API answer change is included.
Rejected: re-read before acquiring the instance lock, which misses a purge already in progress. Rejected: site-row locks, which protect only the administrative-deletion window here. Rejected: checking existence before answering recorded repeats, which breaks their existing answers after deletion.
### 4. Lock order and overlaps with #283
Against `docs/database-locks.md:379`, `:402`, `:576`: intake retains the workspace advisory lock where already required, then the lifecycle instance locks in activation/park order, then request insertion and the cooldown stamp. It takes no site-row or usage lock and makes no farm call. `withWorkspaceTx` and the new single-request transaction use READ COMMITTED: each read must see the commit preceding the instance-lock grant, even when the session default is REPEATABLE READ. No new lock or lock family is added.
Record transactions retain instance locks, site rows (park then activate for a swap), trigger-written placement rows, hold removal, usage lock/recount and request outcomes. The missing-site record transaction rolls back before restoration or the separate decline transaction. That finish takes no site row.
The implementation overlaps #283 in `web/site_actions.go` (`changeStatus`, `swapActiveSite`, `withWorkspaceTx`, and `requestsView.notices` for copy), the store query/generated files, `docs/database-locks.md` and `workflows/README.md`. Preserve candidate ranking and all maintenance scheduling/operator behavior. The shared boot-settle requirement starts from the archived wording in `openspec/specs/fedwiki-sites/spec.md:1112`, including “before it ensures or pauses the maintenance schedule”.
## Failure table
| Path | Today | After this change |
|---|---|---|
| New status/swap intake loses a site before the instance-lock grant | Can record and start against the deleted ID | Refuses under the existing instance lock; no request, workflow or cooldown |
| Repeat after deletion | Reads its request first | Same stored answer, no site required |
| Activation gate sees no row | Can hold a slot or report the limit | Declined/site_missing; finish removes any hold and recounts |
| Status farm authorization sees no row | Declined/site_changed | Declined/site_missing; no farm call; both current and pre-slot-gate branches handle it |
| Status read-back cannot reach farm after local deletion | Retries the domain | Unchanged; if it reaches the record step, the ID guard refuses a missing original |
| Status reconcile loads a missing ID | Failed/retries_exhausted | Declined/site_missing; no farm read |
| Status record sees missing ID or replacement name | Retries missing name or writes replacement | Non-retryable SiteMissing; workflow declines, removes hold and recounts |
| Swap farm or record step sees one/both IDs missing | Farm step declines generically without restoration; record retries or writes replacement | No local swap writes; restore surviving originals through the existing activity, then decline both/site_missing |
| Restoration cannot read DB or reach farm | Retries | Same; both requests stay accepted until restoration completes or receives a final refusal |
| Restoration finds missing/mismatched ID, 404 or final refusal | Skips missing/mismatched ID and 404; logs other final refusal | Same activity; missing-site workflow branch then declines both/site_missing |
| Wrong workspace or domain under record lock | Name-based record can write it | Non-retryable authorization refusal; status declines/site_changed; swap restores and declines/site_changed |
| Successful record loses its answer | Finished-request shortcut avoids a second write | Same, even if the site is deleted before retry |
| Record or finish has a transient DB failure | Retries | Same; never interpreted as missing |
| Create has no local row | Creates its allocated ID | Same |
| Delete has no local row | Succeeds without farm call | Same intake, API answer, record and recount |
| Missing lifecycle request record | Non-retryable history failure | Same; never disguised as a missing site |
## Copy
Use one reason, `site_missing`, for a status request and both swap requests. Add no unused standalone reason sentence: the status/swap surface is the member notice, and the API exposes neither action. Render missing notices directly, before action inference from an absent row (`internal/integrations/fedwiki/web/requests.go:87`). If the activated ID is gone, including when both are gone, use the first row below. If only the parked ID is gone, get its domain from the peer request with the same workflow ID in the already-loaded request rows (`internal/integrations/fedwiki/web/site_actions.go:823`). A replacement with the same name is not the original ID. Keep one warning notice and dismiss both swap requests.
| Where | Old | New | Status |
|---|---|---|---|
| Missing original site, including a swap's activated site | Farm authorization: `<domain> wasn't <action>. This site changed since the request.`; reconcile: `<domain> wasn't kept active. Try again later.`; record: no row and no notice, the list re-renders every three seconds; for a swap the parked site shows Pending | `<domain> was deleted.` | approved |
| Only the parked swap site is missing | Farm authorization: `<domain> wasn't kept active. This site changed since the request.`; record: Pending with no notice | `<domain> wasn't kept active. <parked domain> was deleted.` | approved |
| New status/swap intake finds its primary target missing after the first read | Accepted response, no message | `Site not found` (existing HTTP 200 response, no refresh) | approved |
| New swap intake finds its selected park target missing | Accepted response, no message | `This site changed since the list loaded.` (existing HTTP 200 conflict response with refresh) | approved |
## Landing
The upgrade steps go in the landing commit note, not in `docs/`, a README or a script; this section states them for review. The note calls for a read-only count of accepted FedWiki status/swap requests whose instance has no site row or whose recorded domain now names another ID, then termination of each stuck record-step execution, deployment, and a check that none remains accepted. Boot settle handles closed requests older than two minutes at boot and again two minutes later (`internal/integrations/fedwiki/workflows/settle.go:97`): held activations reach the status reconcile and decline/site_missing; other requests, including both swap halves, finish failed/retries_exhausted through `finishAllWith`. Inspect any replacement an old record step already wrote and each surviving swap site's farm status by hand. No new history compatibility code, backfill or manual SQL finish is added; existing unrelated version gates remain.
## Open questions
The following is one separate bug statement, not work included here.
### FedWiki maintenance can change or delete a newer site that reuses a deleted site's name
### What happens
A site a member just created can be made read-only, or lose its console row, by maintenance meant for the deleted site that had its name.
Periodic wiki maintenance reads sites before it changes them. Its quota sweep makes excess sites read-only; its sync copies the wiki farm's state into the console. Both can later find a site by its reusable name instead of the ID they read.
During a quota sweep's farm call, a member can permanently delete a listed site and create another with the same name. The sweep can then write the old site's status onto the replacement and count usage for the old site's organization.
The sync has the same identity gap in its status write and in its removal of farm-absent System sites, which are sites the console discovered without a member owner. Those paths require a deletion without a member request between the snapshot and write, such as administrative SQL: the status write skips a site with a request newer than its read, which a member's delete leaves; the removal lists only System sites, on which members cannot record requests; and the archive purge runs after the sync in the same maintenance run, never beside it. The absence-removal path can delete a replacement belonging to a member, recount the old organization's usage and release the old site's domain placement, the record connecting its name to its site.
### What should happen
Maintenance should write or delete only the site ID it read, rechecking eligibility under that row's lock. A missing or replaced site should be left alone, and any count or domain release should concern the row actually changed.
### Where
`internal/integrations/fedwiki/workflows/reconcile.go and sync.go`
### Steps
- `internal/integrations/fedwiki/workflows/sync.go:559`, `deleteFarmAbsent`: checks snapshot ownership, deletes by name.
- `internal/integrations/fedwiki/workflows/sync.go:513`, `writeStatus`: locks the current name but checks requests on the snapshot ID.
- `internal/integrations/fedwiki/workflows/reconcile.go:269`, `:309`, `:369`, `parkSite`, `reactivateSite`, `sweepLocally`: check the listed ID before the farm call, then write by name and recount the listed workspace.
### Why it matters
A newly created site can receive an unrequested status or lose its console row. The console can report the wrong organization's usage or leave a domain placement pointing at the deleted replacement. Serial maintenance runs do not prevent a member action between maintenance's read and write.
@@ -0,0 +1,30 @@
## Why
A FedWiki lifecycle request records a member's create, status change or permanent delete. Its status and swap record steps nevertheless find the site by its name. A deleted row makes the record step retry for ten years; a replacement at the same name can receive the old request's status (#284).
Retention now serializes with accepted requests, and maintenance runs no longer overlap. System sync deletes absent System sites, but members cannot record requests there. Administrative deletion can still remove a requested row; older runs can already be stuck. Intake also permits retention to delete a site between its first read and the request's commit.
## What Changes
- Lock and update status and swap sites by their recorded IDs, checking workspace and domain under the lock.
- Decline a status request when its original row is found missing at the gate, farm authorization, reconcile load or record step. Remove any hold and recount in the transaction that finishes it.
- If either swap site is gone, write neither local status, send surviving original sites back through the existing restoration activity, then decline both requests with `site_missing`. Keep its existing retries while the farm cannot answer.
- Refuse a new status or swap request whose site retention deleted before intake acquired its instance lock. Read by ID under the existing lock at READ COMMITTED; add no site-row lock. Preserve repeats, permanent delete and API answers.
- Show which site was deleted, using the peer request's recorded domain when only the parked site is gone.
## Capabilities
### New Capabilities
None.
### Modified Capabilities
- `fedwiki-sites`: request identity, status/swap intake, missing-site outcomes, swap recovery, status reconciliation and the slot gate.
- `provider-lifecycle-contract`: permit a decline when the original instance disappeared before recording or during the provider's specified recovery.
## Impact
FedWiki's activities, workflows, usage gate, status/swap intake and notice rendering change, with ID queries and regenerated store code. No schema, workflow-input, request-parameter, route or form-field change is needed. Both swap requests already record their site IDs. Capture the changed member notices at landing.
The landing commit note instructs the operator to terminate affected stuck runs and let boot settle finish their requests through tested code. No history compatibility code or manual SQL repair is added. Maintenance's stale-name writes and deletes are a separate issue, stated in the design's open questions.
@@ -0,0 +1,390 @@
## MODIFIED Requirements
### Requirement: Lifecycle workflows act on the site they were authorized for
A delete, status-change or swap request SHALL start its workflow with the
site's `site_id` (a swap: both sites') and the authorizing `workspace_id`,
and the workflow id SHALL be derived from that identity and the request's
generation (`delete-fedwiki-site-<site_id>-<generation>`,
`set-fedwiki-status-<site_id>-<generation>`, and
`swap-fedwiki-site-<site_id>-<generation>` keyed by the site being activated),
and a create's workflow id SHALL be `create-fedwiki-site-<instance_id>`. A
second, different request for a site while one is open SHALL be refused by the
one-open-request rule of its lifecycle requests and answered as an action
already in progress. Before an activity mutates a site, provider-side
or locally, it SHALL read the row by `site_id` and SHALL refuse, without a
provider call and without retry, when its `workspace_id` differs from the
input or when its domain differs from the domain the request named. Local status and swap record steps SHALL lock and update by the recorded
`site_id`, including force-reduced marks, and SHALL check workspace and domain
under that lock. A swap SHALL check both original rows before writing either.
When the row is gone, a status change SHALL end `declined` with `site_missing`
and a swap SHALL end both requests `declined` with `site_missing`, without
retrying the missing-site step or writing another site's row. A delete
SHALL complete without a provider call: its local delete is its last step, so
a missing row means the deletion already finished or the site left another
way, and whatever now holds the name is another site.
#### Scenario: A reallocated name is not mutated
- **WHEN** a delete activity runs for `site_id` A whose row has been removed
and whose former domain now names a site of another workspace
- **THEN** the activity SHALL complete without any FarmManager call, and the
other workspace's site SHALL be unchanged
#### Scenario: A site that changed hands is refused
- **WHEN** a status-change activity runs for a `site_id` whose row now
carries another `workspace_id`
- **THEN** the activity SHALL refuse with a non-retryable error before any
FarmManager call
- **AND** the request SHALL end `declined` with the reason `site_changed`
#### Scenario: A second delete while one runs is refused
- **WHEN** a member requests deletion, under a new generation, of a site whose delete request is
still `accepted`
- **THEN** the request SHALL be answered as already in progress, no
second request SHALL be recorded, and no second workflow SHALL start
#### Scenario: A legitimate later action starts
- **WHEN** a site's status-change request has finished and the member
requests another status change for the same site under a higher generation
- **THEN** a new request SHALL be recorded and a workflow whose id carries the new generation SHALL start
#### Scenario: A status record cannot adopt a reused name
- **WHEN** the farm answered a status request for site A, A's row was deleted, and another site now has A's domain
- **THEN** the record step SHALL refuse without retry and SHALL change no replacement row or mark
- **AND** the workflow SHALL finish the request `declined` with `site_missing`, removing any hold and recounting its pool
#### Scenario: A record detects changed authorization
- **WHEN** the original ID still exists but its workspace or domain differs from the request at record time
- **THEN** the step SHALL refuse without writing a site and the request SHALL end `declined` with `site_changed`
#### Scenario: A database error is not a missing site
- **WHEN** an ID lookup in a record step fails for a reason other than no row
- **THEN** the step SHALL remain retryable and SHALL NOT finish the request as a missing-site decline
### Requirement: A repeated FedWiki lifecycle request is answered from its record
When a request arrives, the console SHALL, in one transaction that serializes requests for the instance, compare its address with what is recorded and answer as follows: an address that holds nothing, with a generation above the instance's latest, SHALL be recorded and accepted only after the operation's intake checks pass; an address holding a request with the same content that is still `accepted` SHALL be answered accepted, and nothing new SHALL start; an address holding a request with the same content that has finished SHALL be answered with its stored outcome (`succeeded`, `declined` or `failed`, with its reason code); an address holding a request with different content, or an empty address whose generation is at or below the instance's latest, SHALL be answered as a conflict, and nothing SHALL be recorded or started. A request is `declined` when the instance's rules refused it before anything changed on the farm: a full site limit, a name the domains registry refuses, a reactivation at the active limit, a site that changed hands, or a farm that holds the name for someone else; also when its original site disappeared before the local outcome could be recorded, even if the farm call had succeeded. A request is `failed` when the console tried, could not finish and compensated, or when the workflow never started or closed without recording an outcome. A row action SHALL look its request up by instance and generation before it resolves the site's domain, so a repeat that arrives after a permanent delete finished is still answered from its record; when the domain now names a site whose `site_id` differs from the posted instance, the answer SHALL be a conflict. A conflict on a row action SHALL answer "This site changed since the list loaded." and refresh the list. A recorded request in another workspace SHALL be answered on the row actions exactly as a site that does not exist ("Site not found") and on the create form and the JSON API as a conflict, never with its content. A create SHALL be looked up by its address on the form, and by its instance on the JSON API, before the requested name is checked against the domains registry, so a repeat of a create whose workflow already placed the name is answered from its record; a create whose instance is not recorded but already names a site SHALL be answered as a conflict, and nothing SHALL be recorded or started.
#### Scenario: A repeat after success returns the stored outcome
- **WHEN** a member's archive request has finished `succeeded` and the same instance identifier, generation and content arrive again
- **THEN** the answer SHALL be the stored outcome and no workflow SHALL start
#### Scenario: An older tab is a conflict
- **WHEN** a request for a site finished under generation 20 and a stale tab then submits a request for the same site under generation 10
- **THEN** the answer SHALL be a conflict, the row action SHALL show "This site changed since the list loaded.", and nothing SHALL be recorded or started
#### Scenario: Same address, different content
- **WHEN** a request arrives whose instance identifier and generation are already recorded with a different operation or different parameters
- **THEN** the answer SHALL be a conflict and nothing SHALL be recorded or started
#### Scenario: Another workspace's request is not disclosed
- **WHEN** a member submits a row action whose instance identifier and generation address a request recorded in another workspace
- **THEN** the answer SHALL be identical to the answer for a site that does not exist
#### Scenario: A custom-domain create repeated after its name was placed
- **WHEN** a create for a custom domain was accepted, its workflow has placed the name, and the same form submission or API create arrives again
- **THEN** the answer SHALL be the one the first submission got, from the record
- **AND** nothing SHALL be recorded or started
#### Scenario: A create under an existing site's identifier
- **WHEN** a create arrives whose instance identifier has no recorded request but names an existing site
- **THEN** the answer SHALL be a conflict, and nothing SHALL be recorded or started
### Requirement: Each FedWiki lifecycle workflow records its own outcome
Every FedWiki lifecycle workflow SHALL end by setting its request's status, reason code and finish time, and SHALL fail visibly in its own history when no recorded request matches. A create's site identifier SHALL be the request's instance identifier, used for the registry placement and the `fedwiki.sites` row; a create's success SHALL insert the `fedwiki.sites` row and mark the request `succeeded` in one transaction, so the list never shows the pending create and the new site together. A status change SHALL write the site's new status, remove its request's hold, recount the pool and write its request's outcome in one transaction, and the clearing of a site's force-reduced mark after a rotation SHALL commit in that same transaction. Every outcome, declined and failed ones included, SHALL be recorded under the request's instance lock in a transaction that removes the request's hold and recounts the pool. Workflow inputs SHALL carry the request's connection identifier, instance identifier and generation. A run that was in flight when the console was upgraded SHALL finish on the code that started it: the create and status workflows SHALL choose between their recorded quota steps and the slot gate with a version check before their first command.
#### Scenario: A successful create appears once
- **WHEN** a create workflow inserts its site row
- **THEN** in the same transaction the request SHALL become `succeeded`
- **AND** no rendering of the list SHALL show both the pending create row and the new site
#### Scenario: A swap's retry changes no count
- **WHEN** a swap's farm step fails and is retried, and the swap then succeeds
- **THEN** the pool's figure SHALL equal its active rows and holds, both sites' statuses and both requests' outcomes SHALL have been written in one transaction
- **AND** when the swap fails instead, both sites SHALL keep the statuses they had, on the farm and in the console
- **AND** while the farm cannot answer the PATCHes that send the sites back, both requests SHALL stay `accepted`
- **AND** a 404 for a site SHALL leave that site out of the send-back, and any other refusal from the farm, a 4xx other than 408 or 429, SHALL finish both requests `failed` without another attempt once the other site's send-back has been tried
#### Scenario: A finish with no request fails
- **WHEN** a lifecycle workflow tries to record the outcome of a request that does not exist
- **THEN** the recording step SHALL fail instead of updating nothing
#### Scenario: A run started before an upgrade finishes
- **WHEN** a create workflow started by the previous release is still running when the new release starts
- **THEN** the run SHALL complete without a replay failure
#### Scenario: A status change started before an upgrade finishes
- **WHEN** a status change to `active` started by the previous release is still running when the new release starts
- **THEN** the run SHALL complete without a replay failure, and its outcome SHALL leave the pool's figure equal to its active rows and holds
#### Scenario: A declined request frees its hold
- **WHEN** a create holding a slot ends `declined` with `name_taken`
- **THEN** in the transaction that records the outcome its hold SHALL be removed and the pool's figure SHALL fall by one
#### Scenario: A missing swap site declines both requests
- **WHEN** either original swap site is missing at a farm or local record step
- **THEN** no local swap status or mark SHALL be written and the missing-site step SHALL NOT retry
- **AND** the workflow SHALL send surviving original sites back as a failed swap does, skipping absent or mismatched IDs and farm 404s
- **AND** while the farm cannot answer, or the restoration's database read fails, both requests SHALL stay `accepted` and restoration SHALL retry
- **AND** after restoration completes or receives a final refusal, unless cancelled, the workflow SHALL finish both requests `declined` with `site_missing` in one transaction that removes their holds and recounts
- **AND** a final farm refusal SHALL be logged and SHALL NOT stop the other site's send-back from being tried
- **AND** a same-name replacement SHALL receive no local write or restoration call
#### Scenario: A committed success stays successful after deletion
- **WHEN** the record transaction succeeded but its answer was lost, and the original site is deleted before the retry
- **THEN** the retry SHALL return the recorded success without reapplying any site write or changing the outcome to declined
#### Scenario: A failed decline commit keeps its hold until retry
- **WHEN** a missing-site decline cannot commit
- **THEN** the outcome, hold removal and recount SHALL all roll back
- **AND** retrying the finish SHALL commit the decline and release the hold together
### Requirement: A FedWiki lifecycle request with no outcome is settled at boot
When Temporal is configured, FedWiki's startup SHALL, before it ensures or pauses the maintenance schedule, describe the workflow of every `accepted` request older than two minutes and act on the answer, at boot and once more after two minutes have passed since boot: when the workflow does not exist, the request SHALL end `failed` with `not_started`; when it is running, nothing SHALL change; when it closed and the request is a create, the console SHALL start a reconcile workflow, identified by the instance, that reads the farm's entry for the domain as a failed create does and then records the site and succeeds or compensates and fails; when it closed and the request is a status change to `active` that still holds a slot, the console SHALL start a status reconcile workflow, identified by the instance and generation, that reads the site's entry from the farm and records the status and succeeds when the farm holds the site in the requested status, ends the request `failed` with `retries_exhausted` when the farm holds it otherwise or not at all, ends it `declined` with `site_missing` when its original site row is gone at load or record time, and keeps the request `accepted` and its slot held while the farm cannot answer; when it closed and the request is any other operation, the request SHALL end `failed` with `retries_exhausted` and the site SHALL keep its observed status; on any other error, nothing SHALL change. Every outcome settle records SHALL be written under the request's instance lock, with the request's hold removed and the pool recounted. A request that never started SHALL NOT be restarted. The reconcile workflows' input SHALL carry identifiers only.
#### Scenario: A process that died before the start
- **WHEN** the console boots with an `accepted` request older than two minutes whose workflow Temporal does not know
- **THEN** the request SHALL end `failed` with the reason `not_started`, and no workflow SHALL start for it
#### Scenario: The pass runs with the sync disabled
- **WHEN** the console boots with the FedWiki sync disabled and an `accepted` request whose workflow closed without recording an outcome
- **THEN** the pass SHALL still settle the request
#### Scenario: A closed create the farm holds
- **WHEN** the pass finds a closed create workflow whose request is `accepted` and the farm's entry names the requester as owner
- **THEN** the reconcile workflow SHALL insert the site row under the instance identifier and the request SHALL end `succeeded`
#### Scenario: A closed activation the farm made active
- **WHEN** the pass finds a closed status change to `active` whose request is `accepted` and holds a slot, and the farm's entry shows the site active
- **THEN** the status reconcile SHALL write the site `active`, remove the hold, and end the request `succeeded`, and the site SHALL count once
#### Scenario: A closed activation the farm did not make
- **WHEN** the status reconcile reads the farm's entry and the site is `readonly`
- **THEN** the request SHALL end `failed` with `retries_exhausted`, its hold SHALL be removed, and the site SHALL keep its `readonly` status
#### Scenario: A request committed just before a quick restart
- **WHEN** the console restarts within two minutes of committing an `accepted` request whose workflow never started
- **THEN** the boot pass SHALL leave it, and a second pass, run once two minutes have passed since boot, SHALL end it `failed` with `not_started`
#### Scenario: Temporal unreachable at boot
- **WHEN** describing a workflow fails with an error other than not-found
- **THEN** the request SHALL stay `accepted`
#### Scenario: A closed activation has lost its original site
- **WHEN** the status reconcile finds its original site ID missing, including when another row has the same name
- **THEN** it SHALL end the request `declined` with `site_missing`, remove its hold and recount
- **AND** it SHALL NOT call the farm after discovering the missing ID or write the replacement row
### Requirement: FedWiki request outcomes show the console's own sentence for a reason code
A declined or failed request SHALL carry a reason code, and the console SHALL describe it only with its own sentence for that code, never text from the farm or a workflow error string. The codes with standalone sentences SHALL be:
- `site_limit_reached` (declined): "You have reached your site limit."
- `active_limit_reached` (declined): "You have reached your active site limit."
- `name_refused` (declined): "That name can't be reserved. Choose another."
- `name_taken` (declined): "A site with this name already exists."
- `site_changed` (declined): "This site changed since the request."
- `quota_unavailable` (failed): "Couldn't check your site limit. Try again later."
- `retries_exhausted` (failed): "Couldn't finish. Try again later."
- `not_started` (failed): "Couldn't start. Try again."
The declined reason `site_missing` SHALL render directly as a notice, with no standalone reason sentence. When the original status-change site or the activated swap site is gone, the notice SHALL read "<domain> was deleted.", including when both swap sites are gone. When only the parked site is gone, it SHALL read "<domain> wasn't kept active. <parked domain> was deleted.", taking the parked domain from the peer request with the same workflow ID. Both swap requests SHALL carry `site_missing`, but only the activation request SHALL show a notice. These notices SHALL use the declined warning tone and existing dismissal behavior; neither SHALL infer the original identity from a same-name replacement.
Every other notice on the site list SHALL read "<domain> wasn't <action>." followed by its reason, with the domain as plain text, where the action is "created", "archived", "restored", "kept active" or "deleted" for a create, an archive, a restore, a keep active (a swap included) and a permanent delete. The reason SHALL be the code's sentence, except that `retries_exhausted`, and a code with no sentence, SHALL read "Try again later." and `not_started` SHALL read "Try again.", because the notice's first sentence already says that the request did not happen. The JSON API's message SHALL be the code's sentence. The Status column SHALL read Pending for an open request and never Declined or Failed. The texts "Join us and become a member for more sites." and every "Please try again later" clause of the workflow error strings these codes replace SHALL NOT be shown.
#### Scenario: A reactivation at the active limit
- **WHEN** a restore of an archived site ends `declined` because active usage is at the limit
- **THEN** its notice SHALL read "<domain> wasn't restored. You have reached your active site limit."
#### Scenario: No farm text reaches the notice
- **WHEN** a create fails with a farm error whose body carries its own message
- **THEN** the notice SHALL show only the console's sentence for the request's reason code
#### Scenario: A missing restore does not need the old row to name its outcome
- **WHEN** a restore ends declined/site_missing after its original row is deleted
- **THEN** its notice SHALL read "<domain> was deleted."
- **AND** it SHALL NOT describe the request as kept active because the old status is unavailable
#### Scenario: Only the parked site disappeared
- **WHEN** a swap ends declined/site_missing, its activated ID still exists and its parked ID is gone
- **THEN** its one notice SHALL read "<domain> wasn't kept active. <parked domain> was deleted."
- **AND** the parked domain SHALL come from the peer request's recorded parameters, not a name lookup
- **AND** dismissing the notice SHALL acknowledge both requests
#### Scenario: Both swap sites disappeared
- **WHEN** a swap ends declined/site_missing and both original IDs are gone
- **THEN** its one notice SHALL read "<domain> was deleted." using the activated request's domain
- **AND** a replacement with that name SHALL NOT change which notice is selected
### Requirement: A slot a request holds in flight is a hold its gate stamps
A create, and a status change to `active` that is not part of a swap, SHALL take its slot in a gate that runs in its workflow before the farm call, in one transaction: lock the request's instance, find the request still `accepted`, lock the site row of an activation (a create has none), lock the pool's usage row, count the pool's active rows and holds, and, only when that count is below the pool's `fedwiki_sites` limit, insert a row into `fedwiki.slot_holds` keyed by the request's identifier and write the new count; an activation whose site row is already `active` is stamped without that comparison. A hold SHALL count toward its pool while its request is `accepted` and its instance has no `active` site row. The transaction that records the request's outcome, whatever the outcome, SHALL delete the request's hold and recount.
The gate SHALL decline a create with `site_limit_reached`, and an activation with `active_limit_reached`, when the count has reached the limit or the pool has no `fedwiki_sites` entitlement. It SHALL fail the request with `quota_unavailable` when the pool, the entitlement or the usage row cannot be read, and it SHALL NOT treat a failed read of the entitlement as an absent one. An activation gate SHALL first require the original site row by ID, even when its hold already exists; a missing row SHALL end the request `declined` with `site_missing`, removing any existing hold and recounting in its finish transaction. A create SHALL require no site row. A gate whose hold already exists and whose required site still exists SHALL pass without comparing quota again, so a retry after a lost answer does not decline the request it already admitted. An activation whose site row is already `active` SHALL be stamped without comparing the count with the limit, and its hold SHALL count nothing while the row is active. A swap SHALL take no hold: its record step takes the slot for the site it activates in the transaction that writes both statuses.
#### Scenario: Two creates for the last slot
- **WHEN** a pool's limit is 3, it holds two active sites, and two creates reach their gates at once
- **THEN** one gate SHALL stamp a hold and the other SHALL wait on the usage row, count the first's hold, and decline its create with `site_limit_reached`
#### Scenario: A create recorded but not yet gated holds nothing
- **WHEN** a pool's limit is 3, it holds two active sites, and two creates have been recorded but neither has reached its gate
- **THEN** the pool's figure SHALL read 2, and the first create to reach its gate SHALL hold the last slot
#### Scenario: A retried gate keeps its slot
- **WHEN** the gate's transaction commits a hold that takes the pool's last slot and its answer is lost, and the activity runs again
- **THEN** the second run SHALL find the hold and pass
#### Scenario: A failed entitlement read fails the create
- **WHEN** the gate cannot read the pool's `fedwiki_sites` entitlement because the database returns an error other than no rows
- **THEN** after its attempts the request SHALL end `failed` with `quota_unavailable`, and the notice SHALL say that the site limit could not be checked
#### Scenario: An activation of a site that is already active is stamped
- **WHEN** the quota sweep has reactivated a parked site, and a member's restore of the same site then reaches its gate with the pool at its limit
- **THEN** the gate SHALL stamp a hold without declining, and the pool's figure SHALL be unchanged
- **AND** when the sweep parks the site again before the restore records its outcome, the hold SHALL count from the park's commit, and the restore's record SHALL leave the site counted once
#### Scenario: A finished request's hold never counts
- **WHEN** a hold row remains for a request that is no longer `accepted`
- **THEN** no recount SHALL count it
#### Scenario: A run started before the upgrade holds no hold
- **WHEN** a create started by the previous release passed its quota step before the upgrade and is still running
- **THEN** it SHALL finish on its recorded commands without a hold, and its site SHALL count from the moment its row is inserted
#### Scenario: A missing activation is not a create
- **WHEN** an activation's original site is deleted before its gate, whether or not the request already holds a slot and whether or not the pool is full
- **THEN** the gate SHALL NOT stamp a new hold or answer active_limit_reached
- **AND** the request SHALL end `declined` with `site_missing` and its finish SHALL remove any existing hold and recount
## ADDED Requirements
### Requirement: Status and swap intake reads the original sites under the instance locks
Before recording a new FedWiki status or swap request, intake SHALL read the original site by ID under the request's existing instance lock in the recording transaction at READ COMMITTED and verify its workspace and domain. It SHALL add no site-row lock. A swap SHALL take its existing instance locks in activation/park order, read both original sites before inserting either request, and roll back both records and its cooldown stamp when either check fails. Repeats SHALL still be answered from their records, including after deletion. Create and permanent-delete intake SHALL remain unchanged.
A missing or foreign primary target SHALL refuse a new request synchronously with HTTP 200 and "Site not found", without a list refresh. A missing selected park target SHALL use the existing HTTP 200 conflict response, "This site changed since the list loaded.", with a list refresh. A same-name replacement SHALL never be substituted for the original ID. Other identity conflicts SHALL retain their conflict answer. Refusal SHALL record no request, start no workflow, take no hold and stamp no cooldown. A failure of the protected ID read SHALL remain an error, never an absent-site answer.
#### Scenario: Purge wins before intake
- **WHEN** retention holds the instance lock and deletes the site before a stale status action can record
- **THEN** intake SHALL see the deletion after obtaining the lock and refuse without recording or starting work
#### Scenario: Intake wins before purge
- **WHEN** intake commits an accepted request before retention obtains the instance lock
- **THEN** retention SHALL see the accepted request and leave the site alone
#### Scenario: The session's default isolation cannot hide the purge commit
- **WHEN** a status or swap intake waits for retention's instance lock on a session whose default isolation is REPEATABLE READ
- **THEN** its transaction SHALL explicitly use READ COMMITTED and its protected ID read SHALL see retention's committed deletion
#### Scenario: The activated swap site disappears
- **WHEN** the swap's original activation site is missing at the protected read
- **THEN** intake SHALL answer "Site not found" without a refresh, record neither request and leave the cooldown unchanged
#### Scenario: The selected park site disappears
- **WHEN** the swap's selected park site is missing at the protected read
- **THEN** intake SHALL answer "This site changed since the list loaded." with a refresh, record neither request and leave the cooldown unchanged
#### Scenario: A status repeat survives deletion
- **WHEN** an identical status request is repeated after its original site row is gone
- **THEN** intake SHALL return its recorded answer without requiring a site row
#### Scenario: A failed protected read does not claim a site is gone
- **WHEN** the protected intake ID read fails because the database cannot answer
- **THEN** intake SHALL return its existing operation-error response and SHALL NOT answer "Site not found"
### Requirement: A site's status is changed via the provider
Changing a site's lifecycle state SHALL apply the change provider-side first (FarmManager `PATCH /sites/:domain {status}`) and then update the local observed `status`, recount the pool's active usage and record the request's outcome in a single local transaction. A transition to `active` (reactivate or restore) SHALL be quota-gated by the slot gate before the provider call; a transition to `readonly` or `archived` SHALL free the site's slot through that transaction's recount. If the provider refuses the change (a non-retryable answer, or a site that no longer exists), the local status SHALL be left unchanged, and the transaction that records the outcome SHALL remove any hold the gate stamped. If an activation that holds a slot fails in any other way (its attempts are exhausted or its last attempt times out), the workflow SHALL read the site's entry from the farm before it releases the hold: when the farm holds the site in the requested status, the change SHALL be recorded as a success, unless the site's original row is gone at the record step, which ends the request `declined` with `site_missing`, and otherwise the failure SHALL be recorded and the hold removed; a farm that cannot answer SHALL leave the request open with its hold.
#### Scenario: Set read-only applies provider-side then projects locally
- **WHEN** an active site is set read-only
- **THEN** the system SHALL PATCH the farm to `status = readonly`
- **AND** on success SHALL set the local `status = 'readonly'`, and the pool's figure SHALL fall by one in the same transaction
#### Scenario: Reactivation refused when at the active limit
- **WHEN** a request is made to set a `readonly` or `archived` site back to `active` while the pool's active sites and holds already reach the limit
- **THEN** the reactivation SHALL be refused
- **AND** the site SHALL remain in its non-active state
#### Scenario: Provider refusal leaves local state unchanged
- **WHEN** the FarmManager PATCH is refused for a status change
- **THEN** the local `status` SHALL be unchanged
- **AND** the pool's figure SHALL equal its active rows and holds once the failure is recorded
#### Scenario: A reactivation that the farm applied before its call timed out
- **WHEN** the FarmManager applies a reactivation, the activity's last attempt times out, and the farm's entry then shows the site `active`
- **THEN** the system SHALL record the site `active` and the request `succeeded`, remove the hold, and the site SHALL count once
#### Scenario: A reactivation that the farm did not apply
- **WHEN** a reactivation's attempts are exhausted and the farm's entry shows the site in its earlier status
- **THEN** the request SHALL end `failed` with `retries_exhausted`, the hold SHALL be removed, and the pool's figure SHALL equal its active rows and holds
#### Scenario: A reactivation whose outcome the farm cannot confirm
- **WHEN** a reactivation's attempts are exhausted and the farm cannot answer the read of the site's entry
- **THEN** the request SHALL stay `accepted` with its hold, shown as Pending, until the farm answers
#### Scenario: A confirmed reactivation whose site was deleted
- **WHEN** a reactivation's attempts are exhausted, the farm's entry shows the site `active`, and its original row is gone
- **THEN** the request SHALL end `declined` with `site_missing`, its hold SHALL be removed and no row SHALL be written
@@ -0,0 +1,95 @@
## MODIFIED Requirements
### Requirement: A repeated lifecycle request is answered from its record
A request's content SHALL be its operation, workspace, requesting person and parameters. Recording a request SHALL serialize per instance the check of the instance's latest generation and the insert, and SHALL answer from what holds the address:
- nothing, and the generation is above the instance's latest: **accepted**, subject to the operation's synchronous checks; the request is recorded and its work starts;
- a request with the same content that is still open: **accepted**, the answer the first submission got; nothing starts;
- a request with the same content that has finished: its stored outcome, `succeeded`, `declined` or `failed`, with its reason;
- a request with different content, or nothing at an address whose generation is at or below the instance's latest: **conflict**; nothing is recorded and nothing starts.
`declined` SHALL mean that the instance's rules refused the request before anything changed in the application, such as a limit reached or a name held by someone else. The same request will be refused again until something changes. `declined` SHALL also cover an original instance that disappeared before its outcome could be recorded, even if the application had already answered a call. `failed` SHALL mean the console tried, could not finish and compensated, or the work never started; the same request can succeed on retry. A reason SHALL be a key the console maps to its own wording, never text from the application.
At most one request per instance SHALL be open at a time: a second, different request while one is open SHALL be refused and not recorded. A request recorded in another workspace SHALL never be disclosed to the caller: it SHALL be answered as not found or as a conflict, without its content. Refusals that need no record, such as form validation or a missing session, SHALL stay synchronous and SHALL record nothing.
#### Scenario: An identical repeat while the work runs is accepted again
- **WHEN** a request is submitted twice with the same instance, generation and content while the first is still open
- **THEN** both submissions SHALL be answered accepted
- **AND** exactly one request SHALL be recorded and exactly one workflow started
#### Scenario: An identical repeat after the work finished returns the stored outcome
- **WHEN** a request that finished `declined` with a reason is submitted again with the same instance, generation and content
- **THEN** the answer SHALL be `declined` with the same reason
- **AND** nothing SHALL be recorded and no workflow SHALL start
#### Scenario: The same address with other content is a conflict
- **WHEN** a request is submitted with the instance and generation of a recorded request but different parameters
- **THEN** the answer SHALL be a conflict
- **AND** nothing SHALL be recorded and no workflow SHALL start
#### Scenario: An older tab's lower generation is a conflict
- **WHEN** an older tab submits a request at generation G1 for an instance after a newer tab's request at a generation G2 above G1 was recorded and finished
- **THEN** the answer SHALL be a conflict
- **AND** nothing SHALL be recorded
#### Scenario: One open request per instance
- **WHEN** a request for an instance is open and a different request for the same instance is submitted at a higher generation
- **THEN** the second request SHALL be refused and not recorded
#### Scenario: Another workspace's request is not disclosed
- **WHEN** a caller submits an instance and generation recorded for another workspace
- **THEN** the answer SHALL be not found or a conflict
- **AND** it SHALL carry none of the recorded request's content or outcome
#### Scenario: A disappeared instance is declined after the application answered
- **WHEN** a provider discovers at its record step that the original instance no longer exists
- **THEN** it SHALL decline the request without retrying that missing-instance step or applying the result to a replacement identified only by the old name
### Requirement: Lifecycle handlers answer when the workflow starts
A handler that accepts a lifecycle request SHALL record the request and commit, then start the request's workflow under the workflow identifier stored on the request, and SHALL answer accepted without waiting for the workflow's result. A start that succeeds, or that reports the workflow already started, SHALL be answered accepted. On any other start error the handler SHALL describe the workflow: only when the workflow engine reports that no such workflow exists SHALL the handler delete the request it inserted and answer with a synchronous error; on any other answer it SHALL keep the request open and answer accepted.
A lifecycle workflow's inputs SHALL carry the request's connection identifier, instance identifier and generation. Every lifecycle workflow SHALL record its outcome, reason and finish time on its request as its last step, and that write SHALL fail when no request matches. A request still open more than two minutes after it was recorded, whose workflow has not recorded an outcome, SHALL be settled at the next boot where the workflow engine is configured: when no workflow exists, the request SHALL end `failed` with the reason `not_started`; when the workflow closed and the request is a create, the console SHALL read the instance back from the application and record the outcome that reading supports; when the workflow closed and the request is any other operation, the request SHALL end `failed` with the reason `retries_exhausted`, except where the provider's specified recovery records the observed result or declines because the original instance is gone; when the workflow is running or the engine does not answer, the request SHALL stay open. A request whose workflow never started SHALL never be restarted.
#### Scenario: The handler answers before the work finishes
- **WHEN** a lifecycle request is accepted and its workflow takes longer than the server's write timeout
- **THEN** the handler SHALL answer accepted as soon as the workflow has started
- **AND** the request SHALL stay open until the workflow records its outcome
#### Scenario: A start that never happened removes the request
- **WHEN** starting the workflow fails and describing it reports that no such workflow exists
- **THEN** the handler SHALL delete the request it inserted and answer with a synchronous error
- **AND** a retry by the person SHALL be recorded as a new request
#### Scenario: A requester who disconnects still gets the workflow
- **WHEN** the requester's connection closes after the request committed and while its workflow is being started
- **THEN** the start and any describe that follows SHALL still run
- **AND** the request SHALL NOT stay open with no workflow started
#### Scenario: An ambiguous start error keeps the request
- **WHEN** starting the workflow returns a deadline error and describing it reports the workflow running
- **THEN** the handler SHALL keep the request open and answer accepted
#### Scenario: An outcome with no request fails the workflow step
- **WHEN** a lifecycle workflow records its outcome and no request matches its instance and generation
- **THEN** the step SHALL fail rather than update no row
#### Scenario: A request that never started is settled at boot
- **WHEN** the console boots with Temporal configured and finds a request open for more than two minutes whose workflow does not exist
- **THEN** the request SHALL end `failed` with the reason `not_started`
- **AND** no workflow SHALL be started for it
@@ -0,0 +1,53 @@
Tests follow `docs/testing.md`: use `dbtest` and the existing FedWiki fixtures, unique names, fixture-filtered reads and named table runners whose expected values are row fields. Database tests pass twice without a reset. Lock tests use `dbtest.AwaitLockWait`, never sleeps. Test helpers have doc comments. Learn scripts from their source before running them.
Sections 1–4 are one implementation; section 5 is landing, run by the caller. The caller runs the full suite, breaks, captures and commits. The commit must pass `make test` with the test stack up, which runs `make lint` first. Preserve #283's committed behavior and change only the overlaps named in the design. Existing goldens and replay histories are expected to remain unchanged; missing-site assertions use explicit expected strings, not new goldens. A changed unrelated result is a stop, not a golden update.
**Breaks.** Write section 4's list to **a file outside the repository**, at the path the caller supplies in phase 3. Use `scripts/mutation-check` test records (`file`, `old`, `new`, `pkg`, `run_new`, `want_new`); new tests have only a new side. The caller runs `go run ./scripts/mutation-check -list <file> -side new` with `test/.env` loaded after a green control. Every `want_new` below is an assertion's failure-only text. Every Go break must compile: apply it, `go vet` its package, restore it. Query breaks edit the generated `.sql.go` constant. Correct a record to the implemented text without dropping its intended violation. Every break must report CAUGHT, or its uncovered fact and reason must be named in the report and commit message. Do not write results or implementation logs into this file.
## 1. Preserve site identity through request execution
- [x] 1.1 Extend `LockSiteStatusByID` with the columns in design decision 1; add ID variants of status, set-force-reduced and clear-force-reduced writes. Preserve trigger behavior, #283's ordering queries and maintenance's still-used domain queries. Remove only queries left unused. Regenerate with `make sqlc-generate`; inspect the diff for unrelated output.
- [x] 1.2 Change `RecordSiteStatusActivity` and `RecordSwapActivity` to lock/check original IDs and workspace/domain before any write. Return top-level non-retryable `SiteMissing` only for `sql.ErrNoRows`; mismatches use `SiteAuthorizationError`, other database errors stay retryable. Keep the committed-outcome shortcut and atomic swap guard/recount.
- [x] 1.3 Handle record refusals in `SetFedWikiSiteStatusRequestWorkflow`, `ReconcileFedWikiSiteStatusWorkflow` and `SwapFedWikiActiveSiteRequestWorkflow`: status missing ends declined/site_missing; swap missing calls the existing `restoreSwapFarm(ctx, sites)` and then finishes both declined/site_missing; mismatches end declined/site_changed, after restoration for a swap. Keep restoration's activity and retry policy unchanged, including its final-refusal and cancellation behavior.
- [x] 1.4 Change missing status/swap farm authorization to `SiteMissing`. Add explicit declined/site_missing handling in `statusChangeFailed`, the existing pre-slot-gate `compensateStatusChange`, the swap's farm-error branch, and reconcile's `!req.SiteFound` branch. A missing swap from the farm branch restores before finishing, including when an earlier attempt changed the surviving site. Keep the delete branch on `SiteAuthorizationError`, with missing-row success unchanged. Keep `ReadFarmSiteInput` and the immediate send-back inside `SwapFarmSitesActivity` unchanged; add no per-attempt read-back check.
- [x] 1.5 Make `usage.hold` distinguish a create from an activation when locking a missing row. An activation returns `usage.ErrSiteNotFound`, including with an existing hold; the activity maps it to `SiteMissing`, and `gateActivation` declines/site_missing. Update callers of the regenerated lock result. Preserve create, already-active and existing-live-hold behavior.
## 2. Check new requests before intake commits
- [x] 2.1 Add the FedWiki store status-intake helpers in `internal/integrations/fedwiki/store/request_intake.go`: database-handle and transaction-batch forms at READ COMMITTED. Take the existing instance locks, resolve recorded addresses through core first, then read originals with `GetSiteByID` and check workspace/domain before recording. Add no site-row lock. Missing-site refusals identify which target is gone; protected-read failures remain errors. Do not duplicate core's generation/content rules.
- [x] 2.2 Route `changeStatus` and `swapActiveSite` through those helpers; use READ COMMITTED in `withWorkspaceTx`. Preserve repeat lookup, #283's park-target ranking and busy/conflict responses. A swap takes activation/park instance locks, computes the park generation under its lock and validates both originals before recording; any refusal rolls back both records and cooldown. Missing/foreign primary targets keep HTTP 200 `Site not found` with no refresh; a missing selected park target uses `siteActionConflict` with its refresh. Scope failed-read handling to the new protected read. Leave permanent-delete intake, API responses and the existing first domain-read behavior unchanged.
- [x] 2.3 Add one workflow reason, `site_missing`, for status and both swap requests. Render missing notices directly using the Copy table: missing original or activation ID uses the short notice; only a missing parked ID uses its domain from the peer request with the same workflow ID in the loaded request rows. Do not resolve the peer by domain or query a replacement. Keep one swap notice and dismiss both requests. Add no unused standalone reason sentence or `TestReasonSentences` row; test the rendered wording in 3.8.
- [x] 2.4 Update `docs/database-locks.md` for READ COMMITTED intake reads under existing instance locks and ID-based outcome locks; retain “takes no site-row lock” for intake. Update `internal/integrations/fedwiki/workflows/README.md` for missing-site outcomes and unchanged restoration retries. Preserve #283's boot-settle requirement wording. Keep deployment recovery instructions for the landing commit note only.
## 3. Tests
- [x] 3.1 Add `TestStatusRecordUsesOriginalSite_DB` in `workflows/request_identity_db_test.go`. Rows: archive, restore and clear-force-reduced, each with original removed and same-name replacement in the same or another workspace; missing without replacement; live success. Run the real record activity and workflow. Assert non-retryable SiteMissing on the missing record's one attempt, declined/site_missing, unchanged replacement status/mark/archive time/placement flag, no retained hold and exact usage. Failure texts: `%s: record attempted %d times, want 1`; `%s: missing record error = %v, want non-retryable SiteMissing`; `%s: outcome = %s/%s, want declined/site_missing`; `%s: replacement changed`; `%s: finished request retained a hold or wrong usage`. Schedule `env.RegisterDelayedCallback(env.CancelWorkflow, d)` to end a retry regression, then assert attempts and outcomes with `t.Errorf`; an SDK test-timeout panic cannot print the wanted failure text.
- [x] 3.2 Add `TestRecordIdentityChecksAndRetry_DB` beside it. Rows: workspace mismatch, domain mismatch, transient DB lock-read error, committed success followed by deletion before retry, absent lifecycle request. Run workspace/domain mismatch rows through both status and reconcile workflows; assert `%s: record mismatch outcome = %s/%s, want declined/site_changed`. Assert no writes on mismatch, retryability of DB failure (not SiteMissing), no second write on committed success, and LifecycleRequestMissing for absent request. Failure texts: `%s: identity mismatch wrote a site`; `%s: database error was classified as non-retryable`; `%s: committed outcome was applied again`; `%s: missing request was treated as a missing site`. Include a regression row using `finishAllWith`'s existing `guard` to return an error after hold deletion and recount: rollback preserves accepted status and the hold; retry finishes and recounts. Failure text `finish rollback lost the hold or outcome`. Add no production test seam.
- [x] 3.3 Add `TestSwapWithMissingSite_DB` in `workflows/swap_identity_db_test.go`. Rows: park missing, activate missing, both missing, each with and without a same-name replacement; remove after successful farm writes and before record. Assert one non-retryable record attempt, neither local row/mark changed, both requests declined/site_missing, no remaining holds and exact usage. Record the complete farm call list: restoration calls only surviving original IDs' domains with their current local statuses. Failure texts: `%s: swap record retried a missing site`; `%s: swap wrote a surviving or replacement row`; `%s: swap did not decline both requests`; `%s: swap left wrong usage or holds`; `%s: restoration addressed a replacement or used the wrong status`. Add a workspace/domain mismatch after the farm step: neither local write, restoration of only the unchanged original, both declined/site_changed; failure `%s: swap mismatch did not restore and decline site_changed`. Use delayed workflow cancellation and `t.Errorf` for retry-regression assertions, as in 3.1.
- [x] 3.4 Add `TestMissingSwapUsesExistingRestoration_DB` there. Rows: restoration succeeds, cannot answer (timeout or 503), DB read fails, 403 final refusal, 404, survivor disappears, survivor's workspace/domain changes; also initial farm authorization finds a missing site after an earlier attempt changed the survivor. Timeout/503/DB rows cancel through a delayed callback after several attempts and assert both requests still accepted with retries continuing; recovery rows let a later attempt succeed and assert both decline/site_missing. Final refusal, missing/mismatched ID and 404 end without endless retries. Failure texts: `%s: restoration stopped while it could not get an answer`; `%s: restoration did not finish with both requests declined`; `%s: restoration called a missing or mismatched original site`; `%s: restoration attempts =`. Keep existing quota/farm-failure restoration tests green.
- [x] 3.5 Add `TestActivationGateRequiresItsSite_DB` in `usage/request_identity_db_test.go`. Rows: missing activation without/with a hold, missing activation at the limit, live activation with hold, live already-active activation, create with no row. Assert typed missing error only for missing activations and no new hold. Failure text `%s: gate did not distinguish a missing activation from a create`. Add `TestMissingStatusStagesDecline_DB` in workflows: gate, farm authorization through `statusChangeFailed`, farm authorization through existing `compensateStatusChange`, reconcile load, reconcile record, and confirmed reactivation after ambiguous failure with the original row deleted. Assert declined/site_missing, released hold and exact usage; no farm call for a row found missing at authorization/load, and no replacement write. Failure texts `%s: missing status did not decline and release its hold`; `%s: missing status called the farm after discovering absence`. Update only the missing-row expectation of `TestStatusReconcile_DB`. Keep unanswered read-back and its held-slot behavior unchanged.
- [x] 3.6 Add `TestSiteIntakeRequiresOriginalRows_DB` in `web/request_identity_db_test.go`. Rows: archive, restore, rotation with positive/zero cooldown, swap missing activation, swap missing park candidate; include same-name replacements deleted/recreated after the first handler read but before protected recording. Assert primary missing answers 200 `Site not found` without refresh, park missing answers 200 `This site changed since the list loaded.` with refresh; zero new requests/starts/holds and unchanged cooldown. Failure texts `%s: intake accepted a missing site`; `%s: refused intake recorded work or stamped cooldown`; `%s: missing-site response = %d %q`; `%s: missing-site refresh = %v, want %v`. Add live success, identical repeat after deletion and changed-content repeat rows. Failure texts `%s: repeat after deletion changed its answer`; `%s: changed repeat was accepted`. Add handler-driven rotation with positive cooldown and swap rows under a REPEATABLE READ session default, with purge winning its instance lock first; assert `%s: intake did not use READ COMMITTED`. Existing partial/API delete tests retain their answers; add no delete guard.
- [x] 3.7 Add `TestIntakeAndPurgeOrder_DB` in the store intake test file. Use separate connections and `AwaitLockWait`: purge's instance lock wins first, then intake wins first. Purge-first refuses intake after the purge commits; intake-first commits accepted before purge's check, and purge skips. Include a swap, `console_runtime`, and a session default of REPEATABLE READ to prove the act explicitly uses READ COMMITTED. Use a test-only alias of the unexported intake function to pass a DBTX whose protected QueryRowContext returns a failing statement; assert the error is not a refusal, with `%s: protected read hid a database error as not found`. No direct-delete row-lock test. Failure texts `%s: intake crossed purge without refusing`; `%s: purge deleted an accepted request's site`; `%s: intake did not use READ COMMITTED`. Exercise the real retention helper in the workflow package's `TestIntakeAndPurgeOrder_DB`. Add `TestSiteRequestRepeats_DB` at the store boundary for identical and changed-content repeats after deletion. Keep existing purge tests.
- [x] 3.8 Add `TestMissingSiteNotices_DB` in web. Render declined archive/restore with rows gone, swap activation gone, only park gone and both gone, including same-name replacements. Select notices with `htmltest`; assert exact Copy-table wording using the peer request's recorded domain, warning tone, one swap notice and acknowledgement of both records on dismissal. Assert no Pending row for a deleted ID and unchanged replacement row. Failure texts `%s: missing-site notice = %q, want %q`; `%s: missing swap showed more than one notice`; `%s: missing swap dismissal left a request visible`; `%s: missing-site notice has the wrong tone`; `%s: deleted site remained Pending or replacement changed`. No standalone reason-sentence test or golden change.
- [x] 3.9 Run focused store, usage, workflow and web tests separately, twice without reset, including existing create/delete/repeat, quota-guard, atomic-outcome, purge, boot-settle and `TestWorkflowHistoriesReplay` tests. Run the race detector for concurrent intake tests. Keep existing histories and version gates; add no compatibility path. Run `./scripts/spdx-headers.sh` on directories receiving new Go files and formatter/static checks appropriate to changed packages.
## 4. Break list
- [x] 4.1 Restore domain locks/writes in `RecordSiteStatusActivity`: `TestStatusRecordUsesOriginalSite_DB`, `want_new` `replacement changed`. Omit SiteMissing classification: same test, `want_new` `want non-retryable SiteMissing`. Suppress its workflow decline branch: same test, `want_new` `want declined/site_missing`.
- [x] 4.2 Omit the workspace/domain record guard: `TestRecordIdentityChecksAndRetry_DB`, `want_new` `identity mismatch wrote a site`. Classify every lock error as missing: same test, `want_new` `database error was classified as non-retryable`. Bypass the committed-outcome shortcut: same test, `want_new` `committed outcome was applied again`. Map a record SiteAuthorizationError to failed/retries_exhausted separately in status and reconcile: same test, `want_new` `want declined/site_changed`. Map missing request to SiteMissing: same test, `want_new` `missing request was treated as a missing site`.
- [x] 4.3 Omit hold deletion, and separately omit recount: `TestStatusRecordUsesOriginalSite_DB`, `want_new` `finished request retained a hold or wrong usage`. In `finishAllWith`, return nil on the guard's error so the transaction commits: `TestRecordIdentityChecksAndRetry_DB`, `want_new` `finish rollback lost the hold or outcome`. This tests existing rollback behavior through its actual guard, with no new seam.
- [x] 4.4 Restore name-based swap record locks/writes: `TestSwapWithMissingSite_DB`, `want_new` `swap wrote a surviving or replacement row`. Suppress SiteMissing classification: same test, `want_new` `swap record retried a missing site`. Finish a swap record mismatch without restoration: same test, `want_new` `swap mismatch did not restore`. Finish only activation: same test, `want_new` `swap did not decline both requests`. Skip restoration: same test, `want_new` `restoration addressed a replacement or used the wrong status` (assert the complete expected call list). Omit `finishAllWith`'s recount (rows where the active parked site is gone): same test, `want_new` `swap left wrong usage or holds`.
- [x] 4.5 For each missing-swap entry, bypass `restoreSwapFarm` or prematurely finish on its retryable failure: `TestMissingSwapUsesExistingRestoration_DB`, `want_new` `restoration stopped while it could not get an answer`. Exercise timeout, 503 and failed DB read rows. Return after successful restoration without finishing: same test, `want_new` `restoration did not finish with both requests declined`. Remove missing/mismatched-ID skipping: same test, `want_new` `restoration called a missing or mismatched original site`. Make 403 or 404 retryable: same test, corresponding row, `want_new` `restoration attempts =`.
- [x] 4.6 Treat a missing activation as a create, and separately check the existing hold before the activation row: `TestActivationGateRequiresItsSite_DB`, `want_new` `gate did not distinguish a missing activation from a create`. Remove SiteMissing handling separately from `gateActivation`, `statusChangeFailed`, `compensateStatusChange`, reconcile load and reconcile record: `TestMissingStatusStagesDecline_DB`, corresponding row, `want_new` `missing status did not decline and release its hold`. Continue to the farm after a missing authorization/load result: same test, `want_new` `missing status called the farm after discovering absence`.
- [x] 4.7 Bypass protected existence validation at each status/rotation/swap call site: `TestSiteIntakeRequiresOriginalRows_DB`, corresponding row, `want_new` `intake accepted a missing site`; also use that break against the cooldown row with `want_new` `refused intake recorded work or stamped cooldown`. Check existence before answering repeats: same test, `want_new` `repeat after deletion changed its answer`. Convert protected-read failure to missing: `TestIntakeAndPurgeOrder_DB`, `want_new` `protected read hid a database error as not found`. Accept changed content: same test, `want_new` `changed repeat was accepted`. Send the primary-target response for a missing park candidate: same test, `want_new` `missing-site response =`. Add a refresh on primary absence or omit the park-conflict refresh: same test, `want_new` `missing-site refresh =`.
- [x] 4.8 Omit the intake instance lock before its ID read (core still locks when recording): `TestIntakeAndPurgeOrder_DB`, `want_new` `intake crossed purge without refusing`. Skip retention's open-request check: same test, `want_new` `purge deleted an accepted request's site`. Replace `db.ReadCommitted` with `db.Plain` in each new/changed intake transaction opener: same test, `want_new` `intake did not use READ COMMITTED`. Point the `withWorkspaceTx` break at `TestSiteIntakeRequiresOriginalRows_DB`; the single-request opener stays with the store test. Ordering failures must print those assertions rather than only a generic wait timeout.
- [x] 4.9 Route missing reasons through action-derived notices, substitute the activation domain for the peer's, and treat a same-name replacement as the original: `TestMissingSiteNotices_DB`, corresponding rows, `want_new` `missing-site notice =`. Remove peer suppression: same test, `want_new` `missing swap showed more than one notice`. Dismiss only activation: same test, `want_new` `missing swap dismissal left a request visible` (inspect both acknowledgement records). Render danger tone: same test, `want_new` `missing-site notice has the wrong tone`. Project the missing outcome onto the same-name replacement's row: same test, `want_new` `deleted site remained Pending or replacement changed`.
- [x] 4.10 Break database-error retryability with a different non-retryable type, omit the missing-original restoration skip, log missing rotation and swap refusals, and drop the status record's database-error context. Target `TestRecordIdentityChecksAndRetry_DB`, `TestMissingSwapUsesExistingRestoration_DB/survivor_deleted` and the corresponding `TestSiteIntakeRequiresOriginalRows_DB` rows; expect `database error was classified as non-retryable`, `restoration called a missing or mismatched original site`, `missing-site refusal logged an error` and `record database error lost its domain context`, respectively. Keep every external break compile-clean.
## 5. Landing (caller)
- [x] 5.1 Approve every Copy row before strings land. Run `openspec validate bind-fedwiki-requests-to-site-ids --strict` and `make test` with the stack up. Confirm `make lint` reports `notebook-citations: ok`.
- [x] 5.2 Run the external break list and put every result and explicitly uncovered fact in the commit message. Inspect changes against #283's committed base.
- [x] 5.3 Capture changed member FedWiki Sites notices by hand at desktop and mobile width: missing original/activation and only missing park target. Check warning tone, the parked domain and Dismiss behavior. No unrelated page captures.
- [x] 5.4 Put recovery instructions in the landing commit note only: read-only count of affected accepted status/swap requests, terminate stuck record-step executions, deploy, let boot settle and its two-minute follow-up finish them, then verify none remains accepted. Held activations reach status reconcile and decline/site_missing; other closed requests, including swaps, finish failed/retries_exhausted through the existing tested path. Inspect replacement rows already changed and surviving swap farm statuses by hand. No upgrade note in `docs/`, manual SQL finish or new history compatibility code.
- [ ] 5.5 Commit separately from #283. Keep the one maintenance bug statement in the design's open questions for a later issue; do not file or implement it here. This change's delta was written from #283's archived wording. Archive only on the caller's instruction. State in the landing commit that `swapActiveSite` left #174's complexity exemption list, and comment on #174 after landing.