diff --git a/docs/database-locks.md b/docs/database-locks.md index a08cd6c4..49a6f6b9 100644 --- a/docs/database-locks.md +++ b/docs/database-locks.md @@ -33,7 +33,7 @@ product row `FOR SHARE` (see the product row below). | Invoice | `hashtext()::bigint` | exclusive | `invoiceReconcile.converge` in `internal/integrations/stripe/workflows` | Two reconciles of one invoice (two events, or an event and the sweep), so the second finds what the first wrote. | | Product sync | `hashtext()::bigint` | exclusive | `catalogsync.Sync` in `internal/integrations/stripe/catalogsync` | Two sync clicks for one product, so the mapping writes and the outbox enqueue happen once. | | FedWiki workspace | `hashtext()::bigint` | exclusive | `withWorkspaceTx` in `internal/integrations/fedwiki/web`, for the status-change and swap handlers | One workspace's rotations of its active site, so two tabs cannot interleave the cooldown check and stamp or the choice of the site to park. | -| Lifecycle instance | `hashtextextended('lifecycle:' \|\| , 0)` | exclusive | the `LockLifecycleInstance` query, run by `integration.RecordLifecycleRequest`, `RecordFirstLifecycleRequest` and the FedWiki activity helper `finishAllWith` | One instance's requests: the latest-generation check and the insert are one step, and an outcome is recorded against a request that is still accepted. | +| Lifecycle instance | `hashtextextended('lifecycle:' \|\| , 0)` | exclusive | the `LockLifecycleInstance` query, run by `integration.RecordLifecycleRequest`, `RecordFirstLifecycleRequest` and the FedWiki activity helpers `finishAllWith` and `purgeArchivedSite` | One instance's requests: the latest-generation check and the insert are one step, and an outcome is recorded against a request that is still accepted; and retention's check that no request is open, held through its farm delete and local deletion. | | Desired-state address | `hashtextextended('desired-state:' \|\| \|\| ':' \|\| \|\| ':' \|\| , 0)` | exclusive | the `LockDesiredStateAddress` query, run by `lockAddress` in `internal/workflows/desiredstate` | Every write to the content or generation of the record at one (connection, recipient) address, and to its projection, including an address that has no record yet. | | Domains registry | the constant `registryAllocationLock` (the ASCII bytes of "domains") | exclusive | `domains.Registry.WithLock` | Every mutating allocation in the registry, one lock for the whole namespace, because the disjointness rules span rows no database constraint can compare. | @@ -49,7 +49,7 @@ product row `FOR SHARE` (see the product row below). | Org type row | `core.org_types` row of one type | `FOR NO KEY UPDATE` | the `LockOrgType` query, run by the default change's T0 | The T0 against the demonstration seed and an edit of the type's other columns, which write the row without the rendezvous. Every other writer of the default already waits on the exclusive rendezvous. `FOR NO KEY UPDATE` does not conflict with the `FOR KEY SHARE` an organization insert takes on its type's row. | | Grant lineage head | the `core.grants` row no other grant extends | `FOR UPDATE OF g` | the `GetGrantLineageHeadForUpdate` query, run by `RevokeGrantTx`, `ExtendWaitingGrantTx` and `ManageGrantTx` | Acts on one lineage, so an extension that committed while another act waited is visible to it. | | Usage row | `core.numeric_entitlement_usage` row of one (pool, resource key) | `FOR UPDATE` | the `LockNumericUsage` query, run by `usage.Recount` and by the slot gate, after the site rows the transaction writes and before the request rows it finishes (the hold rows have no place in the order, see rule 5); the count is a statement of its own after the lock | Two recounts of one pool, so each count reads every change committed before its lock was granted; two slot gates for one pool, so the second counts the first's hold and two creates cannot both take the last slot. | -| FedWiki site row | `fedwiki.sites` row of one domain or one site id | `FOR UPDATE` | the `LockSiteStatusByDomain` query, run by `sweepLocally`, `RecordSiteStatusActivity`, `RecordSwapActivity` and the sync's status write (`writeStatus`), and the `LockSiteStatusByID` query, run by the slot gate for an activation | Paths that move one site (a member's request, the downgrade sweep and the farm sync), so each writes the status it finds under the lock and the recount that follows counts the row as it ended; the sync's reading of the site's requests, which therefore reflects every request that committed before the lock was granted; and the gate's reading of an activation whose row the sweep already made active. | +| FedWiki site row | `fedwiki.sites` row of one domain or one site id | `FOR UPDATE` | the `LockSiteStatusByDomain` query, run by `sweepLocally`, `RecordSiteStatusActivity`, `RecordSwapActivity` and the sync's status write (`writeStatus`), and the `LockSiteStatusByID` query, run by the slot gate for an activation, and `LockArchivedSiteForPurge`, run by retention | Paths that move one site (a member's request, the downgrade sweep and the farm sync), so each writes the status it finds under the lock and the recount that follows counts the row as it ended; the sync's reading of the site's requests, which therefore reflects every request that committed before the lock was granted; the gate's reading of an activation whose row the sweep already made active; and retention's eligibility check through deletion. | | Desired-state record | `core.desired_state_records` row | `FOR UPDATE` | the `LockDesiredStateRecord` query (after the address lock); `StampDesiredStateRecordsDispatched`, which locks the records in identifier order | The record's content and generation within one recompute; the dispatch stamp against another stamp. | | Desired-state notice | `core.desired_state_outbox` rows | `FOR UPDATE SKIP LOCKED` | the `ClaimDesiredStateOutbox` query | Two relay passes, so each claims different rows; a pass skips rows another still holds. Triggers write the rows through `core.insert_desired_state_notice` inside the transaction that changed the source, and they take no lock beyond the insert. | | Resource-key row | `core.resource_keys` row of one key | the update lock of `SetResourceKeyProvider` | `integration.RegisterProviders` at boot, stamping keys in byte order | A stamp that changes a key's provider against an open entitlement insert of that key, which holds the row's `FOR KEY SHARE` through the foreign key. The materializer writes a pool's keys in byte order too, so the two wait in one direction. | @@ -452,6 +452,22 @@ one. The slot gate inserts its hold after the usage row's lock and count. package): one `Registry.WithLock` transaction per orphaned placement or adopted site. +### In the retention purge + +`purgeArchivedSite` opens one READ COMMITTED transaction per candidate, takes +its lifecycle instance lock, then locks its still-expired archived row by ID +with `LockArchivedSiteForPurge`. It checks for an accepted request after both +locks. READ COMMITTED sees requests committed before the instance lock was +granted and skips a row whose eligibility changed during the row-lock wait. +The farm DELETE and local row deletion run under both locks, through commit. +No usage lock or recount is needed: the row is archived and has no open request. +Placement release uses a separate registry transaction after commit. + +The farm call can hold these locks for 30 seconds. Waiting member requests +can exceed the server's eight-second WriteTimeout while their transactions +continue (#290). A rollback cannot undo a remote delete, and cancellation +does not prove the farm stopped executing it. + ### In the farm sync `SyncSitesToDBActivity` reads its sets with no lock held (`ListAllSites`, diff --git a/internal/integrations/fedwiki/store/querier.go b/internal/integrations/fedwiki/store/querier.go index e119eb92..c7175b7b 100644 --- a/internal/integrations/fedwiki/store/querier.go +++ b/internal/integrations/fedwiki/store/querier.go @@ -132,6 +132,8 @@ type Querier interface { ListSitesWithoutPlacement(ctx context.Context) ([]Site, error) // Workspaces that own at least one site — the units the quota reconcile sweeps. ListWorkspaceIDsWithSites(ctx context.Context) ([]string, error) + // Retention purge: the candidate's row, locked, while it is still archived before the cutoff. + LockArchivedSiteForPurge(ctx context.Context, arg LockArchivedSiteForPurgeParams) (Site, error) // The site's status, its row locked until the transaction ends. A path that // writes a status change reads the status here first, so of two paths that // move one site (a member's request and the downgrade sweep) each writes the diff --git a/internal/integrations/fedwiki/store/queries/sites.sql b/internal/integrations/fedwiki/store/queries/sites.sql index 5f18514e..49181de5 100644 --- a/internal/integrations/fedwiki/store/queries/sites.sql +++ b/internal/integrations/fedwiki/store/queries/sites.sql @@ -347,6 +347,15 @@ SELECT * FROM fedwiki.sites WHERE status = 'archived' AND archived_at IS NOT NULL AND archived_at < $1 ORDER BY archived_at ASC; +-- name: LockArchivedSiteForPurge :one +-- Retention purge: the candidate's row, locked, while it is still archived before the cutoff. +SELECT * FROM fedwiki.sites +WHERE site_id = sqlc.arg(site_id)::uuid + AND status = 'archived' + AND archived_at IS NOT NULL + AND archived_at < sqlc.arg(cutoff)::timestamptz +FOR UPDATE; + -- name: GetSiteSwapPolicy :one SELECT last_swap_at FROM fedwiki.site_swap_policy WHERE workspace_id = $1; diff --git a/internal/integrations/fedwiki/store/sites.sql.go b/internal/integrations/fedwiki/store/sites.sql.go index ee2a5350..466f8dbf 100644 --- a/internal/integrations/fedwiki/store/sites.sql.go +++ b/internal/integrations/fedwiki/store/sites.sql.go @@ -936,6 +936,39 @@ func (q *Queries) ListWorkspaceIDsWithSites(ctx context.Context) ([]string, erro return items, nil } +const lockArchivedSiteForPurge = `-- name: LockArchivedSiteForPurge :one +SELECT site_id, workspace_id, domain, created_at, updated_at, status, storage_bytes, last_modified_at, force_reduced_at, archived_at FROM fedwiki.sites +WHERE site_id = $1::uuid + AND status = 'archived' + AND archived_at IS NOT NULL + AND archived_at < $2::timestamptz +FOR UPDATE +` + +type LockArchivedSiteForPurgeParams struct { + SiteID string `json:"site_id"` + Cutoff time.Time `json:"cutoff"` +} + +// Retention purge: the candidate's row, locked, while it is still archived before the cutoff. +func (q *Queries) LockArchivedSiteForPurge(ctx context.Context, arg LockArchivedSiteForPurgeParams) (Site, error) { + row := q.db.QueryRowContext(ctx, lockArchivedSiteForPurge, arg.SiteID, arg.Cutoff) + var i Site + err := row.Scan( + &i.SiteID, + &i.WorkspaceID, + &i.Domain, + &i.CreatedAt, + &i.UpdatedAt, + &i.Status, + &i.StorageBytes, + &i.LastModifiedAt, + &i.ForceReducedAt, + &i.ArchivedAt, + ) + return i, err +} + const lockSiteStatusByDomain = `-- name: LockSiteStatusByDomain :one SELECT status, force_reduced_at FROM fedwiki.sites WHERE domain = $1 diff --git a/internal/integrations/fedwiki/workflows/purge_failures_db_test.go b/internal/integrations/fedwiki/workflows/purge_failures_db_test.go new file mode 100644 index 00000000..68dc2e07 --- /dev/null +++ b/internal/integrations/fedwiki/workflows/purge_failures_db_test.go @@ -0,0 +1,226 @@ +// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial +// SPDX-FileCopyrightText: 2025-2026 Christian Galo + +package workflows + +import ( + "bytes" + "context" + "errors" + "io" + "log/slog" + "net/http" + "strings" + "testing" + "time" + + "git.coopcloud.tech/wiki-cafe/member-console/internal/integration" + fwmod "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/store" + "git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest" +) + +// purgeFailureCase supplies the failure and its expected effects, independently of its name. +type purgeFailureCase struct { + name string + run func(*reqEnv, fwmod.Site, purgeFailureCase) + farmStatus int + absent bool + listError bool + later bool + wantError bool + wantCalls int + wantPurged int + wantRow bool + wantPlacement bool + wantLogs []string +} + +// TestRetentionPurgeFailures_DB proves failures preserve local state and do not +// stop later candidates, while a farm 404 permits cleanup. +func TestRetentionPurgeFailures_DB(t *testing.T) { + cases := []purgeFailureCase{ + {name: "list error", run: retentionActivityFailure, listError: true, wantError: true, wantRow: true, wantPlacement: true}, + {name: "begin error", run: retentionBeginFailure, wantError: true, wantRow: true, wantPlacement: true}, + {name: "cancelled lock wait", run: retentionCancelledWait, wantError: true, wantRow: true, wantPlacement: true}, + {name: "farm refusal", run: retentionActivityFailure, farmStatus: http.StatusInternalServerError, wantCalls: 1, wantRow: true, wantPlacement: true, wantLogs: []string{"retention purge: hard delete failed"}}, + {name: "farm already absent", run: retentionActivityFailure, absent: true, farmStatus: http.StatusNotFound, wantCalls: 1, wantPurged: 1}, + {name: "local failure after farm success", run: retentionLocalFailure, wantError: true, wantCalls: 1, wantRow: true, wantPlacement: true}, + {name: "later candidate after refusal", run: retentionActivityFailure, farmStatus: http.StatusInternalServerError, later: true, wantCalls: 2, wantPurged: 1, wantRow: true, wantPlacement: true, wantLogs: []string{"retention purge: hard delete failed", "retention purge: local delete failed"}}, + } + for _, row := range cases { + t.Run(row.name, func(t *testing.T) { retentionFailure(t, row) }) + } +} + +// retentionFailure creates the fixture and applies one table row's failure. +func retentionFailure(t *testing.T, row purgeFailureCase) { + env := newReqEnv(t, 5, 0) + site := expiredPurgeSite(env, "failure") + if row.absent { + if _, err := env.acts.Client.HardDeleteSite(env.ctx, site.Domain); err != nil { + t.Fatal(err) + } + } + row.run(env, site, row) +} + +// retentionBeginFailure proves a closed database cannot authorize a farm call. +func retentionBeginFailure(env *reqEnv, site fwmod.Site, row purgeFailureCase) { + env.t.Helper() + farm := observePurgeFarm(env, nil) + database := dbtest.Closed(env.t) + deleted, err := env.acts.purgeArchivedSite(env.ctx, database, site, time.Now().Add(-ArchiveRetention)) + checkPurgeHelperFailure(env, site, row, deleted, err, farm) +} + +// retentionCancelledWait cancels only after observing the instance-lock wait. +func retentionCancelledWait(env *reqEnv, site fwmod.Site, row purgeFailureCase) { + env.t.Helper() + farm := observePurgeFarm(env, nil) + side := env.beginSide(env.lockWaitContext()) + if err := integration.New(side.tx).LockLifecycleInstance(side.ctx, site.SiteID); err != nil { + env.t.Fatal(err) + } + connection, pid := purgeConnection(env.t, env.db) + ctx, cancel := context.WithTimeout(env.ctx, 30*time.Second) + defer cancel() + var deleted bool + done := inGoroutine(func() error { + var err error + deleted, err = env.acts.purgeArchivedSite(ctx, connection, site, time.Now().Add(-ArchiveRetention)) + return err + }) + defer func() { + cancel() + if done != nil { + <-done + } + }() + dbtest.AwaitLockWait(env.t, env.db, dbtest.Backend{PID: pid}) + cancel() + err := <-done + done = nil + checkPurgeHelperFailure(env, site, row, deleted, err, farm) +} + +// purgeCancelBody cancels after a successful body is read, so the local DELETE +// fails after the farm deleted its site. +type purgeCancelBody struct { + io.ReadCloser + cancel context.CancelFunc +} + +// Read preserves the response bytes and EOF while cancelling the helper's context. +func (b *purgeCancelBody) Read(buffer []byte) (int, error) { + count, err := b.ReadCloser.Read(buffer) + if errors.Is(err, io.EOF) { + b.cancel() + } + return count, err +} + +// purgeCancelTransport wraps a real HTTP response to fail only the following local step. +type purgeCancelTransport struct{ cancel context.CancelFunc } + +// RoundTrip leaves HTTP delivery intact and installs cancellation at body EOF. +func (c purgeCancelTransport) RoundTrip(request *http.Request) (*http.Response, error) { + response, err := http.DefaultTransport.RoundTrip(request) + if err == nil { + response.Body = &purgeCancelBody{ReadCloser: response.Body, cancel: c.cancel} + } + return response, err +} + +// retentionLocalFailure proves remote success does not imply a committed local deletion. +func retentionLocalFailure(env *reqEnv, site fwmod.Site, row purgeFailureCase) { + env.t.Helper() + farm := observePurgeFarm(env, nil) + ctx, cancel := context.WithCancel(env.ctx) + defer cancel() + env.acts.Client.httpClient.Transport = purgeCancelTransport{cancel: cancel} + deleted, err := env.acts.purgeArchivedSite(ctx, env.db, site, time.Now().Add(-ArchiveRetention)) + checkPurgeHelperFailure(env, site, row, deleted, err, farm) + if errors.Is(err, errRetentionFarmDelete) { + env.t.Errorf("%s: purge error = %v, want local failure", row.name, err) + } + client := NewFarmManagerClient(env.farm.URL(), "test") + _, err = client.GetSite(env.ctx, site.Domain) + var apiError *FarmManagerAPIError + if !errors.As(err, &apiError) || apiError.StatusCode != http.StatusNotFound { + env.t.Errorf("farm still holds deleted site: %v", err) + } +} + +// checkPurgeHelperFailure proves a failed transaction reports no deletion and keeps its row. +func checkPurgeHelperFailure(env *reqEnv, site fwmod.Site, row purgeFailureCase, deleted bool, err error, farm *purgeFarm) { + env.t.Helper() + if (err != nil) != row.wantError { + env.t.Errorf("%s: purge error = %v, want failure %t", row.name, err, row.wantError) + } + purged := 0 + if deleted { + purged = 1 + } + checkPurgeActivityFailure(env, site, row, &PurgeExpiredArchivedOutput{Purged: purged}, farm) +} + +// retentionActivityFailure exercises real HTTP responses and checks the activity's warning category. +func retentionActivityFailure(env *reqEnv, site fwmod.Site, row purgeFailureCase) { + env.t.Helper() + var logs bytes.Buffer + env.acts.Logger = slog.New(slog.NewTextHandler(&logs, nil)) + farm := observePurgeFarm(env, func(writer http.ResponseWriter, request *http.Request) bool { + if request.Method != http.MethodDelete || !strings.HasSuffix(request.URL.Path, site.Domain) || row.farmStatus == 0 { + return false + } + writer.WriteHeader(row.farmStatus) + return true + }) + listError := errors.New("candidate list failed") + queries := purgeCandidates{Querier: env.acts.SiteQ, ids: []string{site.SiteID}} + if row.listError { + queries.after = func(time.Time, []fwmod.Site) ([]fwmod.Site, error) { return nil, listError } + } + var later fwmod.Site + if row.later { + later = expiredPurgeSite(env, "later") + queries.ids = append(queries.ids, later.SiteID) + queries.after = func(_ time.Time, listed []fwmod.Site) ([]fwmod.Site, error) { + return []fwmod.Site{listed[0], {SiteID: "invalid", Domain: "invalid"}, listed[1]}, nil + } + } + env.acts.SiteQ = queries + output, err := env.acts.PurgeExpiredArchivedActivity(env.ctx, struct{}{}) + if row.wantError { + if !errors.Is(err, listError) { + env.t.Errorf("%s: purge error = %v, want %v", row.name, err, listError) + } + } else if err != nil { + env.t.Fatalf("%s: purge error = %v, want nil", row.name, err) + } + checkPurgeActivityFailure(env, site, row, output, farm) + if row.later { + checkPurgeRows(env, row.name, later.Domain, "", false) + } + for _, message := range row.wantLogs { + if !strings.Contains(logs.String(), message) { + env.t.Errorf("%s: log lacks %q: %s", row.name, message, &logs) + } + } +} + +// checkPurgeActivityFailure compares calls, committed deletions and retained state to the table. +func checkPurgeActivityFailure(env *reqEnv, site fwmod.Site, row purgeFailureCase, output *PurgeExpiredArchivedOutput, farm *purgeFarm) { + env.t.Helper() + if got := len(farm.calls()); got != row.wantCalls { + env.t.Errorf("%s: farm calls = %d, want %d", row.name, got, row.wantCalls) + } + if output != nil && output.Purged != row.wantPurged { + env.t.Errorf("%s: purged = %d, want %d", row.name, output.Purged, row.wantPurged) + } + wantID := "" + if row.wantRow { + wantID = site.SiteID + } + checkPurgeRows(env, row.name, site.Domain, wantID, row.wantPlacement) +} diff --git a/internal/integrations/fedwiki/workflows/purge_locks_db_test.go b/internal/integrations/fedwiki/workflows/purge_locks_db_test.go new file mode 100644 index 00000000..2b84b361 --- /dev/null +++ b/internal/integrations/fedwiki/workflows/purge_locks_db_test.go @@ -0,0 +1,241 @@ +// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial +// SPDX-FileCopyrightText: 2025-2026 Christian Galo + +package workflows + +import ( + "context" + "database/sql" + "encoding/json" + "errors" + "fmt" + "net/http" + "strings" + "sync" + "testing" + "time" + + "git.coopcloud.tech/wiki-cafe/member-console/internal/db" + "git.coopcloud.tech/wiki-cafe/member-console/internal/integration" + fwmod "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/store" + "git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest" +) + +// purgeCall publishes a helper's result when done closes, so lock probes do not consume it. +type purgeCall struct { + done chan struct{} + deleted bool + err error +} + +// startPurge runs a bounded helper call and cancels before joining it at cleanup. +func startPurge(env *reqEnv, database db.Beginner, site fwmod.Site) *purgeCall { + env.t.Helper() + ctx, cancel := context.WithTimeout(env.ctx, 30*time.Second) + call := &purgeCall{done: make(chan struct{})} + go func() { + defer close(call.done) + call.deleted, call.err = env.acts.purgeArchivedSite(ctx, database, site, time.Now().Add(-ArchiveRetention)) + }() + env.t.Cleanup(func() { cancel(); <-call.done }) + return call +} + +// abort reports completion before the expected lock wait as a test failure. +func (c *purgeCall) abort(message string) func() error { + return func() error { + select { + case <-c.done: + return fmt.Errorf("%s: deleted %t, error %v", message, c.deleted, c.err) + default: + return nil + } + } +} + +// purgeConnection pins a backend so a lock probe observes this call, not another test. +func purgeConnection(t *testing.T, database *sql.DB) (*sql.Conn, int) { + t.Helper() + connection, err := database.Conn(t.Context()) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + if err := connection.Close(); err != nil && !errors.Is(err, sql.ErrConnDone) { + t.Errorf("close purge connection: %v", err) + } + }) + pid := dbtest.On(t, connection).Int(`SELECT pg_backend_pid()`) + return connection, int(pid) +} + +// purgeRepeatableRead opens and verifies a handle whose server default is REPEATABLE READ. +func purgeRepeatableRead(t *testing.T) *sql.DB { + t.Helper() + dsn := dbtest.DSN(t) + separator := "?" + if strings.Contains(dsn, "?") { + separator = "&" + } + database := dbtest.Connect(t, dsn+separator+"default_transaction_isolation=repeatable%20read") + if got := dbtest.On(t, database).String(`SHOW default_transaction_isolation`); got != "repeatable read" { + t.Fatalf("default isolation = %q", got) + } + return database +} + +// purgeWaitCase states the isolation or writer and the expected result after waiting. +type purgeWaitCase struct { + name string + repeatable bool + update string + wantDeleted bool + wantError bool + wantCalls int +} + +// TestRetentionPurgeWaitsForLifecycleIntake_DB proves intake committed during +// the instance-lock wait is visible even with a REPEATABLE READ server default. +func TestRetentionPurgeWaitsForLifecycleIntake_DB(t *testing.T) { + cases := []purgeWaitCase{ + {name: "default isolation", wantDeleted: false, wantError: false, wantCalls: 0}, + {name: "server default repeatable read", repeatable: true, wantDeleted: false, wantError: false, wantCalls: 0}, + } + for _, row := range cases { + t.Run(row.name, func(t *testing.T) { retentionWaitsForIntake(t, row) }) + } +} + +// retentionWaitsForIntake commits a real accepted request while the purge waits for it. +func retentionWaitsForIntake(t *testing.T, row purgeWaitCase) { + env := newReqEnv(t, 5, 0) + site := expiredPurgeSite(env, "intake") + farm := observePurgeFarm(env, nil) + side := env.beginSide(env.lockWaitContext()) + answer, err := integration.RecordLifecycleRequest(side.ctx, side.tx, integration.LifecycleRequestSpec{ + InstanceID: site.SiteID, Generation: 1, ConnectionID: env.conn, ResourceKey: "fedwiki_sites", WorkspaceID: env.ws, + Operation: integration.LifecycleSetStatus, Parameters: json.RawMessage(`{"status":"active"}`), WorkflowID: "purge-" + site.SiteID, RequestedByPersonID: env.person, + }) + if err != nil || answer.Kind != integration.LifecycleRecorded { + t.Fatalf("intake = %+v, %v", answer, err) + } + env.requests = append(env.requests, site.SiteID) + database := env.db + if row.repeatable { + database = purgeRepeatableRead(t) + } + connection, pid := purgeConnection(t, database) + call := startPurge(env, connection, site) + dbtest.AwaitLockWait(t, env.db, dbtest.Backend{PID: pid, Abort: call.abort("purge did not wait for lifecycle intake")}) + side.commit() + <-call.done + if (call.err != nil) != row.wantError { + t.Fatalf("purge error = %v, want failure %t", call.err, row.wantError) + } + if call.deleted != row.wantDeleted || len(farm.calls()) != row.wantCalls { + t.Error("purge deleted a site whose restore committed before its lock") + } + checkPurgeRows(env, row.name, site.Domain, site.SiteID, true) +} + +// TestRetentionPurgeWaitsForSiteEligibility_DB proves a row changed during its +// lock wait is skipped normally, not rejected with a serialization error. +func TestRetentionPurgeWaitsForSiteEligibility_DB(t *testing.T) { + cases := []purgeWaitCase{ + {name: "active", update: "UPDATE fedwiki.sites SET status = 'active' WHERE site_id = $1", wantDeleted: false, wantError: false, wantCalls: 0}, + {name: "fresh archive", update: "UPDATE fedwiki.sites SET archived_at = now() WHERE site_id = $1", wantDeleted: false, wantError: false, wantCalls: 0}, + {name: "cleared archive time", update: "UPDATE fedwiki.sites SET archived_at = NULL WHERE site_id = $1", wantDeleted: false, wantError: false, wantCalls: 0}, + } + for _, row := range cases { + t.Run(row.name, func(t *testing.T) { retentionWaitsForSite(t, row) }) + } +} + +// retentionWaitsForSite commits the table's writer while retention waits on the site row. +func retentionWaitsForSite(t *testing.T, row purgeWaitCase) { + env := newReqEnv(t, 5, 0) + site := expiredPurgeSite(env, "eligibility") + farm := observePurgeFarm(env, nil) + side := env.beginSide(env.lockWaitContext()) + side.exec(`SELECT site_id FROM fedwiki.sites WHERE site_id = $1 FOR UPDATE`, site.SiteID) + connection, pid := purgeConnection(t, purgeRepeatableRead(t)) + call := startPurge(env, connection, site) + dbtest.AwaitLockWait(t, env.db, dbtest.Backend{PID: pid, Abort: call.abort("purge did not wait for the site row")}) + side.exec(row.update, site.SiteID) + side.commit() + <-call.done + if call.deleted != row.wantDeleted || (call.err != nil) != row.wantError { + t.Errorf("%s: purge result = deleted %t, error %v; want false and nil", row.name, call.deleted, call.err) + } + if len(farm.calls()) != row.wantCalls { + t.Errorf("%s: purge used eligibility from before the row lock", row.name) + } + checkPurgeRows(env, row.name, site.Domain, site.SiteID, true) +} + +// TestRetentionPurgeHoldsLocksThroughFarmDelete_DB proves both locks survive the +// farm call and prevent a concurrent purge from deleting the same site twice. +func TestRetentionPurgeHoldsLocksThroughFarmDelete_DB(t *testing.T) { + env := newReqEnv(t, 5, 0) + site := expiredPurgeSite(env, "locked") + arrived, release := make(chan struct{}, 2), make(chan struct{}) + var once sync.Once + open := func() { once.Do(func() { close(release) }) } + farm := observePurgeFarm(env, func(_ http.ResponseWriter, request *http.Request) bool { + if request.Method == http.MethodDelete { + arrived <- struct{}{} + <-release + } + return false + }) + defer open() + connection, pid := purgeConnection(t, env.db) + dbtest.On(t, connection).Exec("SET ROLE " + db.RuntimeGroupRole) + t.Cleanup(func() { dbtest.On(t, connection).Exec(`RESET ROLE`) }) + first := startPurge(env, connection, site) + ctx, cancel := context.WithTimeout(env.ctx, 30*time.Second) + defer cancel() + select { + case <-arrived: + case <-ctx.Done(): + t.Fatal(ctx.Err()) + } + writer := inGoroutine(func() error { + _, err := env.db.ExecContext(ctx, `UPDATE fedwiki.sites SET status = 'active' WHERE site_id = $1`, site.SiteID) + return err + }) + defer func() { + open() + if writer != nil { + env.finished(writer, "site writer") + } + }() + dbtest.AwaitLockWait(t, env.db, dbtest.Backend{BlockedBy: pid, Abort: func() error { + select { + case err := <-writer: + writer = nil + return fmt.Errorf("site writer passed the purge's held row: %v", err) + default: + return nil + } + }}) + secondConnection, _ := purgeConnection(t, env.db) + second := startPurge(env, secondConnection, site) + dbtest.AwaitLockWait(t, env.db, dbtest.Backend{BlockedBy: pid, Waiters: 2, Abort: func() error { + select { + case <-arrived: + return fmt.Errorf("second purge passed the first purge's instance lock") + default: + return second.abort("second purge passed the first purge's instance lock")() + } + }}) + open() + <-first.done + <-second.done + if first.err != nil || second.err != nil || !first.deleted || second.deleted { + t.Errorf("purge results: first %t/%v, second %t/%v", first.deleted, first.err, second.deleted, second.err) + } + if got := len(farm.calls()); got != 1 { + t.Errorf("concurrent purge hard deletes = %d, want 1", got) + } +} diff --git a/internal/integrations/fedwiki/workflows/purge_requests_db_test.go b/internal/integrations/fedwiki/workflows/purge_requests_db_test.go new file mode 100644 index 00000000..273e2d7c --- /dev/null +++ b/internal/integrations/fedwiki/workflows/purge_requests_db_test.go @@ -0,0 +1,337 @@ +// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial +// SPDX-FileCopyrightText: 2025-2026 Christian Galo + +package workflows + +import ( + "context" + "database/sql" + "net/http" + "net/http/httptest" + "net/http/httputil" + "net/url" + "slices" + "sync" + "testing" + "time" + + "git.coopcloud.tech/wiki-cafe/member-console/internal/integration" + fwmod "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/store" +) + +// purgeCandidates filters the real list to this fixture; after mutates it between +// the snapshot and its return, as a concurrent writer would. +type purgeCandidates struct { + fwmod.Querier + ids []string + after func(time.Time, []fwmod.Site) ([]fwmod.Site, error) +} + +// ListArchivedSitesArchivedBefore takes the real cutoff and preserves stale candidates. +func (q purgeCandidates) ListArchivedSitesArchivedBefore(ctx context.Context, cutoff sql.NullTime) ([]fwmod.Site, error) { + listed, err := q.Querier.ListArchivedSitesArchivedBefore(ctx, cutoff) + if err != nil { + return nil, err + } + candidates := make([]fwmod.Site, 0, len(q.ids)) + for _, id := range q.ids { + for _, site := range listed { + if site.SiteID == id { + candidates = append(candidates, site) + } + } + } + if q.after != nil { + return q.after(cutoff.Time, candidates) + } + return candidates, nil +} + +// purgeFarm records the hard deletes that actually reach this fixture's HTTP proxy. +type purgeFarm struct { + mu sync.Mutex + deletes []string +} + +// observePurgeFarm records DELETEs and lets a test hold or refuse a farm call. +func observePurgeFarm(env *reqEnv, intercept func(http.ResponseWriter, *http.Request) bool) *purgeFarm { + env.t.Helper() + upstream, err := url.Parse(env.farm.URL()) + if err != nil { + env.t.Fatal(err) + } + proxy := httputil.NewSingleHostReverseProxy(upstream) + farm := &purgeFarm{} + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + if request.Method == http.MethodDelete { + farm.mu.Lock() + farm.deletes = append(farm.deletes, request.Method+" "+request.URL.RequestURI()) + farm.mu.Unlock() + } + if intercept != nil && intercept(writer, request) { + return + } + proxy.ServeHTTP(writer, request) + })) + env.t.Cleanup(server.Close) + env.acts.Client = NewFarmManagerClient(server.URL, "test") + return farm +} + +// calls returns a synchronized snapshot for assertions on the test goroutine. +func (f *purgeFarm) calls() []string { + f.mu.Lock() + defer f.mu.Unlock() + return slices.Clone(f.deletes) +} + +// expiredPurgeSite creates an expired archive with its name and farm content. +// Cleanup refreshes its time so the unfiltered TestRetentionPurge cannot purge +// a surviving fixture on a later run. +func expiredPurgeSite(env *reqEnv, label string) fwmod.Site { + env.t.Helper() + domain := label + "." + env.root + id := env.insertSite(domain, "archived", false) + env.exec(`UPDATE fedwiki.sites SET archived_at = now() - interval '31 days' WHERE site_id = $1`, id) + env.placeServable(domain, id) + env.farm.HoldIn(domain, env.subject, "archived") + env.t.Cleanup(func() { env.exec(`UPDATE fedwiki.sites SET archived_at = now() WHERE domain = $1`, domain) }) + return fwmod.Site{SiteID: id, Domain: domain} +} + +// purgeCandidateCase states the concurrent write and the expected retained state. +type purgeCandidateCase struct { + name string + status string + archiveTime string + remove bool + replace bool + request *purgeRequest + wantRow bool + wantStatus string + wantPlacement bool + wantPurged int + wantUsage int64 +} + +// purgeRequest describes an accepted request, its optional hold and its outcome. +type purgeRequest struct { + operation string + status string + hold bool + finished bool + swap bool +} + +// TestRetentionPurgeRechecksCandidates_DB proves the list does not authorize +// deletion after eligibility or requests change. +func TestRetentionPurgeRechecksCandidates_DB(t *testing.T) { + cases := []purgeCandidateCase{ + {name: "expired", wantPurged: 1}, + {name: "active", status: "active", wantRow: true, wantStatus: "active", wantPlacement: true, wantUsage: 1}, + {name: "readonly", status: "readonly", wantRow: true, wantStatus: "readonly", wantPlacement: true}, + {name: "fresh archive", archiveTime: "fresh", wantRow: true, wantStatus: "archived", wantPlacement: true}, + {name: "null archive time", archiveTime: "null", wantRow: true, wantStatus: "archived", wantPlacement: true}, + {name: "at cutoff", archiveTime: "cutoff", wantRow: true, wantStatus: "archived", wantPlacement: true}, + {name: "missing", remove: true, wantPlacement: true}, + {name: "replacement", remove: true, replace: true, wantRow: true, wantStatus: "archived", wantPlacement: true}, + {name: "open restore before gate", request: &purgeRequest{operation: integration.LifecycleSetStatus, status: "active"}, wantRow: true, wantStatus: "archived", wantPlacement: true}, + {name: "open restore with hold", request: &purgeRequest{operation: integration.LifecycleSetStatus, status: "active", hold: true}, wantRow: true, wantStatus: "archived", wantPlacement: true, wantUsage: 1}, + {name: "open archive", request: &purgeRequest{operation: integration.LifecycleSetStatus, status: "archived"}, wantRow: true, wantStatus: "archived", wantPlacement: true}, + {name: "open delete", request: &purgeRequest{operation: integration.LifecycleDelete}, wantRow: true, wantStatus: "archived", wantPlacement: true}, + {name: "open swap", request: &purgeRequest{swap: true}, wantRow: true, wantStatus: "archived", wantPlacement: true, wantUsage: 1}, + {name: "finished request", request: &purgeRequest{operation: integration.LifecycleSetStatus, status: "archived", finished: true}, wantPurged: 1}, + } + for _, row := range cases { + t.Run(row.name, func(t *testing.T) { retentionCandidate(t, row) }) + } +} + +// retentionCandidate checks farm calls, rows, placements and usage against one row. +func retentionCandidate(t *testing.T, row purgeCandidateCase) { + env := newReqEnv(t, 5, 0) + site := expiredPurgeSite(env, "candidate") + farm := observePurgeFarm(env, nil) + currentID := site.SiteID + env.acts.SiteQ = purgeCandidates{Querier: env.acts.SiteQ, ids: []string{site.SiteID}, after: func(cutoff time.Time, listed []fwmod.Site) ([]fwmod.Site, error) { + currentID = changePurgeCandidate(env, site, row, cutoff) + env.recount() + return listed, nil + }} + output, err := env.acts.PurgeExpiredArchivedActivity(env.ctx, struct{}{}) + if err != nil { + t.Fatal(err) + } + var wantDeletes []string + if row.wantPurged != 0 { + wantDeletes = []string{"DELETE /plugin/farmmanager/sites/" + site.Domain + "?hard=true"} + } + if got := farm.calls(); !slices.Equal(got, wantDeletes) { + t.Errorf("%s: hard deletes = %v, want %v", row.name, got, wantDeletes) + } + wantID := "" + if row.wantRow { + wantID = currentID + } + checkPurgeRows(env, row.name, site.Domain, wantID, row.wantPlacement) + if row.wantRow { + env.wantSiteStatus(site.Domain, row.wantStatus) + } + if output.Purged != row.wantPurged { + t.Errorf("%s: purged = %d, want %d", row.name, output.Purged, row.wantPurged) + } + if figure, rows := env.figureAndRows(); figure != row.wantUsage || rows != row.wantUsage { + t.Errorf("%s: usage = %d, counted rows = %d, want %d", row.name, figure, rows, row.wantUsage) + } +} + +// changePurgeCandidate applies a writer after listing and returns the current row's ID. +func changePurgeCandidate(env *reqEnv, site fwmod.Site, row purgeCandidateCase, cutoff time.Time) string { + env.t.Helper() + id := site.SiteID + if row.status != "" { + env.exec(`UPDATE fedwiki.sites SET status = $2 WHERE site_id = $1`, id, row.status) + } + switch row.archiveTime { + case "fresh": + env.exec(`UPDATE fedwiki.sites SET archived_at = now() WHERE site_id = $1`, id) + case "null": + env.exec(`UPDATE fedwiki.sites SET archived_at = NULL WHERE site_id = $1`, id) + case "cutoff": + env.exec(`UPDATE fedwiki.sites SET archived_at = $2 WHERE site_id = $1`, id, cutoff) + } + if row.remove { + env.exec(`DELETE FROM fedwiki.sites WHERE site_id = $1`, id) + id = "" + } + if row.replace { + id = env.insertSite(site.Domain, "archived", false) + env.exec(`UPDATE fedwiki.sites SET archived_at = now() - interval '31 days' WHERE site_id = $1`, id) + } + if row.request != nil { + purgeCandidateRequest(env, site, *row.request) + } + return id +} + +// purgeCandidateRequest records the request whose presence retention must respect. +func purgeCandidateRequest(env *reqEnv, site fwmod.Site, request purgeRequest) { + env.t.Helper() + if request.swap { + park := env.insertSite("incumbent."+env.root, "active", false) + env.swapRequests(site.SiteID, park, "purge-swap-"+site.SiteID) + return + } + generation := env.record(site.SiteID, request.operation, "purge-"+site.SiteID, map[string]any{"domain": site.Domain, "status": request.status}) + if request.hold { + holdPurgeRestore(env, site.SiteID, generation) + } + if request.finished { + if err := integration.FinishLifecycleRequest(env.ctx, env.db, integration.LifecycleOutcome{InstanceID: site.SiteID, Generation: generation, Status: integration.LifecycleSucceeded, FinishedAt: time.Now()}); err != nil { + env.t.Fatal(err) + } + } +} + +// holdPurgeRestore proves the real activation gate allocated the request's slot. +func holdPurgeRestore(env *reqEnv, id string, generation int64) { + env.t.Helper() + output, err := env.acts.HoldSiteSlotActivity(env.ctx, HoldSiteSlotInput{InstanceID: id, Generation: generation}) + if err != nil || !output.Held { + env.t.Fatalf("restore gate = %+v, %v", output, err) + } +} + +// checkPurgeRows compares only this fixture's surviving row and placement. +func checkPurgeRows(env *reqEnv, name, domain, wantID string, wantPlacement bool) { + env.t.Helper() + var ids []string + rows, err := env.db.QueryContext(env.ctx, `SELECT site_id FROM fedwiki.sites WHERE domain = $1 ORDER BY site_id`, domain) + if err != nil { + env.t.Fatal(err) + } + defer rows.Close() + for rows.Next() { + var id string + if err := rows.Scan(&id); err != nil { + env.t.Fatal(err) + } + ids = append(ids, id) + } + if err := rows.Err(); err != nil { + env.t.Fatal(err) + } + var wantIDs []string + if wantID != "" { + wantIDs = []string{wantID} + } + if !slices.Equal(ids, wantIDs) { + env.t.Errorf("%s: retained site IDs = %v, want %v", name, ids, wantIDs) + } + if got := env.placed(domain); got != wantPlacement { + env.t.Errorf("%s: retained placements = %v, want %v", name, got, wantPlacement) + } +} + +// TestRetentionPurgeLeavesARestoreInFlight_DB proves a held restore survives a +// purge during PATCH and records an active site with matching usage afterward. +func TestRetentionPurgeLeavesARestoreInFlight_DB(t *testing.T) { + env := newReqEnv(t, 5, 0) + site := expiredPurgeSite(env, "restoring") + generation := env.record(site.SiteID, integration.LifecycleSetStatus, "purge-"+site.SiteID, map[string]any{"domain": site.Domain, "status": "active"}) + holdPurgeRestore(env, site.SiteID, generation) + arrived, release := make(chan struct{}), make(chan struct{}) + var once sync.Once + open := func() { once.Do(func() { close(release) }) } + farm := observePurgeFarm(env, func(_ http.ResponseWriter, request *http.Request) bool { + if request.Method == http.MethodPatch { + close(arrived) + <-release + } + return false + }) + ctx := env.lockWaitContext() + done := inGoroutine(func() error { + return env.acts.ApplyFarmSiteStatusActivity(ctx, SetSiteStatusInput{SiteID: site.SiteID, WorkspaceID: env.ws, Domain: site.Domain, Status: "active"}) + }) + defer func() { + open() + if done != nil { + env.finished(done, "restore PATCH") + } + }() + select { + case <-arrived: + case <-ctx.Done(): + t.Fatal(ctx.Err()) + } + env.acts.SiteQ = purgeCandidates{Querier: env.acts.SiteQ, ids: []string{site.SiteID}} + output, err := env.acts.PurgeExpiredArchivedActivity(ctx, struct{}{}) + if err != nil { + t.Fatal(err) + } + if len(farm.calls()) != 0 || output.Purged != 0 { + t.Error("purge sent DELETE while the accepted restore was in flight") + } + checkPurgeRows(env, "in flight", site.Domain, site.SiteID, true) + if !env.holds(site.SiteID, generation) { + t.Error("purge removed the restore hold") + } + open() + err = <-done + done = nil + if err != nil { + t.Fatal(err) + } + if err := env.acts.RecordSiteStatusActivity(ctx, RecordSiteStatusInput{InstanceID: site.SiteID, Generation: generation, WorkspaceID: env.ws, Domain: site.Domain, Status: "active", FinishedAt: time.Now()}); err != nil { + t.Fatal(err) + } + _, status, _ := env.siteByID(site.SiteID) + request, _ := env.request(site.SiteID, generation) + if status != "active" || request != "succeeded" { + t.Errorf("restore after purge left status %q and request %q, want active and succeeded", status, request) + } + if figure, rows := env.figureAndRows(); figure != 1 || rows != 1 { + t.Errorf("restore after purge left usage %d with %d active rows and holds", figure, rows) + } +} diff --git a/internal/integrations/fedwiki/workflows/reconcile.go b/internal/integrations/fedwiki/workflows/reconcile.go index 93d535b7..003949c0 100644 --- a/internal/integrations/fedwiki/workflows/reconcile.go +++ b/internal/integrations/fedwiki/workflows/reconcile.go @@ -7,11 +7,13 @@ import ( "context" "database/sql" "errors" + "fmt" "log/slog" "time" "git.coopcloud.tech/wiki-cafe/member-console/internal/db" "git.coopcloud.tech/wiki-cafe/member-console/internal/entitlements" + "git.coopcloud.tech/wiki-cafe/member-console/internal/integration" fwmod "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki/store" ) @@ -105,14 +107,18 @@ func (a *Activities) PurgeExpiredArchivedActivity(ctx context.Context, _ struct{ out := &PurgeExpiredArchivedOutput{} for _, site := range expired { - if _, err := a.Client.HardDeleteSite(ctx, site.Domain); err != nil { + deleted, err := a.purgeArchivedSite(ctx, a.Database, site, cutoff) + if errors.Is(err, errRetentionFarmDelete) { a.Logger.Warn("retention purge: hard delete failed", slog.String("domain", site.Domain), slog.Any("error", err)) continue } - if err := a.SiteQ.DeleteSiteByDomain(ctx, site.Domain); err != nil { + if err != nil { a.Logger.Warn("retention purge: local delete failed", slog.String("domain", site.Domain), slog.Any("error", err)) continue } + if !deleted { + continue + } if a.Registry != nil { if _, err := ReleaseSiteDomain(ctx, a.Registry, site.SiteID); err != nil { // The site is gone from farm and database; a stranded @@ -127,6 +133,54 @@ func (a *Activities) PurgeExpiredArchivedActivity(ctx context.Context, _ struct{ return out, nil } +// errRetentionFarmDelete marks a candidate whose farm delete failed, so the +// purge logs it apart from a failed local step. +var errRetentionFarmDelete = errors.New("retention farm delete") + +// purgeArchivedSite holds the lifecycle instance lock and eligible site row +// through the farm delete and local commit. READ COMMITTED sees requests +// committed before the instance lock was granted and rechecks rows changed +// while waiting for their lock. Neither intake nor a status writer can pass +// these locks between the eligibility check and deletion. +// It reports true only after deletion commits; a gone, ineligible or busy +// candidate returns false and nil. An error returns false without reporting +// a committed deletion; a failed commit can have an unknown outcome. +func (a *Activities) purgeArchivedSite(ctx context.Context, database db.Beginner, candidate fwmod.Site, cutoff time.Time) (bool, error) { + deleted := false + err := db.InTx(ctx, database, db.ReadCommitted, func(tx *sql.Tx) error { + if err := integration.New(tx).LockLifecycleInstance(ctx, candidate.SiteID); err != nil { + return err + } + siteQ := fwmod.New(tx) + locked, err := siteQ.LockArchivedSiteForPurge(ctx, fwmod.LockArchivedSiteForPurgeParams{ + SiteID: candidate.SiteID, + Cutoff: cutoff, + }) + if errors.Is(err, sql.ErrNoRows) { + return nil + } + if err != nil { + return err + } + open, err := siteQ.SiteHasOpenRequest(ctx, candidate.SiteID) + if err != nil || open { + return err + } + if _, err := a.Client.HardDeleteSite(ctx, locked.Domain); err != nil { + return fmt.Errorf("%w: %w", errRetentionFarmDelete, err) + } + if err := siteQ.DeleteSite(ctx, locked.SiteID); err != nil { + return err + } + deleted = true + return nil + }) + if err != nil { + return false, err + } + return deleted, nil +} + // reconcilePoolQuota reconciles one pool across every workspace it is // primary for: the surplus parked is the pool's least recently modified // active sites, whichever workspace holds them, and the headroom reactivation diff --git a/openspec/changes/serialize-retention-purge-with-lifecycle/.openspec.yaml b/openspec/changes/serialize-retention-purge-with-lifecycle/.openspec.yaml new file mode 100644 index 00000000..4bfa6709 --- /dev/null +++ b/openspec/changes/serialize-retention-purge-with-lifecycle/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-10-30 diff --git a/openspec/changes/serialize-retention-purge-with-lifecycle/design.md b/openspec/changes/serialize-retention-purge-with-lifecycle/design.md new file mode 100644 index 00000000..4c531074 --- /dev/null +++ b/openspec/changes/serialize-retention-purge-with-lifecycle/design.md @@ -0,0 +1,81 @@ +## Context at HEAD + +Baseline: `dec031e6`. The existing paths explain which locks retention must join. + +| Path | Current behavior | +|---|---| +| Purge | One list, farm DELETE, local delete by domain, then placement release; no reread, lifecycle lock or recount (`internal/integrations/fedwiki/workflows/reconcile.go:99`). | +| Intake | Archive and restore read the site before recording. `RecordLifecycleRequest` takes the instance advisory lock and inserts the accepted request; workflow start follows commit. Rotations and swaps also take the workspace lock, and swaps record activation before park (`internal/integrations/fedwiki/web/site_actions.go:328`, `:475`, `:566`; `internal/integration/lifecycle.go:128`; `internal/integrations/fedwiki/web/requests.go:210`). | +| Status workflow | Restore's slot gate locks instance, site, then usage. Farm PATCH follows an ID-based authorization read without locks. Outcome recording locks instance, site, then usage; it removes holds, recounts and finishes the request in one transaction (`internal/integrations/fedwiki/usage/usage.go:221`; `internal/integrations/fedwiki/workflows/request_activities.go:338`, `:379`, `:207`). Archive needs no hold. | +| Status writers | `SetSiteStatusByDomain` maintains archive time. Farm upsert/insert do not, which is #246. Swap outcome locks both instances before site rows and usage. The quota sweep checks open requests before PATCH, then locks the row, validates status and recounts (`internal/integrations/fedwiki/store/queries/sites.sql:75`, `:88`, `:99`; `internal/integrations/fedwiki/workflows/swap.go:165`; `internal/integrations/fedwiki/workflows/reconcile.go:216`, `:256`, `:316`). | +| Sync | It skips known open-request domains. Status writes lock the row at READ COMMITTED, recheck requests and recount. Insert, observation and System-row deletion have their own paths. Sync takes no instance lock: a later outcome can write last, which cannot repair a hard delete (`internal/integrations/fedwiki/workflows/sync.go:303`, `:352`, `:454`, `:487`, `:512`, `:558`). | +| Usage and explicit delete | Usage is active rows plus accepted requests' holds, counted after locking usage. Explicit delete authorizes by ID, deletes on the farm, releases the name, then deletes locally and recounts (`internal/integrations/fedwiki/usage/usage.go:47`; `internal/integrations/fedwiki/workflows/activities.go:238`, `:301`). | + +## Goal + +Retention must not delete a site with an open accepted lifecycle request, or use a stale candidate to delete a site whose restore completed. The purge must check the current row and requests, and hold that decision through deletion. It must not remove an active row without updating usage. + +## Decisions + +### D1. Lock the instance, then the eligible row, through deletion + +Keep the initial candidate list and one cutoff per invocation. Extract `purgeArchivedSite`, accepting a `db.Beginner` so tests can pin a connection. For each candidate: + +1. Open `db.ReadCommitted`; take `LockLifecycleInstance(candidate.SiteID)`. +2. Run new query `LockArchivedSiteForPurge`: select by ID, require archived status and a non-null archive time strictly before cutoff, and lock the row `FOR UPDATE`. No row means skip. +3. Read `SiteHasOpenRequest` on the same transaction. Any accepted request means skip, with or without a slot hold. Any error stops this candidate before HTTP. +4. Call `HardDeleteSite(ctx, locked.Domain)` while both locks remain held. On success, including the client's existing 404 handling, delete the locked row by ID and commit. +5. Only after successful commit, report deletion, release its placement and increment `Purged`. Return not-deleted on any transaction error, including commit failure. + +A list error still fails the activity. Other errors are logged per candidate and the loop continues; placement-release failure remains best-effort. + +READ COMMITTED makes the request read see commits made before the instance lock was granted. It also lets a row changed during the row-lock wait be skipped normally. At REPEATABLE READ, that concurrent update instead raises 40001; tests must require both no deletion and a nil error. + +An unlocked reread or locks released before HTTP leave a race. A site lock alone cannot stop intake, which takes only the instance lock. A conditional local DELETE after HTTP protects the row too late to protect content. Taking the row before the instance would reverse the outcome/slot-gate order. These alternatives are rejected. + +### D2. Keep usage, names and archive timestamps as they are + +An eligible row is archived and no accepted request can acquire a counting hold under the held instance lock. Its deletion therefore changes no usage count; a completed restore is skipped. Recounting after deleting an active site would hide the counting error without saving its content. + +Release the placement by site ID after commit, in the registry's separate transaction. This preserves current recovery through boot's orphan sweep and avoids holding the global registry lock over HTTP (`internal/integrations/fedwiki/store/queries/sites.sql:292`). + +Use the row's current archive time, with no fallback, new column or backfill. Task 2.1's `active` and `readonly` rows with old timestamps and `null archive time`, and task 2.3's `cleared archive time`, model writers as they behave before #246. #246 rewrites those cases when it changes the writers; this change does not preserve those timestamp defects as a future contract. + +### D3. Accept the waiting cost and state the remote-delete limits + +The locked farm call can run 30 seconds (`internal/integrations/fedwiki/workflows/farmmanager_client.go:27`). The server's WriteTimeout is eight seconds (`internal/server/server.go:387`): a member request on this site waits in `RecordLifecycleRequest`, and if it runs past eight seconds loses its answer while its transaction carries on and may still record a request and start a workflow. Accept this under [forge issue #290](https://git.coopcloud.tech/wiki-cafe/member-console/issues/290), the existing lost-answer issue; do not shorten the farm call. The five-minute activity StartToClose limit bounds the entire purge loop, not each call (`internal/integrations/fedwiki/workflows/sync.go:57`). + +The purge holds only one instance and one site row over HTTP, not a pool, usage or registry lock. An HTTP timeout does not prove the farm stopped executing DELETE, and a database rollback cannot undo a remote deletion. Durable deletion claims or provider-side conditions require separate recovery work. + +## Lock order + +`docs/database-locks.md` rules 5 and 6 order FedWiki locks as instance locks, site rows, usage, then request outcome rows. Swaps take instances in recording order. Retention joins that order: one instance, one site row, HTTP, DELETE of that row, commit. Placement release starts its own registry transaction afterward. Intake, slot gate, outcome, sweep and sync orders remain unchanged. + +Update the lifecycle-instance and site-row entries and add a retention subsection to the lock document. No lock family or global ordering rule changes. + +## Failure and race table + +| Path | Today | After this change / limit | +|---|---|---| +| List fails | No deletion; activity error. | Same. | +| Candidate gone, replaced, active, readonly, archive time absent or at/after cutoff | Stale candidate can delete a changed site or replacement. | Locked ID/eligibility check skips without farm, usage or placement writes. | +| Accepted request, including restore PATCH in flight | Purge ignores it. | Instance lock and request read exclude deletion; hold and row survive. | +| Restore finishes after listing | Purge deletes active row and leaves usage high. | Current status excludes deletion; usage remains correct. | +| Row changes while purge waits | No reread. | READ COMMITTED returns an ordinary skip, with nil error, if eligibility changed. | +| Purge commits before intake | Stale intake can accept. | It waits until deletion commits, then may accept: a restore is declined, holding a slot until then; a delete finishes with nothing to delete. Nothing is deleted after that acceptance. | +| Member waits beyond eight seconds | Server loses the answer (#290). | The locked farm call can cause this wait; transaction and workflow start may still proceed. Accepted cost, not fixed here. | +| Eligible row; farm 200 or 404 | Local deletion and name release. | Same result under locks; delete by ID, count success only after commit, no usage write. | +| Begin, lock or guarded read fails | No guarded step. | No farm call; rollback, log and continue. | +| Farm refuses or response cannot be decoded | Row remains; later tick retries. | Same. An invalid response may conceal a completed remote deletion. | +| Timeout, cancellation, worker death or connection loss during DELETE | Remote effect may outlive caller. | Once the transaction ends, locks release; a newly accepted request can overlap a still-running remote DELETE. This residual race needs separate recovery/fencing. | +| Remote success, local delete/commit fails or commit answer is lost | Content gone; row may remain. | No reported success or placement release on error. A later eligible tick can clean up on 404. Before then a restore can be accepted against the surviving row, with no content to restore. | +| Placement release fails or process dies after commit | Orphan may remain until boot. | Same; registry work holds no site locks. | +| Two purge attempts | Both may send DELETE. | Second waits and skips the deleted ID. Ambiguous remote outcomes can still require repeated DELETEs. | +| Farm-only restore or stale sync observation | Local row can lag the farm; sync can recreate a stale projection. | Unchanged; database locks do not fence farm administrators or stale observations after commit. | +| Null/stale archive timestamp from farm projection | Never expires or uses an earlier archive's time. | #246 repairs the writers; this change reads their current values. | + +## Not in this change + +A request may be accepted for a site retention already removed: a restore is declined, holding a slot until then; a delete finishes with nothing to delete (`internal/integrations/fedwiki/workflows/request_activities.go:339`; `internal/integrations/fedwiki/workflows/activities.go:248`). The same stale acceptance follows a member's permanent delete (`internal/integrations/fedwiki/workflows/activities.go:301`); intake revalidation is recorded on forge issue #284 (comment 45122). + +No handler or visible copy changes; no hand captures are needed. Tests use real database locks and a fake farm; they do not prove that a remote server stops DELETE when its HTTP caller cancels. diff --git a/openspec/changes/serialize-retention-purge-with-lifecycle/proposal.md b/openspec/changes/serialize-retention-purge-with-lifecycle/proposal.md new file mode 100644 index 00000000..79cd7093 --- /dev/null +++ b/openspec/changes/serialize-retention-purge-with-lifecycle/proposal.md @@ -0,0 +1,35 @@ +## Why + +The retention purge lists expired archived sites once, then deletes each from the farm and database without checking it again. A restore accepted after that list can still be running, or can have finished, when the purge destroys the site. If the restore finished, the purge also leaves usage above the active-site count (`internal/integrations/fedwiki/workflows/reconcile.go:99`). + +## What Changes + +- For each candidate, open a READ COMMITTED transaction, lock its lifecycle instance, lock its still-expired archived row by identifier, and check for an accepted request. Hold both locks through the farm delete and local commit. +- Skip missing, changed or busy sites. Delete eligible sites by identifier, release their placements after commit, and leave usage unchanged. +- Add deterministic tests for eligibility, requests in flight, lock lifetime, isolation and failures. Document the lock order and remaining remote-delete failure windows. + +## Capabilities + +### New Capabilities + +None. + +### Modified Capabilities + +- `fedwiki-sites`: retention checks eligibility under locks that serialize deletion with lifecycle requests and site writes. + +## Impact + +Code: `internal/integrations/fedwiki/workflows/reconcile.go`, `internal/integrations/fedwiki/store/queries/sites.sql` and generated files. Tests: the workflows package. Documentation: `docs/database-locks.md`. No handler, migration, workflow command, UI copy, page or golden changes. + +A farm call under the locks can run 30 seconds (`internal/integrations/fedwiki/workflows/farmmanager_client.go:27`), while the server's WriteTimeout is eight seconds (`internal/server/server.go:387`). A member request on that site waits on the instance lock and, if it runs past eight seconds, loses its answer while its transaction carries on and may still record a request and start a workflow. This cost is accepted under [forge issue #290](https://git.coopcloud.tech/wiki-cafe/member-console/issues/290); the farm timeout stays unchanged. The five-minute StartToClose limit bounds the whole purge loop, not one call (`internal/integrations/fedwiki/workflows/sync.go:57`). + +## Not in this change + +A request may be accepted for a site the purge already removed: a restore is declined, holding a slot until then; a delete finishes with nothing to delete. The same stale acceptance occurs after a member's permanent delete; intake revalidation is recorded on forge issue #284 (comment 45122). + +#246 owns archive timestamp maintenance and its test updates. #247 owns placement servability, #283 owns scheduling, and #284 owns status/swap outcome lookup. Durable deletion recovery, provider-side fencing and changes to the member's delete workflow remain separate work. + +## Related issue + +This change addresses #245. diff --git a/openspec/changes/serialize-retention-purge-with-lifecycle/specs/fedwiki-sites/spec.md b/openspec/changes/serialize-retention-purge-with-lifecycle/specs/fedwiki-sites/spec.md new file mode 100644 index 00000000..0ec3cf38 --- /dev/null +++ b/openspec/changes/serialize-retention-purge-with-lifecycle/specs/fedwiki-sites/spec.md @@ -0,0 +1,71 @@ +## MODIFIED Requirements + +### Requirement: Member site archival is reversible + +A member "Archive" action SHALL move the site to the `archived` state — offline but with content retained — rather than immediately erasing it. Archived sites SHALL be hidden from the member's active sites list and presented in a separate "Archived" section. A member SHALL be able to restore an archived site to `active` (quota-gated). An archived site SHALL be permanently purged (FarmManager `?hard=true`) after a retention window of 30 days only while its current row is still archived, its non-null archive timestamp is strictly before the retention cutoff, and no lifecycle request on its instance is open (`accepted`), or by an explicit "Delete permanently" action confirmed by the member (operators may also purge). Permanent purge is the only irreversible operation. + +The retention list SHALL supply candidates, not authority to delete. For each candidate, the purge SHALL take the lifecycle instance lock and then lock the eligible site row by its immutable identifier in a READ COMMITTED transaction. It SHALL check open requests after those locks and SHALL keep both locks through the farm hard delete and local row deletion's commit. A missing or ineligible row or an open request SHALL be skipped until a later tick; a failed eligibility or request read SHALL make no farm call. The local delete SHALL use the locked site's identifier. Only a successful local commit SHALL count as purged and permit placement release. An eligible archived deletion SHALL leave usage unchanged. A farm failure or local transaction failure SHALL leave cleanup to a later eligible attempt; the console SHALL NOT treat a database rollback as reversing a remote deletion. + +#### Scenario: Archive moves the site to the archived state + +- **WHEN** a member archives a site +- **THEN** the site SHALL move to `archived` with its content retained +- **AND** the site SHALL be restorable +- **AND** the site SHALL no longer appear in the active sites list + +#### Scenario: Restore from archived is quota-gated + +- **WHEN** a member restores an archived site and the pool's active sites and holds are below the limit +- **THEN** the restore's gate SHALL hold a slot, and the site SHALL return to `active` and count toward the pool as an active row +- **WHEN** the pool's active sites and holds already reach the limit +- **THEN** the restore SHALL be refused + +#### Scenario: Archived site is purged after retention + +- **WHEN** an archived site's 30-day retention window has elapsed and its locked row is still eligible with no accepted lifecycle request +- **THEN** the system SHALL permanently purge it via a hard delete + +#### Scenario: Explicit delete-permanently purges immediately + +- **WHEN** a member confirms "Delete permanently" on an archived site +- **THEN** the system SHALL permanently purge it via a hard delete + +#### Scenario: A restore is in flight when retention reaches the site + +- **WHEN** the retention list contains an expired archived site whose restore request was accepted before the purge acquired its instance lock +- **AND** the restore's farm call is still in progress +- **THEN** retention SHALL skip the site without a hard delete, local deletion, placement release or usage change +- **AND** the restore SHALL retain its slot hold and be able to record its outcome + +#### Scenario: A restore finishes after the candidate list + +- **WHEN** a restore makes a listed candidate active before the purge acquires its site row lock +- **THEN** retention SHALL leave the site and its placement intact and SHALL leave the restore's usage figure unchanged + +#### Scenario: The archive time changes after the candidate list + +- **WHEN** a listed candidate's locked row has a null archive time or an archive time at or after the cutoff +- **THEN** retention SHALL send no hard delete and SHALL leave the row for a later tick + +#### Scenario: A replacement under the same name is not a purge candidate + +- **WHEN** a candidate's identifier no longer has a row and its former domain now belongs to a different site identifier +- **THEN** retention SHALL send no hard delete for that candidate and SHALL leave the replacement's row and placement intact + +#### Scenario: Retention holds its decision through the farm call + +- **WHEN** retention has locked an eligible site with no accepted request and is waiting for the farm hard delete +- **THEN** new lifecycle intake on that instance and status writes on that row SHALL wait until the purge transaction ends +- **AND** a concurrent purge SHALL check eligibility again after acquiring the instance lock + +#### Scenario: A failed check cannot authorize deletion + +- **WHEN** retention cannot begin its transaction, acquire its locks or read eligibility or open requests +- **THEN** it SHALL send no farm delete for that candidate and SHALL report the failure rather than treat it as eligibility + +#### Scenario: A failed candidate does not prevent another purge + +- **WHEN** one eligible candidate's farm delete fails and the next candidate is eligible and can be deleted +- **THEN** the first candidate's row and placement SHALL remain and the second candidate SHALL be purged +- **AND** the reported purge count SHALL include only successful local commits + diff --git a/openspec/changes/serialize-retention-purge-with-lifecycle/tasks.md b/openspec/changes/serialize-retention-purge-with-lifecycle/tasks.md new file mode 100644 index 00000000..60e4c98f --- /dev/null +++ b/openspec/changes/serialize-retention-purge-with-lifecycle/tasks.md @@ -0,0 +1,235 @@ +Tests follow `docs/testing.md`: open through `dbtest`, build through `fedwikitest` and existing fixtures, filter observations to fixture IDs, and pass twice without reset. Use channels and `dbtest.AwaitLockWait`, never sleeps. Cleanup releases blocked calls before joining goroutines. Run assertions on the test goroutine and run the workflows package with `-count=1 -p 1`; run concurrency tests with `-race` too. + +Implementation, tests and documents form one commit. Landing is the caller's work: `make test` with the stack up, the break runner, captures if a page changes, and the commit. No golden, route, workflow history, page or screen baseline should change. Investigate unexpected differences. Learn script usage from its source. + +**Breaks.** In phase 3 write the section 3 records to a file outside the repository, using `scripts/mutation-check/main.go`'s format. Each is a new-test record with no old side. Apply each break, run `go vet` on its package, then restore the file. The caller runs `go run ./scripts/mutation-check -list -side new` with the test environment loaded, after green controls; every record must be CAUGHT. A `want_new` is failure-only assertion text, never a test name. The commit message lists results and untested limits. + +## 1. Guard each retention candidate + +- [x] 1.1 Add `LockArchivedSiteForPurge` to `internal/integrations/fedwiki/store/queries/sites.sql` with the SQL below. Leave the candidate listing and timestamp writers unchanged. Regenerate with `make sqlc-generate`, including its header-restoration step; retain only relevant generated changes. + +```sql +-- name: LockArchivedSiteForPurge :one +SELECT * FROM fedwiki.sites +WHERE site_id = sqlc.arg(site_id)::uuid + AND status = 'archived' + AND archived_at IS NOT NULL + AND archived_at < sqlc.arg(cutoff)::timestamptz +FOR UPDATE; +``` + +- [x] 1.2 Extract `purgeArchivedSite(ctx, database db.Beginner, candidate fwmod.Site, cutoff time.Time)` as an Activities method, returning a deletion flag and error. Open `db.ReadCommitted`, take the instance lock, create `siteQ := fwmod.New(tx)`, lock the eligible row as `locked`, then read `open, err := siteQ.SiteHasOpenRequest(ctx, candidate.SiteID)`. Use `if err != nil || open { return err }`: a request skips; a read error fails. No eligible row also skips. Document why READ COMMITTED and both locks are needed. +- [x] 1.3 Hold both locks over `HardDeleteSite(ctx, locked.Domain)` and `siteQ.DeleteSite(ctx, locked.SiteID)`. Return not-deleted on any transaction error, including commit failure. The helper wraps farm errors so the activity logs them as `retention purge: hard delete failed`; it logs other candidate errors as `retention purge: local delete failed`. Both branches continue to later candidates; the local-delete branch retains its `continue`. Otherwise it releases the placement and increments `Purged` only for a committed deletion. Keep placement release best-effort. No usage write, intake change, timestamp repair, timeout change or workflow command change. + +## 2. Tests with their rows and failure text + +- [x] 2.1 Add `TestRetentionPurgeRechecksCandidates_DB` in `internal/integrations/fedwiki/workflows/purge_requests_db_test.go`, running table rows through a named helper. Use a test-local `fwmod.Querier` wrapper that lists only the fixture's candidates, captures the actual cutoff, and mutates the fixture after taking its candidate snapshot but before returning it. This exercises the activity and avoids purging unrelated rows from the shared database. Its rows are: + + | Row | State at the guarded check | Expected result | + |---|---|---| + | `expired` | Still archived, timestamp before cutoff, no request | One farm DELETE with `hard=true`, row and placement gone; `Purged = 1` | + | `active` | Restored active after listing, retaining an old timestamp in this fixture to isolate the status guard | No DELETE, row/placement survive | + | `readonly` | Readonly with an old timestamp after listing | No DELETE, row/placement survive | + | `fresh archive` | Re-archived after listing, timestamp newer than cutoff | No DELETE, row/placement survive | + | `null archive time` | Timestamp cleared after listing | No DELETE, row/placement survive | + | `at cutoff` | Timestamp set to the captured cutoff exactly | No DELETE, row/placement survive | + | `missing` | Original row deleted after listing | No DELETE or release of its remaining placement | + | `replacement` | Original row deleted, same name inserted under a new ID with an expired timestamp | Neither replacement farm name nor row/placement is deleted | + | `open restore before gate` | Accepted active request, no hold | No DELETE | + | `open restore with hold` | Accepted active request, real gate stamped hold | No DELETE; held usage survives | + | `open archive` / `open delete` / `open swap` | Accepted corresponding lifecycle request | No DELETE | + | `finished request` | Prior request finished, archive still expired | Purge proceeds | + + Check farm method/path/query, retained site IDs and states, placements and `Purged`, not just the activity's nil error. Recount fixture setup as needed; compare stored usage with active rows plus counting holds, and verify the purge did not change it. Failure texts: `%s: hard deletes = %v, want %v`; `%s: retained site IDs = %v, want %v`; `%s: retained placements = %v, want %v`; `%s: purged = %d, want %d`; `%s: usage = %d, counted rows = %d, want %d`. Use nonfatal checks where a break must reach more than one assertion. + +- [x] 2.2 Add `TestRetentionPurgeWaitsForLifecycleIntake_DB` in `purge_locks_db_test.go`. Hold the instance advisory lock in a side transaction and insert an accepted restore through the real recorder without committing; start the per-candidate helper on a pinned second connection. Await its instance-lock wait, commit intake, then expect no farm DELETE and no removed row. Run rows `default isolation` and `server default repeatable read`; the latter opens a dedicated test handle with `default_transaction_isolation=repeatable%20read` in its DSN, as `internal/server/operator_product_tier_set_db_test.go:366` does. The handle closes on cleanup; no setting needs restoring. The helper must still choose READ COMMITTED and see the committed request. Abort text `purge did not wait for lifecycle intake`; result failure `purge deleted a site whose restore committed before its lock`. This proves acceptance and the fresh post-wait request snapshot, not just a pre-existing open request. + +- [x] 2.3 Add `TestRetentionPurgeWaitsForSiteEligibility_DB`. Hold only the site row in a side transaction, start the helper, await its row-lock wait, then commit one of `active`, `fresh archive`, `cleared archive time`. Each must return not-deleted with a nil error and skip the farm. Assert the result directly, without a helper that fails on any returned error before this assertion: `%s: purge result = deleted %t, error %v; want false and nil`. A `db.Plain` transaction at REPEATABLE READ raises 40001 after the concurrent update and must fail that assertion even though it sends no DELETE. Abort text `purge did not wait for the site row`; result failure `%s: purge used eligibility from before the row lock`. Use the dedicated REPEATABLE READ DSN from 2.2 and pin its connection. This proves an ordinary skip at READ COMMITTED, not merely the absence of a farm call. + +- [x] 2.4 Add `TestRetentionPurgeHoldsLocksThroughFarmDelete_DB`. Use a channel-controlled `httptest` DELETE handler that records arrival and waits for release. Once DELETE arrives, start a direct site update and observe it blocked on the first purge. Only then start a second purge and observe its instance-lock wait. Select the first purge's backend as the blocker and provide bounded abort probes. Release the farm response, join all calls, and assert one successful purge and one hard delete for the original candidate. Abort texts `site writer passed the purge's held row` and `second purge passed the first purge's instance lock`; result text `concurrent purge hard deletes = %d, want 1`. This test also runs the real new locking query under `SET ROLE console_runtime` on the pinned purge connection; reset the role on cleanup. + +- [x] 2.5 Add `TestRetentionPurgeLeavesARestoreInFlight_DB`. Record an actual restore, run its real slot-gate and farm activity with the PATCH held at a channel barrier, then run the purge over its expired candidate. Assert the held slot and row survive, and no DELETE reaches the farm. Release PATCH, run the real status outcome, and assert active row, succeeded request and matching usage. Failure texts `purge sent DELETE while the accepted restore was in flight`; `restore after purge left status %q and request %q, want active and succeeded`; `restore after purge left usage %d with %d active rows and holds`. Reuse `reqEnv` and the existing fake farm where its hooks suffice; use a test-local HTTP wrapper to record DELETEs rather than add a general test framework. + +- [x] 2.6 Add `TestRetentionPurgeFailures_DB` in `purge_failures_db_test.go`, named table runner, fixture-filtered candidates. Rows: `list error` (wrapper returns a sentinel; activity returns it, no calls), `begin error` (closed database passed to the helper, no farm calls), `cancelled lock wait` (cancel only after observing the wait, no farm calls), `farm refusal` (500, retain row/name and zero success), `farm already absent` (404, successful cleanup), `local failure after farm success`, and `later candidate after refusal` (the farm refuses the first, an intervening candidate with an invalid ID fails locally, and the last succeeds). This extra local failure proves that both warning branches continue, including the local-delete branch named by the break. For local failure, wrap the test client's response body: after a valid 200 body's bytes have been read, cancel the helper context on EOF while returning the bytes and EOF normally. The client can decode its successful result, but the following local DELETE cannot use the cancelled context. Verify the fake farm deleted its site, the database row/name remain, and the helper reports no success. Assert errors for the helper versus the activity's best-effort result, calls, rows and placements. Failure texts `%s: purge error = %v, want %v`; `%s: farm calls = %d, want %d`; `%s: purged = %d, want %d`; `%s: retained site IDs = %v, want %v`; `%s: retained placements = %v, want %v`. The cancellation row demonstrates a remote-success/local-failure gap, not its recovery. Keep the existing `TestRetentionPurge` passing. + +## 3. Break records and documents + +These are exact replacements for the planned implementation in section 1, not edits to HEAD. Phase 3 checks that each `old` occurs once and each replacement compiles; if final spelling differs, update the literal record while retaining its test and effect. SQL records edit generated code. The identity record substitutes the current domain owner's ID; the early-commit record releases the locks before HTTP. The open-request check has one record; both request tests exercise the same rule. + +```json +{ + "breaks": [ + { + "name": "current status", + "file": "internal/integrations/fedwiki/store/sites.sql.go", + "old": "WHERE site_id = $1::uuid\n AND status = 'archived'", + "new": "WHERE site_id = $1::uuid\n AND status IN ('active', 'readonly', 'archived')", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeRechecksCandidates_DB$/^active$", + "want_new": "active: hard deletes =", + "side": "new" + }, + { + "name": "current archive time", + "file": "internal/integrations/fedwiki/store/sites.sql.go", + "old": " AND archived_at IS NOT NULL\n AND archived_at < $2::timestamptz", + "new": " AND archived_at IS NOT NULL\n AND $2::timestamptz IS NOT NULL", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeRechecksCandidates_DB$/^fresh_archive$", + "want_new": "fresh archive: hard deletes =", + "side": "new" + }, + { + "name": "null archive time", + "file": "internal/integrations/fedwiki/store/sites.sql.go", + "old": " AND archived_at IS NOT NULL\n AND archived_at < $2::timestamptz", + "new": " AND (archived_at IS NULL OR archived_at < $2::timestamptz)", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeRechecksCandidates_DB$/^null_archive_time$", + "want_new": "null archive time: hard deletes =", + "side": "new" + }, + { + "name": "strict cutoff", + "file": "internal/integrations/fedwiki/store/sites.sql.go", + "old": " AND archived_at < $2::timestamptz", + "new": " AND archived_at <= $2::timestamptz", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeRechecksCandidates_DB$/^at_cutoff$", + "want_new": "at cutoff: hard deletes =", + "side": "new" + }, + { + "name": "candidate identity", + "file": "internal/integrations/fedwiki/workflows/reconcile.go", + "old": "siteQ := fwmod.New(tx)", + "new": "siteQ := fwmod.New(tx)\n\t\tcurrent, err := siteQ.GetSiteByDomain(ctx, candidate.Domain)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tcandidate.SiteID = current.SiteID", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeRechecksCandidates_DB$/^replacement$", + "want_new": "replacement: hard deletes =", + "side": "new" + }, + { + "name": "open request skips", + "file": "internal/integrations/fedwiki/workflows/reconcile.go", + "old": "if err != nil || open {", + "new": "if err != nil || (open && false) {", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeRechecksCandidates_DB$/^open_restore_before_gate$", + "want_new": "open restore before gate: hard deletes =", + "side": "new" + }, + { + "name": "lifecycle instance lock", + "file": "internal/integrations/fedwiki/workflows/reconcile.go", + "old": "integration.New(tx).LockLifecycleInstance(ctx, candidate.SiteID)", + "new": "integration.New(tx).LockLifecycleInstance(ctx, \"00000000-0000-0000-0000-000000000000\")", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeWaitsForLifecycleIntake_DB$/^default_isolation$", + "want_new": "purge did not wait for lifecycle intake", + "side": "new" + }, + { + "name": "request snapshot after waiting", + "file": "internal/integrations/fedwiki/workflows/reconcile.go", + "old": "db.InTx(ctx, database, db.ReadCommitted,", + "new": "db.InTx(ctx, database, db.Plain,", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeWaitsForLifecycleIntake_DB$/^server_default_repeatable_read$", + "want_new": "purge deleted a site whose restore committed before its lock", + "side": "new" + }, + { + "name": "row update is a successful skip", + "file": "internal/integrations/fedwiki/workflows/reconcile.go", + "old": "db.InTx(ctx, database, db.ReadCommitted,", + "new": "db.InTx(ctx, database, db.Plain,", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeWaitsForSiteEligibility_DB$/^active$", + "want_new": "active: purge result =", + "side": "new" + }, + { + "name": "site row lock", + "file": "internal/integrations/fedwiki/store/sites.sql.go", + "old": " AND archived_at < $2::timestamptz\nFOR UPDATE", + "new": " AND archived_at < $2::timestamptz", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeHoldsLocksThroughFarmDelete_DB$", + "want_new": "site writer passed the purge's held row", + "side": "new" + }, + { + "name": "locks last through HTTP", + "file": "internal/integrations/fedwiki/workflows/reconcile.go", + "old": "if _, err := a.Client.HardDeleteSite(ctx, locked.Domain); err != nil {", + "new": "if err := tx.Commit(); err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif _, err := a.Client.HardDeleteSite(ctx, locked.Domain); err != nil {", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeHoldsLocksThroughFarmDelete_DB$", + "want_new": "site writer passed the purge's held row", + "side": "new" + }, + { + "name": "local deletion", + "file": "internal/integrations/fedwiki/workflows/reconcile.go", + "old": "if err := siteQ.DeleteSite(ctx, locked.SiteID); err != nil {", + "new": "if err := error(nil); err != nil {", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeRechecksCandidates_DB$/^expired$", + "want_new": "expired: retained site IDs =", + "side": "new" + }, + { + "name": "placement release", + "file": "internal/integrations/fedwiki/workflows/reconcile.go", + "old": "if a.Registry != nil {", + "new": "if a.Registry != nil && false {", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeRechecksCandidates_DB$/^expired$", + "want_new": "expired: retained placements =", + "side": "new" + }, + { + "name": "continue after candidate failure", + "file": "internal/integrations/fedwiki/workflows/reconcile.go", + "old": "a.Logger.Warn(\"retention purge: local delete failed\", slog.String(\"domain\", site.Domain), slog.Any(\"error\", err))\n\t\t\tcontinue", + "new": "a.Logger.Warn(\"retention purge: local delete failed\", slog.String(\"domain\", site.Domain), slog.Any(\"error\", err))\n\t\t\tbreak", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeFailures_DB$/^later_candidate_after_refusal$", + "want_new": "later candidate after refusal: purged =", + "side": "new" + }, + { + "name": "farm refusal keeps the row", + "file": "internal/integrations/fedwiki/workflows/reconcile.go", + "old": "if _, err := a.Client.HardDeleteSite(ctx, locked.Domain); err != nil {", + "new": "if _, err := a.Client.HardDeleteSite(ctx, locked.Domain); err != nil && false {", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeFailures_DB$/^farm_refusal$", + "want_new": "farm refusal: retained site IDs =", + "side": "new" + }, + { + "name": "404 is success", + "side": "new", + "file": "internal/integrations/fedwiki/workflows/farmmanager_client.go", + "old": "\"?hard=true\", nil)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tdefer resp.Body.Close()\n\n\tbody, err := c.readBody(resp)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// Treat 404 as success - site is already gone\n\tif resp.StatusCode == http.StatusNotFound {", + "new": "\"?hard=true\", nil)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tdefer resp.Body.Close()\n\n\tbody, err := c.readBody(resp)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// Treat 404 as success - site is already gone\n\tif false {", + "pkg": "./internal/integrations/fedwiki/workflows", + "run_new": "^TestRetentionPurgeFailures_DB$/^farm_already_absent$", + "want_new": "farm already absent: retained site IDs =" + } + ] +} +``` + +- [x] 3.1 Write and compile-check all records. +- [x] 3.2 Report evidence limits: connection loss with unknown commit, remote DELETE after cancellation, process death, farm-only changes and stale sync insertion remain analysis-only cases. Tests prove the lock protocol and local rollback, not remote cancellation. A failed eligibility or open-request read inside the purge transaction is checked by reading the code only: `purgeArchivedSite` returns the error before the farm call. It constructs `fwmod.New(tx)` inside the transaction, and neither the activity's candidate-list wrapper nor the database test kit exposes a way to fail that read. No production seam is added for this test. +- [x] 3.3 Update `docs/database-locks.md`: add retention to the instance and site-row lock entries and describe its order, READ COMMITTED, HTTP under those locks and separate registry release. Do not change intake or timestamp documentation. +- [x] 3.4 Run the six new tests, existing `TestRetentionPurge` and `TestWorkflowHistoriesReplay` twice without reset in workflows; run the concurrency tests under `-race`. Keep goldens and histories unchanged and record skips. +- [x] 3.5 Run `openspec validate serialize-retention-purge-with-lifecycle --strict`. + +## 4. Landing (caller only) + +- [x] 4.1 Run `make test` and the break list with green controls; require every record CAUGHT and record results and limits in the commit message. Hand captures only if a page changes; none are planned. +- [ ] 4.2 Review that only #245 is implemented, commit once, and handle closure and archive. No forge write is assigned to the phase worker.