cgalo5758
e52fdb37c8
Second of two changes for #186; closes it. Each kind of object the console serializes on now has a family, and every advisory key is built one way: hashtextextended('<family>:<id>', 0), with a UUID written in canonical form. db.Lock and db.LockShared (internal/db/locks.go) build the key and take the lock on a *sql.Tx only, and each family's statement begins with a name line, so pg_stat_activity and the test traces show which family a backend waits on. Families, in the order a transaction takes them: materialization, domains-registry, fedwiki-workspace, lifecycle, stripe-subscription, stripe-invoice, stripe-product-sync, desired-state. The lifecycle and desired-state keys are unchanged byte for byte; the other six change. Migration 00040 replaces core.assert_rendezvous to look for the new materialization key (owner, grants, SECURITY DEFINER and search_path kept; Down restores migration 18's body), and both seeds take the new key right after BEGIN. docs/database-locks.md lists the families in a table and writes one order every path keeps: the materialization lock first, then the families in table order, then rows. The project lint gains advisory-lock-site (an advisory lock function named in a Go string or in SQL or shell code outside the helpers, the test kit's probe, the migrations that name the rendezvous and the rule's own tests; a seed taking any other key, or the rendezvous anywhere but first after BEGIN), pool-lock-site (a core.resource_pools row lock outside LockResourcePool and LockGrantPool, or a call or method value of those outside entitlements/tx.go) and lock-families (the doc's table against the code). The walk covers cmd/, scripts/, internal/, test/ and the root's Go files, and skips testdata and directories whose name begins with a dot, as the go tool does. Tests take a family's lock through the same helpers and probe one with dbtest.TryLock. Recording a lifecycle request comes in two forms: RecordLifecycleRequest opens its own transaction on a database or a pinned connection, and RecordLifecycleRequestInTx runs in the caller's *sql.Tx, so a connection is never used outside a transaction (which released the instance lock at once) and does not compile in the second. The same holds for RecordFirstLifecycleRequest. docs/production-deployment.md now states the upgrade order for every upgrade: stop every process of the running release, run migrate up if migrations run separately, then start the new release; two releases running at once can build different keys for one lock and share none. docs/go-conventions.md, docs/testing.md and docs/building-an-integration.md follow the code. The lint command's help text names the files the lock rules read. 202 golden lines in 109 files change their statement text. Deploy: stop every process of the previous release first. Rollback: stop this release; as the migration user, in one transaction, run 00040's Down section and DELETE FROM goose_db_version_core WHERE version_id = 40; start the previous release (migrate down rolls back the last integration stream first). A deployment using console_reader re-runs scripts/db-reader-role.sql. Break list: 35 records, 35 CAUGHT. Named without a record: the *sql.Tx parameters of the helpers and the -InTx forms (a break that passes a connection does not compile, and the build is the check), and the comments. Full suite on reset databases: 5727 passed, 16 skipped, none failed.
Test Environment
This directory contains the Docker Compose setup and test data for local development.
LLM agents: read AGENTS.md first. All commands below are run from this
test/directory. If you are working in a git worktree alongside other worktrees of this repo, you must run./bootstrap-stack.shbeforedocker compose upto avoid host port collisions. One exception: the FedWiki HTTPS proxy (Caddy on443) is a cross-stack singleton among stacks that compose it — see "Running multiple isolated stacks" below.
Quick Start
cd test
./bootstrap-stack.sh
docker compose up -d
set -a; . .env; set +a
go run .. start --config mc-config.yaml
That composes the core stack: Postgres, Valkey, Keycloak, Temporal. Every integration (FedWiki, Discourse) sits behind a compose profile and is off by default, so the console boots without them.
To compose one, set the COMPOSE_PROFILES knob — either at bootstrap time:
COMPOSE_PROFILES=fedwiki ./bootstrap-stack.sh
or by editing that one line in the generated test/.env and re-running
docker compose up -d. Then uncomment that integration's MC_* block in the
same file so the app is configured for it, not merely running beside it. See
AGENTS.md.
To run the Go test suite against its own pristine databases (not the app's),
use make test from the repo root. See
AGENTS.md.
Running multiple isolated stacks (parallel worktrees)
compose.yaml accepts per-stack overrides via .env. The
bootstrap-stack.sh helper generates one with a deterministic, collision-free
port allocation derived from the worktree's directory name. Run the four
Quick Start commands above in worktree A and worktree B — the two stacks
coexist on disjoint host ports under separate COMPOSE_PROJECT_NAME
namespaces.
Exception — the HTTPS farm is a singleton. This applies only to stacks that
select the fedwiki profile; core-only stacks never contend for the port.
Browser access to FedWiki sites goes through a Caddy TLS proxy on host port
443 (wiki-security-social sets Secure cookies, which browsers only store
over HTTPS). That port is fixed, not slot-allocated, so only one composing
stack can serve browser login at a time — a second such stack's caddy
container fails to bind 443 while the rest of its services come up normally.
A slot-allocated port wouldn't help anyway: better-auth scopes
cookies to the shared *.localtest.me domain (per-domain, not per-port —
RFC 6265), so parallel stacks would clobber each other's session cookies
regardless. Everything non-browser stays fully isolated: member-console's farm
API calls use plain HTTP over the slot-allocated FEDWIKI_PORT with a bearer
token (no cookies). True per-stack browser isolation needs a per-stack farm
domain — tracked as forge issue #48.
To return a worktree to a clean state (containers + volumes + .env +
secrets/ + testdata/):
./teardown-stack.sh
See AGENTS.md for the full contract, including the Stripe webhook constraint when running stacks in parallel.
Services
URLs below are the single-stack defaults; after bootstrap-stack.sh the host
ports and *.localhost hostnames are per-stack (see the generated .env).
Core services, always composed:
| Service | URL | Credentials |
|---|---|---|
| Member Console | http://localhost:8081 | (via Keycloak) |
| Keycloak | http://keycloak.localhost:8080 | admin / admin |
| Temporal UI | http://localhost:8233 | (via Keycloak) |
| PostgreSQL | localhost:5432 | runtime member_console_runtime / member_console_runtime; migration member_console / member_console |
| Valkey | localhost:6379 | — |
Integration services, composed only when their profile is selected:
| Service | Profile | URL | Credentials |
|---|---|---|---|
| FedWiki (browser) | fedwiki |
https://admin.localtest.me | (via Keycloak) |
| FedWiki API | fedwiki |
http://admin.localtest.me | (bearer token, see the MC_FEDWIKI_* block in .env) |
| Discourse | discourse |
http://discourse.localhost:9292 | (via Keycloak) |
All credentials above are throwaway test-only values — never reuse any of them in a real deployment.
Browser access to FedWiki is HTTPS via the Caddy proxy (Caddy's internal CA — to
trust it for manual browsing, export the root once:
docker compose cp caddy:/data/caddy/pki/authorities/local/root.crt ./caddy-root.crt).
See AGENTS.md for the HTTPS/cert details.
Test Data
Seed identity
The database generates every ID, so a seed never supplies one; it addresses
the rows it owns by key (grammar ^[a-z][a-z0-9_]*$) and upserts on that
key, which is what makes re-running a seed a no-op. mockshot-seed.sql and
mockshot-nuke.sql are the deployment-scale sample-data pair used for
screenshots: both work from the mockshot_ organization key prefix and the
mockshot: OIDC subject prefix, so the nuke removes exactly what the seed
created. The four demo users' Keycloak ids are pinned as literals in
seed/keycloak/seed-keycloak.sh, and the
demo seeder mirrors them as its persons' OIDC subjects.
The testdata/ directory is named like that so that gopls can ignore it by default. It contains sample data for testing the Member Console, such as:
Keycloak
Keycloak runs with start-dev (an ephemeral H2 database, no persistent volume). It does not import a realm file. Instead, the one-shot keycloak-seed service runs seed/keycloak/seed-keycloak.sh on every docker compose up, which — authenticating as the master-realm bootstrap admin — creates a dedicated test app realm and seeds its clients, roles, protocol mappers, client scopes, and test users over the Keycloak admin API (idempotent). App OIDC runs on test, not Keycloak's master admin realm, so the end-user account console works.
Seeded into the test realm:
- Clients: member-console, fedwiki, temporal-ui, temporal-authz (plus service accounts)
- Users: alice (operator), bob, carlos, diana — password
password
To change the realm setup, edit seed/keycloak/seed-keycloak.sh — there is no realm export/import step. (An old testdata/keycloak/master-realm.json export remains on disk for reference but is untracked and unused by compose.)
FedWiki
Everything in this section applies only when the fedwiki profile is selected.
FedWiki runtime data lives in data/fedwiki/ (gitignored status/ dirs). Seed data for owner files and access tokens is committed in seed/fedwiki/.
A fedwiki-init container automatically copies seed files into data/fedwiki/ on docker compose up without overwriting existing files (cp -rn). This means:
- Fresh clone: seed data is copied in automatically, no manual steps needed.
- Existing setup: your local data is preserved.
To update seed data after making changes (e.g., new tokens or wiki sites):
# Copy the relevant files from data/fedwiki/ to seed/fedwiki/
cp data/fedwiki/<site>/status/owner.json seed/fedwiki/<site>/status/
cp data/fedwiki/<site>/status/user-access-tokens.json seed/fedwiki/<site>/status/