member-console/docs/database-management.md

title, audience, summary

title	audience	summary
Database Management	developer admin	goose migrations, sqlc code generation, and the core + per-integration schema and migration-stream conventions.

Database Management

This project uses pressly/goose for database migrations and sqlc for type-safe SQL code generation. Requires PostgreSQL 18+ (for native uuidv7()).

Schema layout

The database has five PostgreSQL schemas:

• core — every domain table (identity, organization, billing, entitlements, and the provider registry that integrations register against). This is the single source of business truth; there is no per-Go-package schema split. internal/db/migrations/ owns this stream.
• domains — the domain registry (claims, name rules, placements), a core module of its own. internal/domains/migrations/ owns this stream.
• stripe — the Stripe provider's own mapping tables (customers, subscriptions, invoices, etc.), FK'd into core. internal/integrations/stripe/store/migrations/ owns this stream.
• fedwiki — the FedWiki provider's own tables (sites, site swap policy), FK'd into core. internal/integrations/fedwiki/store/migrations/ owns this stream.
• discourse — the Discourse provider's own tables (group mappings, observed group members, user links), FK'd into core. internal/integrations/discourse/store/migrations/ owns this stream.

The Go package layout does not mirror the schema layout: internal/identity, internal/organization, internal/billing, internal/entitlements, and internal/integration are separate packages that all read and write tables in the single core schema. Schema boundaries track core domain vs. provider integration, not Go package boundaries — see the schema-consolidation change record under openspec/changes/archive/ for the rationale.

Per-Source Migrations

Each schema is owned by exactly one embedded migrations/ directory:

internal/db/migrations/                            # core schema (identity, organization, billing, entitlements, provider registry)
internal/domains/migrations/                        # domains schema (claims, name_rules, placements)
internal/integrations/fedwiki/store/migrations/     # fedwiki schema (sites, site_swap_policy)
internal/integrations/stripe/store/migrations/      # stripe schema (mapping tables)
internal/integrations/discourse/store/migrations/   # discourse schema (group_mappings, observed_group_members, user_links)

Each source runs against its own goose ledger table, goose_db_version_<name> (goose_db_version_core, goose_db_version_domains, goose_db_version_fedwiki, goose_db_version_stripe, goose_db_version_discourse), and keeps its native 00001… file numbering. There is no shared goose_db_version table and no global version namespace — a source's version numbers are independent of its position in the dependency-ordered source list returned by internal/migrate.Sources(). internal/db/migrations.go loops the sources in order, pointing goose at each source's embedded FS and ledger table in turn.

Source order matters only weakly: core must run first, because the FK DAG is one-directional — the other schemas reference core, never the reverse, and integration schemas never reference each other — so core's tables must exist before any other source's migrations apply. domains runs next, because integration migrations grant on its registry tables (FedWiki's grant domains_writer to fedwiki_writer). Beyond that, integration sources are mutually order-independent. Appending a new integration source, or reordering integration sources relative to each other, does not renumber or otherwise disturb any other stream's ledger.

Rollback (migrate down) walks the sources in reverse and rolls back one step of the most-recently-populated stream — integrations before core. Status (migrate status) prints each stream's ledger under its own == <name> == header.

A release whose migrations normalize data cannot be rolled back by redeploying the previous binary: the Down sections restore views, names, functions and grants, but a data normalization is one-way, so the rollback unit for a bad window is the pre-migration snapshot plus the previous binary. The previous binary does not write through the migrated schema; its direct table writes fail loudly, which is the intended fail-closed posture.

Migrations run automatically on startup, on the migration user's connection, when db-migrate-dsn is set; start closes that connection before it serves. With db-migrate-dsn left out, start runs no migration and refuses to start while the schema is behind the code. The CLI also provides migrate up, migrate down, and migrate status commands, which use db-migrate-dsn and refuse to run without it.

Creating New Migrations

# Install goose CLI tool
go install github.com/pressly/goose/v3/cmd/goose@latest

# Create a new migration in the appropriate source directory
cd internal/db/migrations       # or internal/integrations/stripe/store/migrations,
                                 # internal/integrations/fedwiki/store/migrations
goose create your_migration_name sql

The change that adds a column holding personal data also adds the column to the personal list in scripts/db-reader-role.sql, and every new column needs a mark in internal/migrate/testdata/console-reader-columns.txt, or make test fails (Keeping the list current).

A migration's number is settled when its change lands, not when it is written. internal/db/migrations.go calls goose.Up without allowing missing versions, so goose refuses to migrate a database whose ledger holds a higher version than a file it has not applied. When two changes in flight each add a migration to one stream, the one that lands second takes the next free numbers at landing. make test does not show the clash, because it resets the test databases; a deployment or a development database that applied the other change first does.

Always Use StatementBegin / StatementEnd

Every migration with more than one SQL statement must wrap both Up and Down sections in -- +goose StatementBegin / -- +goose StatementEnd. Without these directives, goose v3 splits statements and runs them individually, which can silently skip failed statements while still marking the migration as applied.

Migrations Run With search_path=public

db.ConnectMigration opens the migration connection, a pool of its own that start migrates and grants on and that migrate up|down|status use. It opens with the startup parameters search_path=public and role=none, replacing any search_path or role the connection string carries (the string is parsed once, and a key matches whatever its case; a -c switch in options loses to the startup parameter because Postgres applies options first), and it does not include the application's custom search_path (core,public). The application's pool, opened by db.Connect on db-dsn, uses core,public and role=none.

Keeping core,public off the migration connection is intentional. A pgx/PostgreSQL 18 interaction causes multi-statement DDL sent via the simple query protocol to silently lose tables when a non-default schema appears in the connection-level search_path. The config commands open db-dsn with db.ConnectPlain, without the application's search_path and without the startup parameters, because an operator runs them by hand and they take no part in the boot. public is also where goose's ledgers live: with search_path=public, goose creates every goose_db_version_<stream> table there, and db.PendingMigrations reads each as public.goose_db_version_<stream> by qualified name. A startup parameter outranks ALTER DATABASE ... SET and ALTER ROLE ... SET, so a search_path or role setting on the database or the login cannot move the ledgers or change who runs the DDL. After migrations complete, the application connection uses the full search_path as normal.

-- +goose Up
-- +goose StatementBegin

CREATE TABLE core.example (...);
CREATE INDEX idx_example ON core.example(...);
GRANT ALL ON ALL TABLES IN SCHEMA core TO core_owner;

-- +goose StatementEnd

-- +goose Down
-- +goose StatementBegin

DROP TABLE IF EXISTS core.example;

-- +goose StatementEnd

This ensures PostgreSQL receives the entire block as a single unit, so failures are atomic — either all statements succeed or none are applied.

SQL run with the migration user's privileges

The start-up check, the boot grant and the pending check (internal/db/runtime_user.go, internal/db/migrations.go) run on the migration connection or print a statement that the operator runs as the migration user, in a database where another role may have created objects. Their SQL names every function and catalog relation with its schema (pg_catalog.quote_ident, pg_catalog.pg_roles) and every goose ledger as public.goose_db_version_<stream>, and each of their transactions begins with SET LOCAL search_path = pg_catalog, pg_temp. Postgres prefers a function or operator whose argument types match exactly, in any schema on the path, to a catalog one that needs a cast, and it looks for a table in pg_temp before the path, so a role that can create objects would otherwise run its own code in the migration user's session or write the statement the operator runs; the pin also covers operators, which a query writes without a schema. TestCheckQueriesQualifyEveryName fails for an unqualified function or catalog relation in a constant of those two files whose name begins query.

Roles

Each schema has an owner/writer/reader role triple: core_owner / core_writer / core_reader, stripe_owner / stripe_writer / stripe_reader, fedwiki_owner / fedwiki_writer / fedwiki_reader. Each triple is created by its owning stream's own baseline migration, not by any other stream: the core baseline (internal/db/migrations/00001_init.sql) creates only the core_* triple, and each integration's baseline (internal/integrations/<key>/store/migrations/00001_init.sql) creates its own {key}_owner / {key}_writer / {key}_reader triple. Because integration tables FK into core, each integration's baseline also grants core_reader to its own {key}_writer; this is safe because core always migrates first (see internal/migrate.Sources()), so core_reader already exists by the time an integration's baseline runs.

The runtime user reaches the writers through console_runtime, a NOLOGIN group role that core migration 00029 creates. Each stream's group-role migration grants console_runtime that stream's {key}_writer and SELECT on that stream's goose ledger, so the role is a member of five writers and reads five ledgers. No baseline names a login, and each group-role migration revokes its writer from member_console on a database that an earlier release migrated. The writers hold at most SELECT, INSERT, UPDATE and DELETE on a table, and no other privilege: each stream's writer_dml_only migration revokes TRUNCATE, REFERENCES, TRIGGER and MAINTAIN, and TestRuntimeRolePrivileges fails when console_runtime holds one of them on any relation. Postgres runs a trigger with the privileges of the user whose statement fired it, so TRIGGER on a table that a SECURITY DEFINER function writes would let a writer run its own code as core_owner, and TRUNCATE and MAINTAIN empty or lock a table without the privileges the enclosures revoke. Counters are identity columns: Postgres does not check the privileges of an identity column's sequence when an insert draws from it, so no stream grants a sequence, whereas a serial column's sequence would need USAGE granted to its writer, and the catalog test reads no sequence. The console grants console_runtime to the runtime user at start. Owner roles are granted to no login. Each schema's authoring stream is solely responsible for its own roles, grants and its group-role migration — there is no privilege wiring performed from outside a schema's own migration stream, apart from the console's grant of console_runtime to the runtime user at start and console_reader, below.

The objects the migrations create belong to the user that runs the migrations, with two exceptions. core_owner owns the SECURITY DEFINER functions of the core schema, and it owns core.webhook_events with its partitions, because creating a partition requires the owner of the parent table and core.ensure_webhook_event_partitions, which console_runtime alone is granted, runs as core_owner. The runtime user owns no object in the schemas and does not own the database.

The console_reader role

scripts/db-reader-role.sql creates and maintains console_reader, a login role for looking into a running deployment's database. It is not one of the per-schema *_reader roles above, which read every column and exist for the console's own grants: console_reader is a member of no role and reads only the columns the script grants it. The script grants columns in the console's schemas only, which it finds by their migration ledgers: a schema is the console's when a table named goose_db_version_<schema> exists. Another application's schema, or an extension's, in the same database stays unreadable.

It reads ids, statuses, amounts, timestamps, catalog text, error messages and public DNS names, including a domain claim's verification token, which the member publishes in DNS. It does not read the columns the script's personal list withholds: names and email addresses (the operator's too, recorded on settings changes), OIDC subjects, login IP addresses, the names of organizations, workspaces and billing accounts, card last four digits and expiry dates, grant descriptions and revocation reasons (and the copy of a description on provision transitions), notes on entitlement-set changes and on name rules, workspace descriptions, and every payload column. The descriptions of products, entitlement sets and their rules, plan ladders, resource pools, roles and organization types stay readable, as do invoice line descriptions and the error_message and last_error columns.

That withholds direct identifiers and the text written about members, not every trace of a person: an id still links one person's payments and times, and a site domain that spells a member's name leads to that member's organization. Two readable settings can name a person. A Discourse API username stored as an override in core.integration_config_overrides is a forum admin's username unless it is system. The Stripe business name and statement descriptor, stored together in the stripe.public_details row of core.instance_settings, are the account holder's legal name when an individual's Stripe account has not set them (stripe.md); members see both at checkout and on card statements anyway. A column grant cannot hide one row, so withholding either would hide every override or every setting.

Each run revokes every privilege the role holds in the database, then grants SELECT column by column and USAGE on the console's schemas; the role holds nothing else of its own. Like every role, it keeps what PUBLIC holds, such as TEMP on the database. Its sessions start read-only, stop a statement after 15 seconds, and are closed when they stay idle inside a transaction for a minute: an open transaction keeps its locks, and the next migration's ALTER TABLE would wait behind them, holding up every query queued after it. A session can change all three with SET, so they guard against mistakes; the grants are what keep the role from writing. A table with a withheld column refuses SELECT *, and refuses the column in a WHERE clause as well. Partitions get no grant of their own; the role reads them through their parent table.

A view reads its tables with its owner's rights, so a view that reads a withheld column would show it to the role, whether the view selects the column under the same name, selects it under another name, or only filters on it. Postgres records which columns a view reads, not which of the view's columns each one feeds, so the script withholds such a view whole. It also withholds whole a view that reads an entire row of a table with a withheld column, and a view that reads a withheld view.

The migrations do not create this role. Its grants span every schema, which no single migration stream owns; they are recomputed column by column from the live schema on each run; and a deployment decides whether it wants a login role at all.

Running the script

Run it as the migration user, the login that owns the console's schemas and that db-migrate-dsn names; the script also needs CREATEROLE. With the official postgres image the migration user is the POSTGRES_USER, a superuser. On a managed database, run it as a user that has CREATEROLE and owns the console's tables. The runtime user (db-dsn) owns nothing and cannot run it:

psql -d <database> -v ON_ERROR_STOP=1 -f scripts/db-reader-role.sql

Run it again after every deploy that applies a migration: a table or column a migration adds stays unreadable to the role until the next run. A run prints how many columns the role can read and how many it withholds, and names each view it withholds whole. The script is a single statement, so a run that refuses or fails leaves the role as the previous run left it, in psql or any other client; ON_ERROR_STOP makes psql exit with a non-zero status.

A run refuses when the role has SUPERUSER, CREATEROLE, CREATEDB, REPLICATION or BYPASSRLS, when it is a member of any role, when it owns any object, when it could execute a SECURITY DEFINER function, or when a privilege survives the revoke: one another role granted that the running user cannot revoke, or default privileges that name the role. The migrations revoke EXECUTE on SECURITY DEFINER functions from PUBLIC; a database restored with pg_dump --no-privileges has lost those revokes. After granting, a run fails if the role can read a withheld column or write to a table, which a GRANT ... TO PUBLIC would cause.

Logging in

The role has no password. The official postgres image keeps initdb's default pg_hba.conf, which trusts the Unix socket and the loopback address and asks every other client for a password, so the role logs in from inside the database container only:

docker exec -it <postgres container> psql -U console_reader -d <database>

POSTGRES_HOST_AUTH_METHOD=trust adds a pg_hba.conf line that admits every role, the superuser included, without a password, to anyone who can reach the database port. console_reader adds nothing to that exposure, and a password on it would not be asked for; the remedy is scram-sha-256 in that line. Distribution packages authenticate the socket by peer, which admits psql -U console_reader only from an operating-system user of that name or through a pg_ident.conf map. On a managed database the migration user may lack CREATEROLE, and the run fails; run it as a user that has CREATEROLE and owns the console's tables, then give the role a password and allow it in the provider's network rules.

Keeping the list current

The next run grants every column that no pattern in the personal list matches. The change that adds a column holding personal data extends the list, and so does a change whose code starts writing personal data into an existing column. Personal data here is a name, an email address, a phone number, a postal address, an IP address, an identity provider's subject or username, text a member writes or an operator writes about a member, a raw provider payload, or a copy of any of these; a column holding a secret, such as an API key, is withheld too. A pattern names a partitioned table's columns under the parent table's name, and a view that reads a withheld column needs no pattern of its own.

TestConsoleReaderColumns in internal/migrate runs the script against the migrated test database, inside a transaction it rolls back, and compares the role's SELECT privilege on every column of every table and view in the console's schemas with internal/migrate/testdata/console-reader-columns.txt, which marks each column read or withhold (partitions are not listed). make test fails on a column the file does not list, on a listed column that no longer exists, and on a column whose privilege contradicts its mark, as happens when a pattern is dropped from the list. It also fails when, after the run, the role is a member of a role, can execute a SECURITY DEFINER function, lacks USAGE on one of the console's schemas, or holds a privilege outside them. With TEST_DATABASE_URL from test/.env set,

go test ./internal/migrate/ -run TestConsoleReaderColumns -count=1 -update

rewrites the file from the database. A column that holds personal data goes in the personal list first, so the rewrite marks it withhold; the diff shows each new column's mark for review. The test cannot catch code that starts writing personal data into an existing column: that column keeps its read mark, and the role still reads it. The change that adds such a writer extends the list itself.

sqlc Code Generation

Each Go package has its own sqlc.yaml that generates type-safe Go code from SQL queries.

# Regenerate sqlc code after schema or query changes
make sqlc-generate

make sqlc-generate runs sqlc generate in every package that has a sqlc.yaml, then restores the SPDX license headers that sqlc drops from its output. Running sqlc generate in one directory by hand leaves that package's generated files without their headers.

Deleting a query is deleting its block and regenerating; make lint fails on a query nothing calls.

A query that reads from a set-returning function names the function's columns with an alias list and casts each one (FROM core.lock_plan_act_obligation(...) AS l(act_id, ...), then l.act_id::uuid AS act_id). sqlc cannot read a function's output columns, so it refuses a column the alias list does not name and types a column without a cast interface{}.

internal/db/sqlc_schemas.sql

This file is not a goose migration — it is a sqlc-only helper that declares all PostgreSQL schemas (CREATE SCHEMA IF NOT EXISTS ...) so that sqlc can resolve schema-qualified table names (e.g., core.accounts) across packages. It declares core, stripe, and fedwiki, and should be listed in every package's sqlc.yaml schema paths alongside the migration directories it needs for FK resolution.

Naming Convention

Table names must not repeat the schema name:

• Use core.accounts, not core.billing_accounts
• Use core.persons, not core.identity_persons

Generated Go type names carry no schema prefix — the package itself provides the namespace:

• billing.Account, not billing.CoreAccount
• identity.Person, not identity.CorePerson

This follows Go's own convention: http.Request not http.HttpRequest.

Standard sqlc.yaml Template

Every package follows this pattern:

schema:
  - "../db/sqlc_schemas.sql"   # always first — declares all schemas for cross-package resolution
  - "../db/migrations/"        # the core baseline (most packages read/write core tables)
  # add the provider's own migrations only for stripe/fedwiki packages
  # e.g. - "migrations/"       # (internal/integrations/stripe/store, internal/integrations/fedwiki/store only)
queries: "queries/"

An integration's sqlc.yaml sits two directories deeper than a typical package's — internal/integrations/<key>/store/sqlc.yaml, with store/ nested under the integration's own tree root — so its relative paths climb two extra levels: ../../../db/migrations/ and ../../../db/sqlc_schemas.sql, alongside its own migrations/ for the provider's tables.

rename: Block Placement

The rename: block goes inside gen.go:, after emit_empty_slices and before overrides:. Keys use the {schema}_{singular_table} format — with a single domain schema, most keys are core_* regardless of which Go package owns the query:

gen:
  go:
    package: "billing"
    emit_empty_slices: true
    rename:
      core_account: Account    # core.accounts → Account
      core_price: Price
    overrides:
      - db_type: "uuid"
        go_type: "string"

Rename maps are pinned per package so that generated type names do not churn when the underlying schema is shared — e.g. internal/billing/sqlc.yaml and internal/identity/sqlc.yaml both see core.persons but only one may need to generate a Person type, depending on which package's queries touch it.