member-console/docs/database-management.md

title, audience, summary

title	audience	summary
Database Management	developer admin	goose migrations, sqlc code generation, and the core + per-integration schema and migration-stream conventions.

Database Management

This project uses pressly/goose for database migrations and sqlc for type-safe SQL code generation. Requires PostgreSQL 18+ (for native uuidv7()).

Schema layout

The database has three PostgreSQL schemas:

• core — every domain table (identity, organization, billing, entitlements, and the provider registry that integrations register against). This is the single source of business truth; there is no per-Go-package schema split. internal/db/migrations/ owns this stream.
• stripe — the Stripe provider's own mapping tables (customers, subscriptions, invoices, etc.), FK'd into core. internal/integrations/stripe/store/migrations/ owns this stream.
• fedwiki — the FedWiki provider's own tables (sites, site swap policy), FK'd into core. internal/integrations/fedwiki/store/migrations/ owns this stream.

The Go package layout does not mirror the schema layout: internal/identity, internal/organization, internal/billing, internal/entitlements, and internal/integration are separate packages that all read and write tables in the single core schema. Schema boundaries track core domain vs. provider integration, not Go package boundaries — see the schema-consolidation change record under openspec/changes/archive/ for the rationale.

Per-Source Migrations

Each schema is owned by exactly one embedded migrations/ directory:

internal/db/migrations/                          # core schema (identity, organization, billing, entitlements, provider registry)
internal/integrations/fedwiki/store/migrations/   # fedwiki schema (sites, site_swap_policy)
internal/integrations/stripe/store/migrations/    # stripe schema (mapping tables)

Each source runs against its own goose ledger table, goose_db_version_<name> (goose_db_version_core, goose_db_version_fedwiki, goose_db_version_stripe), and keeps its native 00001… file numbering. There is no shared goose_db_version table and no global version namespace — a source's version numbers are independent of its position in the dependency-ordered source list returned by internal/migrate.Sources(). internal/db/migrations.go loops the sources in order, pointing goose at each source's embedded FS and ledger table in turn.

Source order matters only weakly: core must run first, because the FK DAG is one-directional — integration schemas reference core, never the reverse and never each other — so core's tables must exist before any integration's migrations apply. Beyond that, integration sources are mutually order-independent. Appending a new integration source, or reordering integration sources relative to each other, does not renumber or otherwise disturb any other stream's ledger.

Rollback (migrate down) walks the sources in reverse and rolls back one step of the most-recently-populated stream — integrations before core. Status (migrate status) prints each stream's ledger under its own == <name> == header.

A release whose migrations normalize data cannot be rolled back by redeploying the previous binary: the Down sections restore views, names, functions and grants, but a data normalization is one-way, so the rollback unit for a bad window is the pre-migration snapshot plus the previous binary. The previous binary does not write through the migrated schema; its direct table writes fail loudly, which is the intended fail-closed posture.

Migrations run automatically on startup. The CLI also provides migrate up, migrate down, and migrate status commands.

Creating New Migrations

# Install goose CLI tool
go install github.com/pressly/goose/v3/cmd/goose@latest

# Create a new migration in the appropriate source directory
cd internal/db/migrations       # or internal/integrations/stripe/store/migrations,
                                 # internal/integrations/fedwiki/store/migrations
goose create your_migration_name sql

The change that adds a column holding personal data also adds the column to the personal list in scripts/db-reader-role.sql, and every new column needs a mark in internal/migrate/testdata/console-reader-columns.txt, or make test fails (Keeping the list current).

Always Use StatementBegin / StatementEnd

Every migration with more than one SQL statement must wrap both Up and Down sections in -- +goose StatementBegin / -- +goose StatementEnd. Without these directives, goose v3 splits statements and runs them individually, which can silently skip failed statements while still marking the migration as applied.

Migrations Run Without search_path

The ConnectAndMigrate function runs migrations on a separate database connection that does not include the custom search_path (core,public). This is intentional — a pgx/PostgreSQL 18 interaction causes multi-statement DDL sent via the simple query protocol to silently lose tables when a non-default schema appears in the connection-level search_path. After migrations complete, the application connection uses the full search_path as normal.

-- +goose Up
-- +goose StatementBegin

CREATE TABLE core.example (...);
CREATE INDEX idx_example ON core.example(...);
GRANT ALL ON ALL TABLES IN SCHEMA core TO core_owner;

-- +goose StatementEnd

-- +goose Down
-- +goose StatementBegin

DROP TABLE IF EXISTS core.example;

-- +goose StatementEnd

This ensures PostgreSQL receives the entire block as a single unit, so failures are atomic — either all statements succeed or none are applied.

Roles

Each schema has an owner/writer/reader role triple: core_owner / core_writer / core_reader, stripe_owner / stripe_writer / stripe_reader, fedwiki_owner / fedwiki_writer / fedwiki_reader. Each triple is created by its owning stream's own baseline migration, not by any other stream: the core baseline (internal/db/migrations/00001_init.sql) creates only the core_* triple, and each integration's baseline (internal/integrations/<key>/store/migrations/00001_init.sql) creates its own {key}_owner / {key}_writer / {key}_reader triple. Because integration tables FK into core, each integration's baseline also grants core_reader to its own {key}_writer; this is safe because core always migrates first (see internal/migrate.Sources()), so core_reader already exists by the time an integration's baseline runs. The member_console login role (the app's DSN user) ends up a member of core_writer and every {key}_writer — membership in core_writer is granted by the core baseline, and membership in each {key}_writer is granted by that integration's own baseline. Owner roles are reserved for running migrations. Each schema's authoring stream is solely responsible for its own roles, grants, and member_console membership — there is no privilege wiring performed from outside a schema's own migration stream. The one exception is console_reader, below.

The console_reader role

scripts/db-reader-role.sql creates and maintains console_reader, a login role for looking into a running deployment's database. It is not one of the per-schema *_reader roles above, which read every column and exist for the console's own grants: console_reader is a member of no role and reads only the columns the script grants it. The script grants columns in the console's schemas only, which it finds by their migration ledgers: a schema is the console's when a table named goose_db_version_<schema> exists. Another application's schema, or an extension's, in the same database stays unreadable.

It reads ids, statuses, amounts, timestamps, catalog text, error messages and public DNS names, including a domain claim's verification token, which the member publishes in DNS. It does not read the columns the script's personal list withholds: names and email addresses (the operator's too, recorded on settings changes), OIDC subjects, login IP addresses, the names of organizations, workspaces and billing accounts, card last four digits and expiry dates, grant descriptions and revocation reasons (and the copy of a description on provision transitions), notes on entitlement-set changes and on name rules, workspace descriptions, and every payload column. The descriptions of products, entitlement sets and their rules, plan ladders, resource pools, roles and organization types stay readable, as do invoice line descriptions and the error_message and last_error columns.

That withholds direct identifiers and the text written about members, not every trace of a person: an id still links one person's payments and times, and a site domain that spells a member's name leads to that member's organization. Two readable settings can name a person. A Discourse API username stored as an override in core.integration_config_overrides is a forum admin's username unless it is system. The Stripe business name and statement descriptor, stored together in the stripe.public_details row of core.instance_settings, are the account holder's legal name when an individual's Stripe account has not set them (stripe.md); members see both at checkout and on card statements anyway. A column grant cannot hide one row, so withholding either would hide every override or every setting.

Each run revokes every privilege the role holds in the database, then grants SELECT column by column and USAGE on the console's schemas; the role holds nothing else of its own. Like every role, it keeps what PUBLIC holds, such as TEMP on the database. Its sessions start read-only, stop a statement after 15 seconds, and are closed when they stay idle inside a transaction for a minute: an open transaction keeps its locks, and the next migration's ALTER TABLE would wait behind them, holding up every query queued after it. A session can change all three with SET, so they guard against mistakes; the grants are what keep the role from writing. A table with a withheld column refuses SELECT *, and refuses the column in a WHERE clause as well. Partitions get no grant of their own; the role reads them through their parent table.

A view reads its tables with its owner's rights, so a view that reads a withheld column would show it to the role, whether the view selects the column under the same name, selects it under another name, or only filters on it. Postgres records which columns a view reads, not which of the view's columns each one feeds, so the script withholds such a view whole. It also withholds whole a view that reads an entire row of a table with a withheld column, and a view that reads a withheld view.

The migrations do not create this role. Its grants span every schema, which no single migration stream owns; they are recomputed column by column from the live schema on each run; and a deployment decides whether it wants a login role at all.

Running the script

Run it as the role that owns the console's schemas, which also needs CREATEROLE. With the official postgres image that is the POSTGRES_USER the console connects as, a superuser:

psql -d <database> -v ON_ERROR_STOP=1 -f scripts/db-reader-role.sql

Run it again after every deploy that applies a migration: a table or column a migration adds stays unreadable to the role until the next run. A run prints how many columns the role can read and how many it withholds, and names each view it withholds whole. The script is a single statement, so a run that refuses or fails leaves the role as the previous run left it, in psql or any other client; ON_ERROR_STOP makes psql exit with a non-zero status.

A run refuses when the role has SUPERUSER, CREATEROLE, CREATEDB, REPLICATION or BYPASSRLS, when it is a member of any role, when it owns any object, when it could execute a SECURITY DEFINER function, or when a privilege survives the revoke: one another role granted that the running user cannot revoke, or default privileges that name the role. The migrations revoke EXECUTE on SECURITY DEFINER functions from PUBLIC; a database restored with pg_dump --no-privileges has lost those revokes. After granting, a run fails if the role can read a withheld column or write to a table, which a GRANT ... TO PUBLIC would cause.

Logging in

The role has no password. The official postgres image keeps initdb's default pg_hba.conf, which trusts the Unix socket and the loopback address and asks every other client for a password, so the role logs in from inside the database container only:

docker exec -it <postgres container> psql -U console_reader -d <database>

POSTGRES_HOST_AUTH_METHOD=trust adds a pg_hba.conf line that admits every role, the superuser included, without a password, to anyone who can reach the database port. console_reader adds nothing to that exposure, and a password on it would not be asked for; the remedy is scram-sha-256 in that line. Distribution packages authenticate the socket by peer, which admits psql -U console_reader only from an operating-system user of that name or through a pg_ident.conf map. On a managed database the user the console connects as may lack CREATEROLE, and the run fails; run it as a user that has CREATEROLE and owns the console's tables, then give the role a password and allow it in the provider's network rules.

Keeping the list current

The next run grants every column that no pattern in the personal list matches. The change that adds a column holding personal data extends the list, and so does a change whose code starts writing personal data into an existing column. Personal data here is a name, an email address, a phone number, a postal address, an IP address, an identity provider's subject or username, text a member writes or an operator writes about a member, a raw provider payload, or a copy of any of these; a column holding a secret, such as an API key, is withheld too. A pattern names a partitioned table's columns under the parent table's name, and a view that reads a withheld column needs no pattern of its own.

TestConsoleReaderColumns in internal/migrate runs the script against the migrated test database, inside a transaction it rolls back, and compares the role's SELECT privilege on every column of every table and view in the console's schemas with internal/migrate/testdata/console-reader-columns.txt, which marks each column read or withhold (partitions are not listed). make test fails on a column the file does not list, on a listed column that no longer exists, and on a column whose privilege contradicts its mark, as happens when a pattern is dropped from the list. It also fails when, after the run, the role is a member of a role, can execute a SECURITY DEFINER function, lacks USAGE on one of the console's schemas, or holds a privilege outside them. With TEST_DATABASE_URL from test/.env set,

go test ./internal/migrate/ -run TestConsoleReaderColumns -count=1 -update

rewrites the file from the database. A column that holds personal data goes in the personal list first, so the rewrite marks it withhold; the diff shows each new column's mark for review. The test cannot catch code that starts writing personal data into an existing column: that column keeps its read mark, and the role still reads it. The change that adds such a writer extends the list itself.

sqlc Code Generation

Each Go package has its own sqlc.yaml that generates type-safe Go code from SQL queries.

# Regenerate sqlc code after schema or query changes
cd internal/entitlements && sqlc generate

Deleting a query is deleting its block and regenerating; make lint fails on a query nothing calls.

internal/db/sqlc_schemas.sql

This file is not a goose migration — it is a sqlc-only helper that declares all PostgreSQL schemas (CREATE SCHEMA IF NOT EXISTS ...) so that sqlc can resolve schema-qualified table names (e.g., core.accounts) across packages. It declares core, stripe, and fedwiki, and should be listed in every package's sqlc.yaml schema paths alongside the migration directories it needs for FK resolution.

Naming Convention

Table names must not repeat the schema name:

• Use core.accounts, not core.billing_accounts
• Use core.persons, not core.identity_persons

Generated Go type names carry no schema prefix — the package itself provides the namespace:

• billing.Account, not billing.CoreAccount
• identity.Person, not identity.CorePerson

This follows Go's own convention: http.Request not http.HttpRequest.

Standard sqlc.yaml Template

Every package follows this pattern:

schema:
  - "../db/sqlc_schemas.sql"   # always first — declares all schemas for cross-package resolution
  - "../db/migrations/"        # the core baseline (most packages read/write core tables)
  # add the provider's own migrations only for stripe/fedwiki packages
  # e.g. - "migrations/"       # (internal/integrations/stripe/store, internal/integrations/fedwiki/store only)
queries: "queries/"

An integration's sqlc.yaml sits two directories deeper than a typical package's — internal/integrations/<key>/store/sqlc.yaml, with store/ nested under the integration's own tree root — so its relative paths climb two extra levels: ../../../db/migrations/ and ../../../db/sqlc_schemas.sql, alongside its own migrations/ for the provider's tables.

rename: Block Placement

The rename: block goes inside gen.go:, after emit_empty_slices and before overrides:. Keys use the {schema}_{singular_table} format — with a single domain schema, most keys are core_* regardless of which Go package owns the query:

gen:
  go:
    package: "billing"
    emit_empty_slices: true
    rename:
      core_account: Account    # core.accounts → Account
      core_price: Price
    overrides:
      - db_type: "uuid"
        go_type: "string"

Rename maps are pinned per package so that generated type names do not churn when the underlying schema is shared — e.g. internal/billing/sqlc.yaml and internal/identity/sqlc.yaml both see core.persons but only one may need to generate a Person type, depending on which package's queries touch it.