Run the console on a runtime user and migrate on a second connection string
Fourth section of #23. db-dsn now names the console's runtime user, and the new db-migrate-dsn (or db-migrate-dsn-file) names the user that runs migrations. start opens both, migrates on the migration connection, grants console_runtime to the runtime user there (WITH INHERIT TRUE, the name quoted by the database), closes it, and serves on the runtime pool alone. With db-migrate-dsn left out, start runs no migration and refuses when the schema is behind its code, which is how a host runs migrations as a separate step; migrate up, down and status use the migration string and refuse without it, with no fallback to db-dsn. Both pools are opened from one pgx.ParseConfig: the migration pool's search_path is public and both pools' role is none, set in the startup parameters after every key of either name is removed case-insensitively, so neither a setting the database or role carries nor a key the string repeats in another case moves a session off its login or its path. PendingMigrations reads each stream's ledger by qualified name inside a transaction pinned to search_path pg_catalog, pg_temp. The test stack runs on two users: postgres-seed creates member_console_runtime in new and existing volumes, bootstrap-stack.sh writes MC_DB_DSN as that user and MC_DB_MIGRATE_DSN as member_console, and the four scripts refuse a .env without the migration string, with the command that regenerates it. Scripts and fixtures that need a superuser on the application database (the reset, the screens' timestamp pinning) use the migration string. This worktree's stack was regenerated with that command and the console booted as member_console_runtime: its sessions were the runtime user's only, no migration-user session stayed open, and the runtime user could not insert into core.pool_provisions. Full suite on reset databases: 4277 pass, 15 skip, none failed. The walkthroughs' names, results and skip messages match the previous section's run. make screens lists three changed screens, all data: regenerated ids on operator-person-detail and operator-billing-invoice-detail (the demo snapshot was rebuilt) and a domain's age on operator-domains (294d to 295d); the baseline was not accepted. make lint passes. Deliberate breaks, scripts/mutation-check -side new, all CAUGHT (36): dbRuntime runs the migrations on the runtime pool dbRuntime skips the boot grant dbRuntime leaves the migration pool open dbRuntime skips the pending check when db-migrate-dsn is unset dbRuntime migrates on db-dsn when db-migrate-dsn is unset GrantRuntimeRole without WITH INHERIT TRUE GrantRuntimeRole with %s in place of %I PendingMigrations takes a missing ledger as current PendingMigrations resolves the ledger without the public. qualifier the migration pool opened without search_path=public the migration pool opened without role=none the runtime pool opened without role=none ConnectMigration keeps the search_path and role the string carries dbMigration falls back to db-dsn ValidateStart accepts a malformed db-migrate-dsn db-migrate-dsn-file is not a database secret pair db-migrate-dsn-file is not a database secret pair (the migrate commands read db-dsn-file only) the bootstrap guard removed reset-test-db.sh guard removed seed-demo.sh guard removed reset-app-db.sh guard after its first psql startup parameters match their keys case-sensitively the migration pool opened without role=none (keys and options in the string) the runtime pool opened without role=none (keys and options in the string) the runtime pool replaces the search_path the string carries the string's spaces are written as plus signs ValidateStart reads db-migrate-dsn as set whatever it holds PendingMigrations lists only the versions the ledger shows when it is missing migrationDSN keeps whitespace migrate up accepts a whitespace-only db-migrate-dsn runtimeDSN accepts a whitespace-only db-dsn openDB runs the migrating boot for an unknown mode bootstrap-stack.sh trusts the exported MC_DB_MIGRATE_DSN reset-test-db.sh trusts the exported MC_DB_MIGRATE_DSN reset-app-db.sh trusts the exported MC_DB_MIGRATE_DSN seed-demo.sh trusts the exported MC_DB_MIGRATE_DSN
This commit is contained in:
+1
-1
@@ -140,7 +140,7 @@ func connectSettings() (*integration.SettingsService, func(), []config.ConfigKey
|
||||
if err := resolveDatabaseSecretFiles(); err != nil {
|
||||
return nil, nil, nil, err
|
||||
}
|
||||
database, closeDB, err := openDB(context.Background(), logger, dbPlain)
|
||||
database, closeDB, err := openDB(context.Background(), logger, dbConfig)
|
||||
if err != nil {
|
||||
return nil, nil, nil, fmt.Errorf("connecting to the database: %w", err)
|
||||
}
|
||||
|
||||
@@ -79,7 +79,7 @@ func runValidateConfig(ctx context.Context, specs []config.ConfigKey) (ok bool,
|
||||
if logging.AppLogger != nil {
|
||||
logger = logging.AppLogger
|
||||
}
|
||||
database, closeDB, err := openDB(ctx, logger, dbPlain)
|
||||
database, closeDB, err := openDB(ctx, logger, dbConfig)
|
||||
if err != nil {
|
||||
return false, fmt.Sprintf("config validate: connecting to the database: %v", err)
|
||||
}
|
||||
|
||||
+152
-20
@@ -7,43 +7,72 @@ import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/db"
|
||||
"github.com/spf13/viper"
|
||||
)
|
||||
|
||||
// dbMode says what openDB does after it connects.
|
||||
// dbMode says which connection openDB opens and what it runs before it
|
||||
// returns.
|
||||
type dbMode int
|
||||
|
||||
const (
|
||||
// dbPlain connects and leaves the schema as it is.
|
||||
dbPlain dbMode = iota
|
||||
// dbMigrated connects and applies every pending migration first.
|
||||
dbMigrated
|
||||
// dbRuntime opens the pool the console serves from, on db-dsn.
|
||||
dbRuntime dbMode = iota
|
||||
// dbMigration opens the migration pool on db-migrate-dsn and nothing else.
|
||||
dbMigration
|
||||
// dbConfig opens db-dsn plain and leaves the schema as it is.
|
||||
dbConfig
|
||||
)
|
||||
|
||||
// openDB opens the database the db-dsn setting names, in the given mode. The
|
||||
// returned closer is for the caller to defer; it logs a failed Close at Warn,
|
||||
// because by then the command has nothing left to do with the error.
|
||||
// openDB opens the database in the given mode. Each command uses one:
|
||||
//
|
||||
// - start and seed-demo open dbRuntime: they serve from, or write through,
|
||||
// the runtime user. When db-migrate-dsn is set the open migrates and
|
||||
// grants the runtime group role on that connection, then closes it; when
|
||||
// it is not set the open runs no migration and refuses a schema that is
|
||||
// behind the code.
|
||||
// - migrate up, down and status open dbMigration: they run DDL, which the
|
||||
// runtime user cannot, and they refuse to fall back to db-dsn.
|
||||
// - config list, set, clear and validate open dbConfig: they are tools an
|
||||
// operator runs by hand on the runtime user's string, read and write
|
||||
// core.integration_config_overrides, and take no part in the boot.
|
||||
//
|
||||
// The returned closer is for the caller to defer; it logs a failed Close at
|
||||
// Warn, because by then the command has nothing left to do with the error.
|
||||
//
|
||||
// It does not resolve the -file keys: a second resolution after
|
||||
// resolveSecretFiles would report the pair as ambiguous, so each command
|
||||
// resolves once before it calls openDB. It refuses an empty db-dsn, so a
|
||||
// command that forgot to resolve fails here and pgx never connects with
|
||||
// libpq's defaults.
|
||||
// resolves once before it calls openDB. It refuses a mode whose connection
|
||||
// string is empty, so a command that forgot to resolve fails here and pgx
|
||||
// never connects with libpq's defaults.
|
||||
func openDB(ctx context.Context, logger *slog.Logger, mode dbMode) (*sql.DB, func(), error) {
|
||||
dsn := viper.GetString("db-dsn")
|
||||
if dsn == "" {
|
||||
return nil, nil, errors.New("db-dsn is not set (MC_DB_DSN or MC_DB_DSN_FILE)")
|
||||
}
|
||||
dbConfig := db.DefaultDBConfig(dsn)
|
||||
var database *sql.DB
|
||||
var err error
|
||||
if mode == dbMigrated {
|
||||
database, err = db.ConnectAndMigrate(ctx, logger, dbConfig, migrationSources())
|
||||
} else {
|
||||
database, err = db.ConnectPlain(ctx, logger, dbConfig)
|
||||
switch mode {
|
||||
case dbMigration:
|
||||
dsn := migrationDSN()
|
||||
if dsn == "" {
|
||||
return nil, nil, errors.New("member-console migrate needs db-migrate-dsn (MC_DB_MIGRATE_DSN or MC_DB_MIGRATE_DSN_FILE)")
|
||||
}
|
||||
database, err = db.ConnectMigration(ctx, logger, db.DefaultDBConfig(dsn))
|
||||
case dbConfig:
|
||||
dsn, derr := runtimeDSN()
|
||||
if derr != nil {
|
||||
return nil, nil, derr
|
||||
}
|
||||
database, err = db.ConnectPlain(ctx, logger, db.DefaultDBConfig(dsn))
|
||||
case dbRuntime:
|
||||
dsn, derr := runtimeDSN()
|
||||
if derr != nil {
|
||||
return nil, nil, derr
|
||||
}
|
||||
database, err = openRuntimeDatabase(ctx, logger, dsn, migrationDSN())
|
||||
default:
|
||||
return nil, nil, fmt.Errorf("unknown database mode %d", mode)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
@@ -54,3 +83,106 @@ func openDB(ctx context.Context, logger *slog.Logger, mode dbMode) (*sql.DB, fun
|
||||
}
|
||||
}, nil
|
||||
}
|
||||
|
||||
// migrationDSN returns db-migrate-dsn without surrounding whitespace, as
|
||||
// ValidateStart reads it: a value of only whitespace is unset, never a
|
||||
// connection string for pgx to fill with libpq's defaults.
|
||||
func migrationDSN() string {
|
||||
return strings.TrimSpace(viper.GetString("db-migrate-dsn"))
|
||||
}
|
||||
|
||||
// runtimeDSN returns db-dsn, or the refusal for a command that has none. A
|
||||
// value of only whitespace is none.
|
||||
func runtimeDSN() (string, error) {
|
||||
dsn := strings.TrimSpace(viper.GetString("db-dsn"))
|
||||
if dsn == "" {
|
||||
return "", errors.New("db-dsn is not set (MC_DB_DSN or MC_DB_DSN_FILE)")
|
||||
}
|
||||
return dsn, nil
|
||||
}
|
||||
|
||||
// logOpenFailure logs why start could not open the database: a refusal is
|
||||
// "refused to start" with the sentence as its error, anything else is a
|
||||
// failure to initialize.
|
||||
func logOpenFailure(logger *slog.Logger, err error) {
|
||||
var refusal *db.RuntimeUserRefusal
|
||||
if errors.As(err, &refusal) {
|
||||
logger.Error("refused to start", slog.Any("error", err))
|
||||
return
|
||||
}
|
||||
logger.Error("failed to initialize database", slog.Any("error", err))
|
||||
}
|
||||
|
||||
// openRuntimeDatabase is the boot sequence of dbRuntime (design, "The boot
|
||||
// sequence"): it opens the runtime pool, and with migrateDSN set it opens the
|
||||
// migration pool, runs every stream's migrations there, grants the runtime
|
||||
// group role to the runtime user there, and closes it before it returns;
|
||||
// then it refuses a schema that is behind the code. The runtime user runs
|
||||
// no DDL: a pending migration is the migration connection's, or the
|
||||
// operator's `migrate up`.
|
||||
func openRuntimeDatabase(ctx context.Context, logger *slog.Logger, dsn, migrateDSN string) (*sql.DB, error) {
|
||||
runtime, err := db.Connect(ctx, logger, db.DefaultDBConfig(dsn))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if migrateDSN == "" {
|
||||
logger.Info("migrations not run: db-migrate-dsn is not set")
|
||||
} else if err := migrateAndGrant(ctx, logger, runtime, migrateDSN); err != nil {
|
||||
return nil, errors.Join(err, runtime.Close())
|
||||
}
|
||||
if err := refuseBehindSchema(ctx, runtime); err != nil {
|
||||
return nil, errors.Join(err, runtime.Close())
|
||||
}
|
||||
return runtime, nil
|
||||
}
|
||||
|
||||
// migrateAndGrant runs the migrations and the boot grant on the migration
|
||||
// connection, which it closes before it returns, and logs that the
|
||||
// migrations were applied.
|
||||
func migrateAndGrant(ctx context.Context, logger *slog.Logger, runtime *sql.DB, migrateDSN string) error {
|
||||
migration, err := db.ConnectMigration(ctx, logger, db.DefaultDBConfig(migrateDSN))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := runMigrationsAndGrant(ctx, runtime, migration); err != nil {
|
||||
return errors.Join(err, migration.Close())
|
||||
}
|
||||
if err := migration.Close(); err != nil {
|
||||
logger.Warn("closing the migration connection", slog.Any("error", err))
|
||||
}
|
||||
logger.Info("database migrations applied")
|
||||
return nil
|
||||
}
|
||||
|
||||
// runMigrationsAndGrant runs every stream's migrations on migration, then
|
||||
// grants the runtime group role to the user runtime logs in as.
|
||||
func runMigrationsAndGrant(ctx context.Context, runtime, migration *sql.DB) error {
|
||||
if err := db.RunMigrations(migration, migrationSources()); err != nil {
|
||||
return fmt.Errorf("failed to run migrations: %w", err)
|
||||
}
|
||||
user, err := db.CurrentUser(ctx, runtime)
|
||||
if err != nil {
|
||||
return &db.RuntimeUserRefusal{Message: fmt.Sprintf("could not check the db-dsn user: %v", err)}
|
||||
}
|
||||
err = db.InTx(ctx, migration, db.Plain, func(tx *sql.Tx) error {
|
||||
return db.GrantRuntimeRole(ctx, tx, user)
|
||||
})
|
||||
if err != nil {
|
||||
return &db.RuntimeUserRefusal{Message: fmt.Sprintf(
|
||||
"could not grant %s to %s on the db-migrate-dsn connection: %v", db.RuntimeGroupRole, user, err)}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// refuseBehindSchema reads the ledgers as the runtime user and refuses a
|
||||
// schema that is behind this release.
|
||||
func refuseBehindSchema(ctx context.Context, runtime *sql.DB) error {
|
||||
pending, err := db.PendingMigrations(ctx, runtime, migrationSources())
|
||||
if err != nil {
|
||||
return &db.RuntimeUserRefusal{Message: fmt.Sprintf("could not check the db-dsn user: %v", err)}
|
||||
}
|
||||
if len(pending) > 0 {
|
||||
return db.SchemaBehindRefusal(pending)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,286 @@
|
||||
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
|
||||
package cmd
|
||||
|
||||
// Tests of the runtime database mode (openspec change runtime-database-user,
|
||||
// design D1, D8 and D11, task 4.8). Every row works on a scratch database and
|
||||
// on login roles the test creates with a password and drops in its cleanup:
|
||||
// the migration user is the superuser TEST_DATABASE_URL names, the runtime
|
||||
// user is a login that owns nothing.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/viper"
|
||||
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/db"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/migrate"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest"
|
||||
)
|
||||
|
||||
const runtimeLoginPassword = "runtime-login-password"
|
||||
|
||||
var migrationFileVersion = regexp.MustCompile(`^(\d+)_`)
|
||||
|
||||
// highestVersions maps each stream to the highest version it embeds.
|
||||
func highestVersions(t *testing.T) map[string]int64 {
|
||||
t.Helper()
|
||||
out := map[string]int64{}
|
||||
for _, src := range migrate.Sources() {
|
||||
entries, err := fs.ReadDir(src.Migrations, src.Dir)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s migrations: %v", src.Name, err)
|
||||
}
|
||||
for _, e := range entries {
|
||||
if m := migrationFileVersion.FindStringSubmatch(e.Name()); m != nil {
|
||||
v, err := strconv.ParseInt(m[1], 10, 64)
|
||||
if err != nil {
|
||||
t.Fatalf("migration %s: %v", e.Name(), err)
|
||||
}
|
||||
out[src.Name] = max(out[src.Name], v)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// bootBox is one scratch database with its two logins' addresses.
|
||||
type bootBox struct {
|
||||
t *testing.T
|
||||
scratch string // the migration user's address: the superuser
|
||||
name string // the scratch database's name
|
||||
login string // the runtime login
|
||||
runtime string // the runtime login's address
|
||||
admin *sql.DB
|
||||
}
|
||||
|
||||
// newBootBox creates a scratch database and a runtime login that owns nothing.
|
||||
func newBootBox(t *testing.T) *bootBox {
|
||||
t.Helper()
|
||||
scratch := dbtest.Scratch(t)
|
||||
u, err := url.Parse(scratch)
|
||||
if err != nil {
|
||||
t.Fatalf("parse the scratch address: %v", err)
|
||||
}
|
||||
box := &bootBox{t: t, scratch: scratch, name: strings.TrimPrefix(u.Path, "/"), login: dbtest.Name("mc_rt_"),
|
||||
admin: dbtest.Connect(t, dbtest.DSN(t))}
|
||||
dbtest.On(t, box.admin).Exec(fmt.Sprintf("CREATE ROLE %s LOGIN PASSWORD '%s'", box.login, runtimeLoginPassword))
|
||||
t.Cleanup(func() {
|
||||
if _, err := box.admin.ExecContext(context.Background(), "DROP ROLE IF EXISTS "+box.login); err != nil {
|
||||
t.Errorf("drop role %s: %v", box.login, err)
|
||||
}
|
||||
})
|
||||
u.User = url.UserPassword(box.login, runtimeLoginPassword)
|
||||
box.runtime = u.String()
|
||||
return box
|
||||
}
|
||||
|
||||
// configure points the settings at the box: db-dsn always, db-migrate-dsn when
|
||||
// withMigration is true.
|
||||
func (b *bootBox) configure(withMigration bool) {
|
||||
b.t.Helper()
|
||||
viper.Reset()
|
||||
b.t.Cleanup(viper.Reset)
|
||||
viper.Set("env", "development")
|
||||
viper.Set("db-dsn", b.runtime)
|
||||
if withMigration {
|
||||
viper.Set("db-migrate-dsn", b.scratch)
|
||||
}
|
||||
}
|
||||
|
||||
// migrateTo migrates the scratch database as the migration user, every stream
|
||||
// to its highest version except stripe, which stops one version short when
|
||||
// short is true.
|
||||
func (b *bootBox) migrateTo(short bool) {
|
||||
b.t.Helper()
|
||||
scratch := dbtest.Connect(b.t, b.scratch)
|
||||
highest := highestVersions(b.t)
|
||||
for _, src := range migrate.Sources() {
|
||||
version := highest[src.Name]
|
||||
if short && src.Name == "stripe" {
|
||||
version--
|
||||
}
|
||||
dbtest.UpTo(b.t, scratch, src, version)
|
||||
}
|
||||
dbtest.On(b.t, scratch).Exec(fmt.Sprintf("GRANT %s TO %s WITH INHERIT TRUE", db.RuntimeGroupRole, b.login))
|
||||
}
|
||||
|
||||
// open opens the database in runtime mode and returns the pool and the log.
|
||||
func (b *bootBox) open() (*sql.DB, *bytes.Buffer, error) {
|
||||
b.t.Helper()
|
||||
var logs bytes.Buffer
|
||||
logger := slog.New(slog.NewTextHandler(&logs, nil))
|
||||
database, closeDB, err := openDB(context.Background(), logger, dbRuntime)
|
||||
if err != nil {
|
||||
return nil, &logs, err
|
||||
}
|
||||
b.t.Cleanup(closeDB)
|
||||
return database, &logs, nil
|
||||
}
|
||||
|
||||
// ledgers lists, per stream, the highest applied version and the number of
|
||||
// applied rows, read as the migration user.
|
||||
func (b *bootBox) ledgers() string {
|
||||
b.t.Helper()
|
||||
scratch := dbtest.Connect(b.t, b.scratch)
|
||||
var parts []string
|
||||
for _, src := range migrate.Sources() {
|
||||
var top, n sql.NullInt64
|
||||
query := "SELECT max(version_id), count(*) FROM public.goose_db_version_" + src.Name + " WHERE is_applied"
|
||||
if err := scratch.QueryRow(query).Scan(&top, &n); err != nil {
|
||||
parts = append(parts, src.Name+": "+err.Error())
|
||||
continue
|
||||
}
|
||||
parts = append(parts, fmt.Sprintf("%s:%d:%d", src.Name, top.Int64, n.Int64))
|
||||
}
|
||||
return strings.Join(parts, " ")
|
||||
}
|
||||
|
||||
// migrationSessions counts the sessions of the migration user on the scratch
|
||||
// database, waiting briefly for a closed pool's sessions to end.
|
||||
func (b *bootBox) migrationSessions() int64 {
|
||||
b.t.Helper()
|
||||
q := dbtest.On(b.t, b.admin)
|
||||
var n int64
|
||||
for range 30 {
|
||||
n = q.Int(`SELECT count(*) FROM pg_stat_activity
|
||||
WHERE datname = $1 AND usename = (SELECT current_user::text)`, b.name)
|
||||
if n == 0 {
|
||||
return 0
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// openRuntimeDatabaseRows are the rows of TestOpenRuntimeDatabase, each run
|
||||
// by its own named function.
|
||||
var openRuntimeDatabaseRows = []struct {
|
||||
name string
|
||||
run func(t *testing.T)
|
||||
}{
|
||||
{"both strings on a fresh database", bootWithBothStrings},
|
||||
{"without db-migrate-dsn on a database behind", bootBehindWithoutMigrationString},
|
||||
{"without db-migrate-dsn on a current database", bootCurrentWithoutMigrationString},
|
||||
{"migrate up without db-migrate-dsn", migrateUpWithoutMigrationString},
|
||||
{"a runtime login that set its own role", bootWithSelfSetRole},
|
||||
}
|
||||
|
||||
// TestOpenRuntimeDatabase covers task 4.8 (design D1, D8, D11): the runtime
|
||||
// mode migrates and grants on the migration connection, closes it, and serves
|
||||
// as the runtime login; without db-migrate-dsn it runs nothing and refuses a
|
||||
// schema that is behind.
|
||||
func TestOpenRuntimeDatabase(t *testing.T) {
|
||||
// viper.Reset drops the environment binding; blanking the variables as
|
||||
// well keeps the rows off a database that make test exports.
|
||||
t.Setenv("MC_DB_DSN", "")
|
||||
t.Setenv("MC_DB_DSN_FILE", "")
|
||||
t.Setenv("MC_DB_MIGRATE_DSN", "")
|
||||
t.Setenv("MC_DB_MIGRATE_DSN_FILE", "")
|
||||
for _, row := range openRuntimeDatabaseRows {
|
||||
t.Run(row.name, row.run)
|
||||
}
|
||||
}
|
||||
|
||||
func bootWithBothStrings(t *testing.T) {
|
||||
box := newBootBox(t)
|
||||
box.configure(true)
|
||||
database, _, err := box.open()
|
||||
if err != nil {
|
||||
t.Fatalf("openDB: %v", err)
|
||||
}
|
||||
q := dbtest.On(t, database)
|
||||
if got := q.String("SELECT current_user::text"); got != box.login {
|
||||
t.Errorf("SELECT current_user on the returned pool = %q, want the runtime login %q", got, box.login)
|
||||
}
|
||||
if q.String(fmt.Sprintf("SELECT pg_has_role(current_user, '%s', 'USAGE')::text", db.RuntimeGroupRole)) != "true" {
|
||||
t.Errorf("the runtime login %q does not inherit %s", box.login, db.RuntimeGroupRole)
|
||||
}
|
||||
for stream, want := range highestVersions(t) {
|
||||
got := q.Int("SELECT max(version_id) FROM public.goose_db_version_" + stream + " WHERE is_applied")
|
||||
if got != want {
|
||||
t.Errorf("ledger %s through the returned pool is at %d, want %d", stream, got, want)
|
||||
}
|
||||
}
|
||||
if n := box.migrationSessions(); n != 0 {
|
||||
t.Errorf("%d sessions of the migration user remain open on the scratch database, want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
func bootBehindWithoutMigrationString(t *testing.T) {
|
||||
box := newBootBox(t)
|
||||
box.migrateTo(true)
|
||||
before := box.ledgers()
|
||||
box.configure(false)
|
||||
_, _, err := box.open()
|
||||
want := fmt.Sprintf("the schema is behind this release (pending migrations: stripe %d); "+
|
||||
"run member-console migrate up with db-migrate-dsn set, then start again", highestVersions(t)["stripe"])
|
||||
var refusal *db.RuntimeUserRefusal
|
||||
if !errors.As(err, &refusal) || err.Error() != want {
|
||||
t.Errorf("openDB error = %v, want the refusal %q", err, want)
|
||||
}
|
||||
if after := box.ledgers(); after != before {
|
||||
t.Errorf("the ledgers changed: %s, want %s", after, before)
|
||||
}
|
||||
}
|
||||
|
||||
func bootCurrentWithoutMigrationString(t *testing.T) {
|
||||
box := newBootBox(t)
|
||||
box.migrateTo(false)
|
||||
before := box.ledgers()
|
||||
box.configure(false)
|
||||
database, logs, err := box.open()
|
||||
if err != nil {
|
||||
t.Fatalf("openDB: %v", err)
|
||||
}
|
||||
if got := dbtest.On(t, database).String("SELECT current_user::text"); got != box.login {
|
||||
t.Errorf("SELECT current_user on the returned pool = %q, want the runtime login %q", got, box.login)
|
||||
}
|
||||
if !strings.Contains(logs.String(), "migrations not run: db-migrate-dsn is not set") {
|
||||
t.Errorf("the log lacks the separate-step line:\n%s", logs)
|
||||
}
|
||||
if strings.Contains(logs.String(), "database migrations applied") {
|
||||
t.Errorf("migrations ran without db-migrate-dsn:\n%s", logs)
|
||||
}
|
||||
if after := box.ledgers(); after != before {
|
||||
t.Errorf("the ledgers changed: %s, want %s", after, before)
|
||||
}
|
||||
}
|
||||
|
||||
func migrateUpWithoutMigrationString(t *testing.T) {
|
||||
viper.Reset()
|
||||
t.Cleanup(viper.Reset)
|
||||
viper.Set("db-dsn", "postgres://runtime:pw@127.0.0.1:1/x?connect_timeout=1")
|
||||
const want = "member-console migrate needs db-migrate-dsn (MC_DB_MIGRATE_DSN or MC_DB_MIGRATE_DSN_FILE)"
|
||||
if err := migrateUpCmd.RunE(migrateUpCmd, nil); err == nil || err.Error() != want {
|
||||
t.Errorf("migrate up error = %v, want %q", err, want)
|
||||
}
|
||||
}
|
||||
|
||||
func bootWithSelfSetRole(t *testing.T) {
|
||||
box := newBootBox(t)
|
||||
admin := dbtest.On(t, box.admin)
|
||||
admin.Exec(fmt.Sprintf("GRANT %s TO %s WITH INHERIT TRUE", db.RuntimeGroupRole, box.login))
|
||||
admin.Exec(fmt.Sprintf("ALTER ROLE %s SET role = '%s'", box.login, db.RuntimeGroupRole))
|
||||
box.configure(true)
|
||||
database, _, err := box.open()
|
||||
if err != nil {
|
||||
t.Fatalf("openDB: %v", err)
|
||||
}
|
||||
if got := dbtest.On(t, database).String("SELECT current_user::text"); got != box.login {
|
||||
t.Errorf("SELECT current_user on the returned pool = %q, want the login %q, not the role it set", got, box.login)
|
||||
}
|
||||
}
|
||||
+78
-16
@@ -4,6 +4,8 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -20,18 +22,27 @@ import (
|
||||
type databaseCommandPath struct {
|
||||
name string
|
||||
run func() error
|
||||
// fileKey is the -file key the command reads, and unset is the refusal
|
||||
// it ends with when neither the key nor its direct variant is set.
|
||||
fileKey string
|
||||
unset string
|
||||
}
|
||||
|
||||
const (
|
||||
dbDSNNotSet = "db-dsn is not set (MC_DB_DSN or MC_DB_DSN_FILE)"
|
||||
dbMigrateDSNNotSet = "member-console migrate needs db-migrate-dsn (MC_DB_MIGRATE_DSN or MC_DB_MIGRATE_DSN_FILE)"
|
||||
)
|
||||
|
||||
func databaseCommandPaths() []databaseCommandPath {
|
||||
return []databaseCommandPath{
|
||||
{"migrate up", func() error { return migrateUpCmd.RunE(migrateUpCmd, nil) }},
|
||||
{"migrate down", func() error { return migrateDownCmd.RunE(migrateDownCmd, nil) }},
|
||||
{"migrate status", func() error { return migrateStatusCmd.RunE(migrateStatusCmd, nil) }},
|
||||
{"migrate up", func() error { return migrateUpCmd.RunE(migrateUpCmd, nil) }, "db-migrate-dsn-file", dbMigrateDSNNotSet},
|
||||
{"migrate down", func() error { return migrateDownCmd.RunE(migrateDownCmd, nil) }, "db-migrate-dsn-file", dbMigrateDSNNotSet},
|
||||
{"migrate status", func() error { return migrateStatusCmd.RunE(migrateStatusCmd, nil) }, "db-migrate-dsn-file", dbMigrateDSNNotSet},
|
||||
{"connectSettings", func() error {
|
||||
_, _, _, err := connectSettings()
|
||||
return err
|
||||
}},
|
||||
{"seed-demo", func() error { return seedDemoCmd.RunE(&cobra.Command{}, nil) }},
|
||||
}, "db-dsn-file", dbDSNNotSet},
|
||||
{"seed-demo", func() error { return seedDemoCmd.RunE(&cobra.Command{}, nil) }, "db-dsn-file", dbDSNNotSet},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -49,27 +60,25 @@ func closedPort(t *testing.T) string {
|
||||
return port
|
||||
}
|
||||
|
||||
const dbDSNNotSet = "db-dsn is not set (MC_DB_DSN or MC_DB_DSN_FILE)"
|
||||
|
||||
// checkFileKeyRead sets only db-dsn-file and asserts that the path read the
|
||||
// checkFileKeyRead sets only the command's file key and asserts that the path read the
|
||||
// file: its error is the connection failure to the file's port.
|
||||
func checkFileKeyRead(t *testing.T, p databaseCommandPath, port string) {
|
||||
t.Helper()
|
||||
viper.Reset()
|
||||
t.Cleanup(viper.Reset)
|
||||
|
||||
path := filepath.Join(t.TempDir(), "db-dsn")
|
||||
path := filepath.Join(t.TempDir(), p.fileKey)
|
||||
dsn := "postgres://u:p@127.0.0.1:" + port + "/x?connect_timeout=1"
|
||||
if err := os.WriteFile(path, []byte(dsn+"\n"), 0o600); err != nil {
|
||||
t.Fatalf("write fixture file: %v", err)
|
||||
}
|
||||
viper.Set("db-dsn-file", path)
|
||||
viper.Set(p.fileKey, path)
|
||||
|
||||
err := p.run()
|
||||
if err == nil {
|
||||
t.Fatalf("%s: want a connection failure, got nil", p.name)
|
||||
}
|
||||
if strings.Contains(err.Error(), dbDSNNotSet) {
|
||||
if strings.Contains(err.Error(), p.unset) {
|
||||
t.Errorf("%s: error = %q, want a connection failure, not the unset refusal", p.name, err.Error())
|
||||
}
|
||||
if !strings.Contains(err.Error(), port) {
|
||||
@@ -87,21 +96,42 @@ func checkUnsetRefused(t *testing.T, p databaseCommandPath) {
|
||||
if err == nil {
|
||||
t.Fatalf("%s: want the unset refusal, got nil", p.name)
|
||||
}
|
||||
if !strings.Contains(err.Error(), dbDSNNotSet) {
|
||||
t.Errorf("%s: error = %q, want it to contain %q", p.name, err.Error(), dbDSNNotSet)
|
||||
if !strings.Contains(err.Error(), p.unset) {
|
||||
t.Errorf("%s: error = %q, want it to contain %q", p.name, err.Error(), p.unset)
|
||||
}
|
||||
}
|
||||
|
||||
// checkWhitespaceRefused sets the command's direct key to whitespace only and
|
||||
// asserts the unset refusal: pgx would otherwise connect with libpq's defaults.
|
||||
func checkWhitespaceRefused(t *testing.T, p databaseCommandPath) {
|
||||
t.Helper()
|
||||
viper.Reset()
|
||||
t.Cleanup(viper.Reset)
|
||||
viper.Set(strings.TrimSuffix(p.fileKey, "-file"), " \t")
|
||||
|
||||
err := p.run()
|
||||
if err == nil {
|
||||
t.Fatalf("%s: want the unset refusal, got nil", p.name)
|
||||
}
|
||||
if !strings.Contains(err.Error(), p.unset) {
|
||||
t.Errorf("%s: error = %q, want it to contain %q", p.name, err.Error(), p.unset)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDatabaseCommandsReadTheFileKeys covers task 2.4 (design D12): every
|
||||
// command that opens the database resolves db-dsn-file first, and one that
|
||||
// command that opens the database resolves its file key first, and one that
|
||||
// finds no connection string refuses instead of letting the driver fall back
|
||||
// to its defaults.
|
||||
func TestDatabaseCommandsReadTheFileKeys(t *testing.T) {
|
||||
// viper.Reset drops the environment binding, so the rows never read
|
||||
// MC_DB_DSN; blanking it as well keeps migrate down, which this test runs,
|
||||
// off a real database even if that ever changes.
|
||||
// MC_DB_DSN or MC_DB_MIGRATE_DSN; blanking them as well keeps migrate down,
|
||||
// which this test runs, off a real database even if that ever changes.
|
||||
t.Setenv("MC_DB_DSN", "")
|
||||
t.Setenv("MC_DB_DSN_FILE", "")
|
||||
t.Setenv("MC_DB_MIGRATE_DSN", "")
|
||||
t.Setenv("MC_DB_MIGRATE_DSN_FILE", "")
|
||||
t.Setenv("MC_DB_MIGRATE_DSN", "")
|
||||
t.Setenv("MC_DB_MIGRATE_DSN_FILE", "")
|
||||
port := closedPort(t)
|
||||
for _, p := range databaseCommandPaths() {
|
||||
t.Run("file key read: "+p.name, func(t *testing.T) {
|
||||
@@ -113,4 +143,36 @@ func TestDatabaseCommandsReadTheFileKeys(t *testing.T) {
|
||||
checkUnsetRefused(t, p)
|
||||
})
|
||||
}
|
||||
for _, p := range databaseCommandPaths() {
|
||||
t.Run("whitespace only: "+p.name, func(t *testing.T) {
|
||||
checkWhitespaceRefused(t, p)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestMigrationDSNIsUnsetWhenOnlyWhitespace covers the read start and
|
||||
// seed-demo make of db-migrate-dsn: the boot migrates only when it holds a
|
||||
// string, and ValidateStart already treats whitespace as none.
|
||||
func TestMigrationDSNIsUnsetWhenOnlyWhitespace(t *testing.T) {
|
||||
viper.Reset()
|
||||
t.Cleanup(viper.Reset)
|
||||
viper.Set("db-migrate-dsn", " \n")
|
||||
if got := migrationDSN(); got != "" {
|
||||
t.Errorf("migrationDSN() = %q, want it unset", got)
|
||||
}
|
||||
viper.Set("db-migrate-dsn", " postgres://u@h/db ")
|
||||
if got := migrationDSN(); got != "postgres://u@h/db" {
|
||||
t.Errorf("migrationDSN() = %q, want the string without its whitespace", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestOpenDBRefusesAnUnknownMode covers the mode switch: a mode that no case
|
||||
// names opens nothing, instead of running the migrating boot.
|
||||
func TestOpenDBRefusesAnUnknownMode(t *testing.T) {
|
||||
viper.Reset()
|
||||
t.Cleanup(viper.Reset)
|
||||
_, _, err := openDB(context.Background(), slog.New(slog.DiscardHandler), dbMode(99))
|
||||
if err == nil || !strings.Contains(err.Error(), "unknown database mode 99") {
|
||||
t.Errorf("openDB error = %v, want unknown database mode 99", err)
|
||||
}
|
||||
}
|
||||
|
||||
+3
-3
@@ -34,7 +34,7 @@ var migrateUpCmd = &cobra.Command{
|
||||
return err
|
||||
}
|
||||
|
||||
database, closeDB, err := openDB(context.Background(), logger, dbPlain)
|
||||
database, closeDB, err := openDB(context.Background(), logger, dbMigration)
|
||||
if err != nil {
|
||||
logger.Error("failed to connect to database", slog.Any("error", err))
|
||||
return err
|
||||
@@ -66,7 +66,7 @@ var migrateDownCmd = &cobra.Command{
|
||||
return err
|
||||
}
|
||||
|
||||
database, closeDB, err := openDB(context.Background(), logger, dbPlain)
|
||||
database, closeDB, err := openDB(context.Background(), logger, dbMigration)
|
||||
if err != nil {
|
||||
logger.Error("failed to connect to database", slog.Any("error", err))
|
||||
return err
|
||||
@@ -98,7 +98,7 @@ var migrateStatusCmd = &cobra.Command{
|
||||
return err
|
||||
}
|
||||
|
||||
database, closeDB, err := openDB(context.Background(), logger, dbPlain)
|
||||
database, closeDB, err := openDB(context.Background(), logger, dbMigration)
|
||||
if err != nil {
|
||||
logger.Error("failed to connect to database", slog.Any("error", err))
|
||||
return err
|
||||
|
||||
@@ -55,4 +55,41 @@ func TestResolveSecretFiles_DBDSN(t *testing.T) {
|
||||
t.Errorf("error = %q, want it to name db-dsn", err.Error())
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("db-migrate-dsn-file loads the DSN", func(t *testing.T) {
|
||||
viper.Reset()
|
||||
t.Cleanup(viper.Reset)
|
||||
|
||||
dsn := "postgres://migrator:hunter2@db.example.com:5432/widgets"
|
||||
path := filepath.Join(t.TempDir(), "db-migrate-dsn")
|
||||
if err := os.WriteFile(path, []byte(dsn+"\n"), 0o600); err != nil {
|
||||
t.Fatalf("write fixture file: %v", err)
|
||||
}
|
||||
viper.Set("db-migrate-dsn-file", path)
|
||||
|
||||
if err := resolveSecretFiles(nil); err != nil {
|
||||
t.Fatalf("resolveSecretFiles: %v", err)
|
||||
}
|
||||
if got := viper.GetString("db-migrate-dsn"); got != dsn {
|
||||
t.Errorf("db-migrate-dsn = %q, want %q", got, dsn)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("db-migrate-dsn and db-migrate-dsn-file both set is ambiguous", func(t *testing.T) {
|
||||
viper.Reset()
|
||||
t.Cleanup(viper.Reset)
|
||||
|
||||
path := filepath.Join(t.TempDir(), "db-migrate-dsn")
|
||||
if err := os.WriteFile(path, []byte("postgres://migrator:hunter2@db.example.com:5432/widgets"), 0o600); err != nil {
|
||||
t.Fatalf("write fixture file: %v", err)
|
||||
}
|
||||
viper.Set("db-migrate-dsn", "postgres://migrator:hunter2@db.example.com:5432/widgets")
|
||||
viper.Set("db-migrate-dsn-file", path)
|
||||
|
||||
const want = "db-migrate-dsn: both a direct value and a file path are set; use only one"
|
||||
err := resolveSecretFiles(nil)
|
||||
if err == nil || err.Error() != want {
|
||||
t.Errorf("error = %v, want %q", err, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
+1
-1
@@ -37,7 +37,7 @@ the org-type default) to exercise the OIDC lazy-creation path. See OpenSpec chan
|
||||
return err
|
||||
}
|
||||
|
||||
database, closeDB, err := openDB(ctx, logger, dbMigrated)
|
||||
database, closeDB, err := openDB(ctx, logger, dbRuntime)
|
||||
if err != nil {
|
||||
logger.Error("failed to initialize database", slog.Any("error", err))
|
||||
return err
|
||||
|
||||
+7
-4
@@ -162,9 +162,9 @@ var startCmd = &cobra.Command{
|
||||
stripeKeyFingerprint := stripeintegration.FingerprintForKey(viper.GetString("stripe-api-key"))
|
||||
|
||||
// Database Setup
|
||||
database, closeDB, err := openDB(ctx, logger, dbMigrated)
|
||||
database, closeDB, err := openDB(ctx, logger, dbRuntime)
|
||||
if err != nil {
|
||||
logger.Error("failed to initialize database", slog.Any("error", err))
|
||||
logOpenFailure(logger, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer closeDB()
|
||||
@@ -580,8 +580,10 @@ func init() {
|
||||
startCmd.Flags().String("base-url", "", "Address at which the server is exposed")
|
||||
startCmd.Flags().String("env", "", "Environment label; development selects text logs, anything else JSON")
|
||||
startCmd.Flags().String("deployment-name", "", "Name this deployment presents for itself on the member and operator mastheads, page titles, and OpenGraph tags (default \"Member Console\")")
|
||||
startCmd.Flags().String("db-dsn", "", "PostgreSQL connection string (e.g., postgres://user:pass@localhost:5432/dbname?sslmode=disable; required, or db-dsn-file)")
|
||||
startCmd.Flags().String("db-dsn-file", "", "Path to file containing the PostgreSQL connection string")
|
||||
startCmd.Flags().String("db-dsn", "", "PostgreSQL connection string of the runtime user (e.g., postgres://user:pass@localhost:5432/dbname?sslmode=disable; required, or db-dsn-file)")
|
||||
startCmd.Flags().String("db-dsn-file", "", "Path to file containing the runtime user's PostgreSQL connection string")
|
||||
startCmd.Flags().String("db-migrate-dsn", "", "PostgreSQL connection string of the migration user; start runs migrations only when it is set")
|
||||
startCmd.Flags().String("db-migrate-dsn-file", "", "Path to file containing the migration user's PostgreSQL connection string")
|
||||
startCmd.Flags().String("valkey-addr", "", "Valkey/Redis address for session storage (host:port)")
|
||||
|
||||
// OIDC configuration
|
||||
@@ -742,6 +744,7 @@ func loadFromFile(path string) (string, error) {
|
||||
// as a file runs migrate, config and seed-demo as it runs start.
|
||||
var databaseSecretPairs = []config.SecretPair{
|
||||
{Name: "db-dsn", FileName: "db-dsn-file"},
|
||||
{Name: "db-migrate-dsn", FileName: "db-migrate-dsn-file"},
|
||||
}
|
||||
|
||||
// resolveSecretFiles loads every "<name>-file" secret variant (core's own
|
||||
|
||||
@@ -81,8 +81,12 @@ binary. The previous binary does not write through the migrated schema;
|
||||
its direct table writes fail loudly, which is the intended fail-closed
|
||||
posture.
|
||||
|
||||
Migrations run automatically on startup. The CLI also provides `migrate up`,
|
||||
`migrate down`, and `migrate status` commands.
|
||||
Migrations run automatically on startup, on the migration user's connection,
|
||||
when `db-migrate-dsn` is set; `start` closes that connection before it serves.
|
||||
With `db-migrate-dsn` left out, `start` runs no migration and refuses to start
|
||||
while the schema is behind the code. The CLI also provides `migrate up`,
|
||||
`migrate down`, and `migrate status` commands, which use `db-migrate-dsn` and
|
||||
refuse to run without it.
|
||||
|
||||
## Creating New Migrations
|
||||
|
||||
@@ -105,9 +109,11 @@ needs a mark in `internal/migrate/testdata/console-reader-columns.txt`, or
|
||||
|
||||
Every migration with more than one SQL statement **must** wrap both Up and Down sections in `-- +goose StatementBegin` / `-- +goose StatementEnd`. Without these directives, goose v3 splits statements and runs them individually, which can silently skip failed statements while still marking the migration as applied.
|
||||
|
||||
### Migrations Run Without `search_path`
|
||||
### Migrations Run With `search_path=public`
|
||||
|
||||
The `ConnectAndMigrate` function runs migrations on a **separate database connection** that does not include the custom `search_path` (`core,public`). This is intentional — a pgx/PostgreSQL 18 interaction causes multi-statement DDL sent via the simple query protocol to silently lose tables when a non-default schema appears in the connection-level `search_path`. After migrations complete, the application connection uses the full `search_path` as normal.
|
||||
`db.ConnectMigration` opens the **migration connection**, a pool of its own that `start` migrates and grants on and that `migrate up|down|status` use. It opens with the startup parameters `search_path=public` and `role=none`, replacing any `search_path` or `role` the connection string carries (the string is parsed once, and a key matches whatever its case; a `-c` switch in `options` loses to the startup parameter because Postgres applies `options` first), and it does not include the application's custom `search_path` (`core,public`). The application's pool, opened by `db.Connect` on `db-dsn`, uses `core,public` and `role=none`.
|
||||
|
||||
Keeping `core,public` off the migration connection is intentional. A pgx/PostgreSQL 18 interaction causes multi-statement DDL sent via the simple query protocol to silently lose tables when a non-default schema appears in the connection-level `search_path`. The `config` commands open `db-dsn` with `db.ConnectPlain`, without the application's `search_path` and without the startup parameters, because an operator runs them by hand and they take no part in the boot. `public` is also where goose's ledgers live: with `search_path=public`, goose creates every `goose_db_version_<stream>` table there, and `db.PendingMigrations` reads each as `public.goose_db_version_<stream>` by qualified name. A startup parameter outranks `ALTER DATABASE ... SET` and `ALTER ROLE ... SET`, so a `search_path` or `role` setting on the database or the login cannot move the ledgers or change who runs the DDL. After migrations complete, the application connection uses the full `search_path` as normal.
|
||||
|
||||
```sql
|
||||
-- +goose Up
|
||||
|
||||
@@ -55,8 +55,10 @@ from `internal/config/validate.go`'s (`ValidateStart`) checks.
|
||||
|
||||
| Key | Type | Env override | Purpose | Default | Required |
|
||||
|---|---|---|---|---|---|
|
||||
| `db-dsn` | url (postgres) | `MC_DB_DSN` | PostgreSQL connection string, e.g. `postgres://user:pass@host:5432/dbname?sslmode=disable`. Use this or `db-dsn-file`, never both. Treated as a secret: never logged; the boot log names host, port, database and user only. | none | One of this or `db-dsn-file` |
|
||||
| `db-dsn-file` | path | `MC_DB_DSN_FILE` | File holding the PostgreSQL DSN, read at startup. | none | One of this or `db-dsn` |
|
||||
| `db-dsn` | url (postgres) | `MC_DB_DSN` | PostgreSQL connection string of the runtime user, the user the console serves from: not a superuser, owns nothing, and holds no role but `console_runtime`. E.g. `postgres://user:pass@host:5432/dbname?sslmode=disable`. Use this or `db-dsn-file`, never both. Treated as a secret: never logged; the boot log names host, port, database and user only. | none | One of this or `db-dsn-file` |
|
||||
| `db-dsn-file` | path | `MC_DB_DSN_FILE` | File holding the runtime user's PostgreSQL DSN, read at startup. | none | One of this or `db-dsn` |
|
||||
| `db-migrate-dsn` | url (postgres) | `MC_DB_MIGRATE_DSN` | PostgreSQL connection string of the migration user, the user that owns the schemas. `start` runs the migrations and grants `console_runtime` to the runtime user on this connection, then closes it; with the key unset `start` runs no migrations and refuses a schema that is behind the code. `member-console migrate up\|down\|status` use this key and refuse to run without it. Use this or `db-migrate-dsn-file`, never both. Treated as a secret: never logged; the boot log names host, port, database and user only. | none | Optional for `start`; required by `migrate` |
|
||||
| `db-migrate-dsn-file` | path | `MC_DB_MIGRATE_DSN_FILE` | File holding the migration user's PostgreSQL DSN, read at startup. | none | Optional |
|
||||
| `valkey-addr` | string | `MC_VALKEY_ADDR` | Valkey/Redis address (`host:port`) for server-side session storage. | `localhost:6379` | Required (satisfied by the default unless explicitly cleared) |
|
||||
| `valkey-username` | string | `MC_VALKEY_USERNAME` | ACL username for the session store, when it uses one. | none | Optional |
|
||||
| `valkey-password` | string | `MC_VALKEY_PASSWORD` | Password for the session store. Use this or `valkey-password-file`, never both. | none | **Required** (the store must not be reachable unauthenticated) |
|
||||
|
||||
+7
-3
@@ -30,10 +30,14 @@ docker buildx build \
|
||||
|
||||
## Secrets
|
||||
|
||||
Four configuration keys are secrets, each with a `-file` variant that reads
|
||||
Five configuration keys are secrets, each with a `-file` variant that reads
|
||||
the value from a file at startup (`docs/environment-reference.md`):
|
||||
|
||||
- `db-dsn` — the PostgreSQL connection string, password included.
|
||||
- `db-dsn` — the PostgreSQL connection string of the runtime user, password
|
||||
included.
|
||||
- `db-migrate-dsn` — the PostgreSQL connection string of the migration user.
|
||||
Optional: with it unset, `start` runs no migrations, and only the step that
|
||||
runs `member-console migrate up` holds the migration user's string.
|
||||
- `oidc-sp-client-secret` — the OIDC client's secret, issued by the identity
|
||||
provider.
|
||||
- `valkey-password` — the session store's password. Required; the console
|
||||
@@ -44,7 +48,7 @@ None of them is generated by or for the console: each is a credential for
|
||||
something external, obtained from that system. Sessions are stored server-side
|
||||
in Valkey and carry no signing secret of their own, and cross-origin request
|
||||
protection issues no token, so there is no signing key for it either; rotating
|
||||
any of the four means updating the deployment's configuration and restarting.
|
||||
any of the five means updating the deployment's configuration and restarting.
|
||||
|
||||
## Session store limits
|
||||
|
||||
|
||||
+8
-4
@@ -53,8 +53,9 @@ Its role:
|
||||
- **Is where the test databases come from.** `./reset-test-db.sh` drops,
|
||||
recreates and migrates `member_console_test` (`TEST_DATABASE_URL`) and
|
||||
`member_console_e2e` (`E2E_DATABASE_URL`); `make test` runs it first.
|
||||
`test/.env` holds both addresses beside `MC_DB_DSN`, the running app's
|
||||
database, which only the browser tests write.
|
||||
`test/.env` holds both addresses beside `MC_DB_DSN` and `MC_DB_MIGRATE_DSN`,
|
||||
the running app's database as its runtime user and as its migration user;
|
||||
only the browser tests write it. `TEST_DATABASE_URL` stays the superuser.
|
||||
- **Houses the end-to-end Go suites** under `test/e2e/`: `plan-management/`
|
||||
(in-process `httptest` over the real handlers + a migrated DB) and
|
||||
`operator-walkthroughs/` (Go tests that drive a browser against the running
|
||||
@@ -344,8 +345,11 @@ Exercise whole flows against a running stack. Three flavors:
|
||||
- **`Recover`.** Each walkthrough defers `browsertest.Recover(t)` first, so a
|
||||
rod panic fails that walkthrough with its stack and the package's other
|
||||
walkthroughs still run.
|
||||
- **`AppDB`** opens the running app's database (`MC_DB_DSN` in `test/.env`),
|
||||
never migrates it, and skips when the key is missing.
|
||||
- **`AppDB`** opens the running app's database as the runtime user
|
||||
(`MC_DB_DSN` in `test/.env`), never migrates it, and skips when the key is
|
||||
missing. **`AppAdminDB`** opens it as the migration user
|
||||
(`MC_DB_MIGRATE_DSN`) for work that needs a superuser, such as the screens'
|
||||
timestamp pinning, and skips when that key is missing.
|
||||
- **Free functions that take `t`**, and no helper stores it, so a subtest
|
||||
never calls a parent's.
|
||||
- **A long walkthrough is phases**: named functions, or methods on a struct
|
||||
|
||||
@@ -44,6 +44,14 @@ func ValidateStart(integrationSpecs []ConfigKey) error {
|
||||
errs = append(errs, errors.New("db-dsn is not a valid PostgreSQL URL (want postgres://user:pass@host:5432/dbname)"))
|
||||
}
|
||||
|
||||
// db-migrate-dsn is optional: db-dsn alone is the separate-step mode, in
|
||||
// which start runs no migration. Set, it has the shape of db-dsn.
|
||||
if mdsn := strings.TrimSpace(viper.GetString("db-migrate-dsn")); mdsn != "" {
|
||||
if u, err := url.Parse(mdsn); err != nil || (u.Scheme != "postgres" && u.Scheme != "postgresql") {
|
||||
errs = append(errs, errors.New("db-migrate-dsn is not a valid PostgreSQL URL (want postgres://user:pass@host:5432/dbname)"))
|
||||
}
|
||||
}
|
||||
|
||||
if strings.TrimSpace(viper.GetString("valkey-addr")) == "" {
|
||||
errs = append(errs, required("valkey-addr", "the Valkey/Redis session store, e.g. localhost:6379"))
|
||||
}
|
||||
|
||||
@@ -44,6 +44,19 @@ func TestValidateStart_Valid(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestValidateStart_MigrateDSNIsOptional covers task 4.1: db-dsn alone is a
|
||||
// valid configuration (the separate-step mode), and a well-formed
|
||||
// db-migrate-dsn passes beside it.
|
||||
func TestValidateStart_MigrateDSNIsOptional(t *testing.T) {
|
||||
for _, dsn := range []string{"", "postgres://m:p@localhost:5432/db?sslmode=disable"} {
|
||||
validConfig()
|
||||
viper.Set("db-migrate-dsn", dsn)
|
||||
if err := ValidateStart(stripeSpec); err != nil {
|
||||
t.Errorf("db-migrate-dsn %q: want a valid configuration, got: %v", dsn, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateStart_Errors(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
@@ -52,6 +65,8 @@ func TestValidateStart_Errors(t *testing.T) {
|
||||
}{
|
||||
{"missing db-dsn", func() { viper.Set("db-dsn", "") }, "db-dsn is required"},
|
||||
{"malformed db-dsn", func() { viper.Set("db-dsn", "mysql://x") }, "db-dsn is not a valid PostgreSQL URL"},
|
||||
{"malformed db-migrate-dsn", func() { viper.Set("db-migrate-dsn", "mysql://x") },
|
||||
"db-migrate-dsn is not a valid PostgreSQL URL (want postgres://user:pass@host:5432/dbname)"},
|
||||
{"missing valkey-addr", func() { viper.Set("valkey-addr", "") }, "valkey-addr is required"},
|
||||
{"missing valkey-password", func() { viper.Set("valkey-password", "") }, "valkey-password is required"},
|
||||
{"base-url with a scheme a browser cannot use", func() { viper.Set("base-url", "ftp://console.example.com") }, "base-url must be http or https"},
|
||||
|
||||
+118
-72
@@ -10,10 +10,12 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
_ "github.com/jackc/pgx/v5/stdlib"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/stdlib"
|
||||
)
|
||||
|
||||
// DBConfig holds database configuration.
|
||||
@@ -36,27 +38,86 @@ func DefaultDBConfig(dsn string) *DBConfig {
|
||||
}
|
||||
}
|
||||
|
||||
// ensureSearchPath appends search_path with the core schema to the DSN if
|
||||
// not already present, so every connection in the pool resolves core-schema
|
||||
// tables without qualification (Decision 113). Provider schemas (stripe,
|
||||
// fedwiki) stay out of search_path and their queries stay schema-qualified.
|
||||
func ensureSearchPath(dsn string) string {
|
||||
const moduleSearchPath = "core,public"
|
||||
u, err := url.Parse(dsn)
|
||||
if err != nil || u.Scheme == "" {
|
||||
// Key=value DSN format — append if not already set.
|
||||
if !strings.Contains(dsn, "search_path") {
|
||||
return dsn + " search_path=" + moduleSearchPath
|
||||
// startupParam is one connection parameter that the pool sets in the
|
||||
// connection's startup message.
|
||||
type startupParam struct{ key, value string }
|
||||
|
||||
// The parameters the pools open with. A startup parameter outranks
|
||||
// ALTER DATABASE ... SET and ALTER ROLE ... SET, so a setting that a database
|
||||
// or a role carries cannot change where names resolve or who current_user is.
|
||||
// It also outranks a -c switch in the string's options, which Postgres applies
|
||||
// before the startup parameters.
|
||||
var (
|
||||
// roleNone keeps current_user the login, whatever role a database or a
|
||||
// role setting names.
|
||||
roleNone = startupParam{"role", "none"}
|
||||
// migrationSearchPath is where goose's ledgers live: the schema every
|
||||
// migration connection resolves unqualified names in.
|
||||
migrationSearchPath = startupParam{"search_path", "public"}
|
||||
// moduleSearchPath is where the runtime pool resolves names (Decision
|
||||
// 113). Provider schemas (stripe, fedwiki) stay out of it and their
|
||||
// queries stay schema-qualified.
|
||||
moduleSearchPath = startupParam{"search_path", "core,public"}
|
||||
)
|
||||
|
||||
// deleteParam removes every runtime parameter whose key matches key without
|
||||
// regard to case, as Postgres matches parameter names, and returns the value
|
||||
// of the last one in key order (empty when there is none).
|
||||
func deleteParam(params map[string]string, key string) string {
|
||||
var matches []string
|
||||
for k := range params {
|
||||
if strings.EqualFold(k, key) {
|
||||
matches = append(matches, k)
|
||||
}
|
||||
return dsn
|
||||
}
|
||||
// URI format — append as query parameter.
|
||||
q := u.Query()
|
||||
if q.Get("search_path") == "" {
|
||||
q.Set("search_path", moduleSearchPath)
|
||||
u.RawQuery = q.Encode()
|
||||
slices.Sort(matches)
|
||||
var value string
|
||||
for _, k := range matches {
|
||||
value = params[k]
|
||||
delete(params, k)
|
||||
}
|
||||
return u.String()
|
||||
return value
|
||||
}
|
||||
|
||||
// setParams replaces each parameter's value, whatever case its key has in the
|
||||
// string.
|
||||
func setParams(params map[string]string, set ...startupParam) {
|
||||
for _, p := range set {
|
||||
deleteParam(params, p.key)
|
||||
params[p.key] = p.value
|
||||
}
|
||||
}
|
||||
|
||||
// keepSearchPath keeps the search_path the string carries, and sets
|
||||
// core,public when it carries none or an empty one.
|
||||
func keepSearchPath(params map[string]string) {
|
||||
if value := deleteParam(params, moduleSearchPath.key); value != "" {
|
||||
params[moduleSearchPath.key] = value
|
||||
return
|
||||
}
|
||||
setParams(params, moduleSearchPath)
|
||||
}
|
||||
|
||||
// openPool parses config.DSN once with the parser that connects, lets shape
|
||||
// set the startup parameters on the parsed parameters, and opens the pool with
|
||||
// the settings of config. The string is never edited as text, so no spelling
|
||||
// of a key, quoting or fragment can survive beside the values shape sets.
|
||||
func openPool(config *DBConfig, shape func(params map[string]string)) (*sql.DB, error) {
|
||||
cfg, err := pgx.ParseConfig(config.DSN)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to open database: %w", err)
|
||||
}
|
||||
shape(cfg.RuntimeParams)
|
||||
return configurePool(stdlib.OpenDB(*cfg), config), nil
|
||||
}
|
||||
|
||||
// configurePool applies the pool settings of config.
|
||||
func configurePool(db *sql.DB, config *DBConfig) *sql.DB {
|
||||
db.SetMaxOpenConns(config.MaxOpenConns)
|
||||
db.SetMaxIdleConns(config.MaxIdleConns)
|
||||
db.SetConnMaxLifetime(config.ConnMaxLifetime)
|
||||
db.SetConnMaxIdleTime(config.ConnMaxIdleTime)
|
||||
return db
|
||||
}
|
||||
|
||||
// dsnLogAttrs parses dsn as a PostgreSQL connection URL and returns only the
|
||||
@@ -84,34 +145,25 @@ func dsnLogAttrs(dsn string) []slog.Attr {
|
||||
}
|
||||
}
|
||||
|
||||
// openAndConfigureDB opens the database connection and configures the connection pool.
|
||||
// openAndConfigureDB opens the runtime pool: the application's search_path
|
||||
// (kept when the string carries one) and role=none.
|
||||
func openAndConfigureDB(config *DBConfig) (*sql.DB, error) {
|
||||
db, err := sql.Open("pgx", ensureSearchPath(config.DSN))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to open database: %w", err)
|
||||
}
|
||||
|
||||
// Configure connection pool
|
||||
db.SetMaxOpenConns(config.MaxOpenConns)
|
||||
db.SetMaxIdleConns(config.MaxIdleConns)
|
||||
db.SetConnMaxLifetime(config.ConnMaxLifetime)
|
||||
db.SetConnMaxIdleTime(config.ConnMaxIdleTime)
|
||||
|
||||
return db, nil
|
||||
return openPool(config, func(params map[string]string) {
|
||||
keepSearchPath(params)
|
||||
setParams(params, roleNone)
|
||||
})
|
||||
}
|
||||
|
||||
// ConnectPlain opens a database connection without the custom search_path.
|
||||
// Use this for migrations — the custom search_path causes a pgx/PG18
|
||||
// interaction that silently drops tables from multi-statement DDL.
|
||||
// ConnectPlain opens a database connection without the custom search_path
|
||||
// and without the startup parameters of the other two pools. The config
|
||||
// commands use it on db-dsn: they are tools an operator runs by hand, not the
|
||||
// serving process or the migration run.
|
||||
func ConnectPlain(ctx context.Context, logger *slog.Logger, config *DBConfig) (*sql.DB, error) {
|
||||
db, err := sql.Open("pgx", config.DSN)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to open database: %w", err)
|
||||
}
|
||||
db.SetMaxOpenConns(config.MaxOpenConns)
|
||||
db.SetMaxIdleConns(config.MaxIdleConns)
|
||||
db.SetConnMaxLifetime(config.ConnMaxLifetime)
|
||||
db.SetConnMaxIdleTime(config.ConnMaxIdleTime)
|
||||
configurePool(db, config)
|
||||
if err := db.PingContext(ctx); err != nil {
|
||||
return nil, errors.Join(fmt.Errorf("failed to connect to database: %w", err), db.Close())
|
||||
}
|
||||
@@ -120,8 +172,33 @@ func ConnectPlain(ctx context.Context, logger *slog.Logger, config *DBConfig) (*
|
||||
return db, nil
|
||||
}
|
||||
|
||||
// Connect initializes and returns a new database connection pool.
|
||||
// This is the basic connection function without any automatic operations.
|
||||
// ConnectMigration opens the migration connection: the pool the migrations,
|
||||
// the boot grant and the migrate commands run on. It opens with the startup
|
||||
// parameters search_path=public and role=none, replacing any search_path or
|
||||
// role the string carries, whatever case its keys have. The runtime pool's
|
||||
// rule, which keeps an operator's search_path, is not reused: a
|
||||
// db-migrate-dsn copied from the runtime string would otherwise open with
|
||||
// core,public, and a database-level or role-level search_path or role setting
|
||||
// would decide where goose's ledgers land and who runs the DDL (design D2,
|
||||
// rule 2).
|
||||
func ConnectMigration(ctx context.Context, logger *slog.Logger, config *DBConfig) (*sql.DB, error) {
|
||||
db, err := openPool(config, func(params map[string]string) {
|
||||
setParams(params, migrationSearchPath, roleNone)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := db.PingContext(ctx); err != nil {
|
||||
return nil, errors.Join(fmt.Errorf("failed to connect to database: %w", err), db.Close())
|
||||
}
|
||||
logger.LogAttrs(ctx, slog.LevelInfo, "database connection established (migration)",
|
||||
dsnLogAttrs(config.DSN)...)
|
||||
return db, nil
|
||||
}
|
||||
|
||||
// Connect initializes and returns the runtime connection pool: the
|
||||
// application's search_path (core,public, kept when the string carries one)
|
||||
// and role=none, always replacing the string's value. It runs no migration.
|
||||
func Connect(ctx context.Context, logger *slog.Logger, config *DBConfig) (*sql.DB, error) {
|
||||
db, err := openAndConfigureDB(config)
|
||||
if err != nil {
|
||||
@@ -139,34 +216,3 @@ func Connect(ctx context.Context, logger *slog.Logger, config *DBConfig) (*sql.D
|
||||
|
||||
return db, nil
|
||||
}
|
||||
|
||||
// ConnectAndMigrate initializes a database connection and automatically runs migrations.
|
||||
// The sources parameter specifies migration sources in dependency order.
|
||||
//
|
||||
// Migrations run on a separate connection WITHOUT the custom search_path to
|
||||
// avoid a pgx/PG18 interaction where setting the core schema in the search_path
|
||||
// at connection time causes multi-statement DDL to silently lose tables. After
|
||||
// migrations complete, the returned connection uses the full search_path for
|
||||
// application queries.
|
||||
func ConnectAndMigrate(ctx context.Context, logger *slog.Logger, config *DBConfig, sources []MigrationSource) (*sql.DB, error) {
|
||||
// Run migrations on a plain connection (no custom search_path).
|
||||
migDB, err := ConnectPlain(ctx, logger, config)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := RunMigrations(migDB, sources); err != nil {
|
||||
return nil, errors.Join(fmt.Errorf("failed to run migrations: %w", err), migDB.Close())
|
||||
}
|
||||
if err := migDB.Close(); err != nil {
|
||||
logger.Warn("closing the migration connection", slog.Any("error", err))
|
||||
}
|
||||
|
||||
// Now open the application connection with full search_path.
|
||||
db, err := Connect(ctx, logger, config)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
logger.Info("database migrations applied")
|
||||
return db, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
|
||||
package db_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/db"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest"
|
||||
)
|
||||
|
||||
// TestGrantRuntimeRole covers task 4.8 (design D8): the boot grant, run in a
|
||||
// transaction the test rolls back because its setting is local to one, makes
|
||||
// a NOINHERIT login whose name needs quoting inherit console_runtime, and a
|
||||
// second grant succeeds and changes nothing. Role memberships are
|
||||
// cluster-wide, so the rollback is what keeps the shared cluster as it was.
|
||||
func TestGrantRuntimeRole(t *testing.T) {
|
||||
q := dbtest.Tx(t, dbtest.Open(t))
|
||||
login := dbtest.Name("Mc-Runtime-")
|
||||
q.Exec(`CREATE ROLE "` + login + `" LOGIN NOINHERIT`)
|
||||
|
||||
members := func() int64 {
|
||||
return q.Int(`SELECT count(*) FROM pg_auth_members m
|
||||
JOIN pg_roles g ON g.oid = m.roleid
|
||||
JOIN pg_roles u ON u.oid = m.member
|
||||
WHERE g.rolname = $1 AND u.rolname = $2`, db.RuntimeGroupRole, login)
|
||||
}
|
||||
inherits := func() bool {
|
||||
return q.String(`SELECT pg_has_role($1, $2, 'USAGE')::text`, login, db.RuntimeGroupRole) == "true"
|
||||
}
|
||||
if inherits() || members() != 0 {
|
||||
t.Fatalf("before the grant, %q must hold no membership (members=%d)", login, members())
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
for _, attempt := range []string{"first grant", "second grant"} {
|
||||
if err := db.GrantRuntimeRole(ctx, q.Tx(), login); err != nil {
|
||||
t.Fatalf("%s: %v", attempt, err)
|
||||
}
|
||||
if !inherits() {
|
||||
t.Errorf("%s: %q does not inherit %s", attempt, login, db.RuntimeGroupRole)
|
||||
}
|
||||
if n := members(); n != 1 {
|
||||
t.Errorf("%s: %q has %d memberships in %s, want 1", attempt, login, n, db.RuntimeGroupRole)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
|
||||
package db_test
|
||||
|
||||
// Tests of the migration pool (openspec change runtime-database-user, design
|
||||
// D2 rule 2, task 4.8). It lives in a db_test file because it exercises
|
||||
// db.ConnectMigration and compares schemaSnapshot, which cmd cannot import.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/db"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/migrate"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest"
|
||||
)
|
||||
|
||||
// scratchDatabaseName is the database a scratch address names.
|
||||
func scratchDatabaseName(t *testing.T, dsn string) string {
|
||||
t.Helper()
|
||||
u, err := url.Parse(dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("parse %q: %v", dsn, err)
|
||||
}
|
||||
return strings.TrimPrefix(u.Path, "/")
|
||||
}
|
||||
|
||||
// poolTestRole creates a role that the test drops, for a database setting or a
|
||||
// connection string to name.
|
||||
func poolTestRole(t *testing.T) string {
|
||||
t.Helper()
|
||||
admin := dbtest.Connect(t, dbtest.DSN(t))
|
||||
role := dbtest.Name("mc_pool_role_")
|
||||
dbtest.On(t, admin).Exec("CREATE ROLE " + role)
|
||||
t.Cleanup(func() {
|
||||
if _, err := admin.ExecContext(context.Background(), "DROP ROLE IF EXISTS "+role); err != nil {
|
||||
t.Errorf("drop role %s: %v", role, err)
|
||||
}
|
||||
})
|
||||
return role
|
||||
}
|
||||
|
||||
// carrying returns dsn with the parameters a db-migrate-dsn copied from the
|
||||
// runtime string would carry: the application's search_path and a role.
|
||||
func carrying(dsn, role string) string {
|
||||
sep := "?"
|
||||
if strings.Contains(dsn, "?") {
|
||||
sep = "&"
|
||||
}
|
||||
return dsn + sep + "search_path=core,public&role=" + role
|
||||
}
|
||||
|
||||
// openMigrationPool opens the migration pool on dsn and closes it with the test.
|
||||
func openMigrationPool(t *testing.T, dsn string) *sql.DB {
|
||||
t.Helper()
|
||||
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
|
||||
pool, err := db.ConnectMigration(context.Background(), logger, db.DefaultDBConfig(dsn))
|
||||
if err != nil {
|
||||
t.Fatalf("ConnectMigration: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = pool.Close() })
|
||||
return pool
|
||||
}
|
||||
|
||||
// alterDatabase runs ALTER DATABASE name SET setting on the cluster.
|
||||
func alterDatabase(t *testing.T, name, setting string) {
|
||||
t.Helper()
|
||||
admin := dbtest.Connect(t, dbtest.DSN(t))
|
||||
dbtest.On(t, admin).Exec("ALTER DATABASE " + name + " SET " + setting)
|
||||
}
|
||||
|
||||
// TestMigrationPoolIgnoresADatabaseSearchPath covers task 4.8: a search_path
|
||||
// or a role that the database, or the connection string, carries does not
|
||||
// reach the migration pool, which opens with search_path=public and
|
||||
// role=none, and the schema it migrates equals a plain replay's.
|
||||
func TestMigrationPoolIgnoresADatabaseSearchPath(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
role := poolTestRole(t)
|
||||
|
||||
t.Run("search_path of the database", func(t *testing.T) {
|
||||
scratch := dbtest.Scratch(t)
|
||||
decoy := dbtest.Name("mc_decoy_")
|
||||
dbtest.On(t, dbtest.Connect(t, scratch)).Exec("CREATE SCHEMA " + decoy)
|
||||
alterDatabase(t, scratchDatabaseName(t, scratch), "search_path = "+decoy+", public")
|
||||
|
||||
pool := openMigrationPool(t, carrying(scratch, role))
|
||||
q := dbtest.On(t, pool)
|
||||
if got := q.String("SHOW search_path"); got != "public" {
|
||||
t.Fatalf("SHOW search_path on the migration pool = %q, want public", got)
|
||||
}
|
||||
if err := db.RunMigrations(pool, migrate.Sources()); err != nil {
|
||||
t.Fatalf("migrate through the migration pool: %v", err)
|
||||
}
|
||||
const outside = `SELECT n.nspname || '.' || c.relname FROM pg_class c
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE c.relname LIKE 'goose\_db\_version\_%' AND c.relkind = 'r' AND n.nspname <> 'public'`
|
||||
if got := q.Strings(outside); len(got) > 0 {
|
||||
t.Errorf("ledgers outside public: %v", got)
|
||||
}
|
||||
pending, err := db.PendingMigrations(ctx, pool, migrate.Sources())
|
||||
if err != nil {
|
||||
t.Fatalf("PendingMigrations: %v", err)
|
||||
}
|
||||
if len(pending) > 0 {
|
||||
t.Errorf("the ledgers in public leave pending: %s", db.PendingDetail(pending))
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("role of the database", func(t *testing.T) {
|
||||
scratch := dbtest.Scratch(t)
|
||||
alterDatabase(t, scratchDatabaseName(t, scratch), "role = '"+role+"'")
|
||||
want := dbtest.On(t, dbtest.Connect(t, dbtest.DSN(t))).String("SELECT current_user::text")
|
||||
|
||||
pool := openMigrationPool(t, carrying(scratch, role))
|
||||
if got := dbtest.On(t, pool).String("SELECT current_user::text"); got != want {
|
||||
t.Errorf("SELECT current_user on the migration pool = %q, want the migration login %q", got, want)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("the schema equals a plain replay's", func(t *testing.T) {
|
||||
viaPool := openMigrationPool(t, dbtest.Scratch(t))
|
||||
if err := db.RunMigrations(viaPool, migrate.Sources()); err != nil {
|
||||
t.Fatalf("migrate through the migration pool: %v", err)
|
||||
}
|
||||
plain := dbtest.Connect(t, dbtest.Scratch(t))
|
||||
if err := db.RunMigrations(plain, migrate.Sources()); err != nil {
|
||||
t.Fatalf("plain replay: %v", err)
|
||||
}
|
||||
got, want := schemaSnapshot(t, ctx, viaPool), schemaSnapshot(t, ctx, plain)
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("the schema migrated through the migration pool differs from a plain replay's:\n%s", snapshotDiff(want, got))
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -4,10 +4,15 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"embed"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/pressly/goose/v3"
|
||||
)
|
||||
@@ -142,3 +147,170 @@ func BaseSources() []MigrationSource {
|
||||
{Name: "core", Migrations: embedMigrations, Dir: "migrations"},
|
||||
}
|
||||
}
|
||||
|
||||
// queryLedgerState reads, for the ledger whose qualified name is $1, who is
|
||||
// asking, whether the ledger exists, and whether that user may read it. Every
|
||||
// function is qualified with pg_catalog (design D2, the qualification
|
||||
// principle).
|
||||
const queryLedgerState = `SELECT current_user::text,
|
||||
pg_catalog.to_regclass($1) IS NOT NULL,
|
||||
COALESCE(pg_catalog.has_table_privilege(current_user, pg_catalog.to_regclass($1), 'SELECT'), false)`
|
||||
|
||||
// queryLedgerApplied is the template of the read of one ledger; its single
|
||||
// verb is the ledger's table name, which comes from a source's name and never
|
||||
// from input. The ledger is named with its schema.
|
||||
const queryLedgerApplied = `SELECT version_id FROM public.%s WHERE is_applied`
|
||||
|
||||
// PendingStream is one migration stream whose ledger does not record every
|
||||
// embedded version.
|
||||
type PendingStream struct {
|
||||
// Stream is the source's name, which is also its schema's.
|
||||
Stream string
|
||||
// Versions are the embedded versions the ledger does not record as
|
||||
// applied, ascending. A stream with no ledger lists every embedded
|
||||
// version; a stream whose ledger the user cannot read lists none.
|
||||
Versions []int64
|
||||
// NoLedger is true when the ledger does not exist.
|
||||
NoLedger bool
|
||||
// UnreadableBy is the user who cannot read an existing ledger; it is
|
||||
// empty when the ledger is readable.
|
||||
UnreadableBy string
|
||||
}
|
||||
|
||||
// migrationFile matches a migration file's name and captures its version.
|
||||
var migrationFile = regexp.MustCompile(`^(\d+)_.*\.sql$`)
|
||||
|
||||
// PendingMigrations returns the streams of sources whose ledger lacks an
|
||||
// embedded version, in source order; a database that is current returns none.
|
||||
// It reads each ledger as public.goose_db_version_<stream> and changes
|
||||
// nothing.
|
||||
//
|
||||
// It does not use goose's provider, for two reasons. Provider.HasPending
|
||||
// creates a ledger it does not find, and a check that writes is wrong on the
|
||||
// runtime connection, whose user may not create one. And it looks for the
|
||||
// ledger in current_schema(), which is core on the runtime connection, while
|
||||
// the ledgers live in public because the migration connection opens with
|
||||
// search_path=public.
|
||||
//
|
||||
// The names are qualified on purpose. On the runtime connection's
|
||||
// search_path (core,public) Postgres looks in the session's pg_temp schema
|
||||
// first, so a runtime user with arbitrary SQL could create a temporary table
|
||||
// named like a ledger and make an unqualified lookup read it. The functions
|
||||
// are qualified for the same reason as every statement the console runs with
|
||||
// privilege (design D2): a function a role planted in a schema on the path
|
||||
// would otherwise run in the caller's session. The transaction begins with
|
||||
// the search-path pin for the operators and for any name a later edit leaves
|
||||
// bare.
|
||||
func PendingMigrations(ctx context.Context, q Beginner, sources []MigrationSource) ([]PendingStream, error) {
|
||||
tx, err := Begin(ctx, q, pinned)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
var pending []PendingStream
|
||||
for _, src := range sources {
|
||||
stream, err := pendingStream(ctx, tx, src)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if stream != nil {
|
||||
pending = append(pending, *stream)
|
||||
}
|
||||
}
|
||||
return pending, nil
|
||||
}
|
||||
|
||||
// pendingStream compares one stream's embedded versions with its ledger and
|
||||
// returns nil when the ledger records them all.
|
||||
func pendingStream(ctx context.Context, tx *sql.Tx, src MigrationSource) (*PendingStream, error) {
|
||||
embedded, err := embeddedVersions(src)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
table := ledgerTableName(src.Name)
|
||||
var user string
|
||||
var exists, readable bool
|
||||
if err := tx.QueryRowContext(ctx, queryLedgerState, "public."+table).Scan(&user, &exists, &readable); err != nil {
|
||||
return nil, fmt.Errorf("checking the %s ledger: %w", src.Name, err)
|
||||
}
|
||||
switch {
|
||||
case !exists:
|
||||
return &PendingStream{Stream: src.Name, Versions: embedded, NoLedger: true}, nil
|
||||
case !readable:
|
||||
return &PendingStream{Stream: src.Name, UnreadableBy: user}, nil
|
||||
}
|
||||
rows, err := tx.QueryContext(ctx, fmt.Sprintf(queryLedgerApplied, table))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading the %s ledger: %w", src.Name, err)
|
||||
}
|
||||
defer rows.Close()
|
||||
applied := map[int64]bool{}
|
||||
for rows.Next() {
|
||||
var v int64
|
||||
if err := rows.Scan(&v); err != nil {
|
||||
return nil, fmt.Errorf("reading the %s ledger: %w", src.Name, err)
|
||||
}
|
||||
applied[v] = true
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("reading the %s ledger: %w", src.Name, err)
|
||||
}
|
||||
var missing []int64
|
||||
for _, v := range embedded {
|
||||
if !applied[v] {
|
||||
missing = append(missing, v)
|
||||
}
|
||||
}
|
||||
if len(missing) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
return &PendingStream{Stream: src.Name, Versions: missing}, nil
|
||||
}
|
||||
|
||||
// embeddedVersions lists the version numbers of a source's migration files,
|
||||
// ascending.
|
||||
func embeddedVersions(src MigrationSource) ([]int64, error) {
|
||||
entries, err := fs.ReadDir(src.Migrations, src.Dir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read %s migrations: %w", src.Name, err)
|
||||
}
|
||||
var versions []int64
|
||||
for _, e := range entries {
|
||||
m := migrationFile.FindStringSubmatch(e.Name())
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
v, err := strconv.ParseInt(m[1], 10, 64)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("migration %s: %w", e.Name(), err)
|
||||
}
|
||||
versions = append(versions, v)
|
||||
}
|
||||
slices.Sort(versions)
|
||||
return versions, nil
|
||||
}
|
||||
|
||||
// PendingDetail renders pending as the detail of the start refusal: one
|
||||
// entry per stream with no ledger or an unreadable one, then one list of the
|
||||
// versions the readable ledgers lack, separated by semicolons.
|
||||
func PendingDetail(pending []PendingStream) string {
|
||||
var parts, behind []string
|
||||
for _, p := range pending {
|
||||
switch {
|
||||
case p.NoLedger:
|
||||
parts = append(parts, p.Stream+" has no migration ledger")
|
||||
case p.UnreadableBy != "":
|
||||
parts = append(parts, p.UnreadableBy+" cannot read "+ledgerTableName(p.Stream))
|
||||
default:
|
||||
versions := make([]string, len(p.Versions))
|
||||
for i, v := range p.Versions {
|
||||
versions[i] = strconv.FormatInt(v, 10)
|
||||
}
|
||||
behind = append(behind, p.Stream+" "+strings.Join(versions, ", "))
|
||||
}
|
||||
}
|
||||
if len(behind) > 0 {
|
||||
parts = append(parts, "pending migrations: "+strings.Join(behind, "; "))
|
||||
}
|
||||
return strings.Join(parts, "; ")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,221 @@
|
||||
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
|
||||
package db_test
|
||||
|
||||
// Tests of the pending check (openspec change runtime-database-user, design
|
||||
// D10, task 4.8). Every row migrates a scratch database as the superuser
|
||||
// TEST_DATABASE_URL login and then reads the ledgers as console_runtime, on a
|
||||
// connection whose search_path is the application's, core,public.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/db"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/migrate"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest"
|
||||
)
|
||||
|
||||
// coreGroupRoleVersion is core's group-role migration, the last version a
|
||||
// stream can stop at and still leave its ledger readable to console_runtime.
|
||||
const coreGroupRoleVersion = 29
|
||||
|
||||
var pendingVersionPrefix = regexp.MustCompile(`^(\d+)_`)
|
||||
|
||||
// embeddedVersionsOf lists the version numbers of a source's files, ascending.
|
||||
func embeddedVersionsOf(t *testing.T, src db.MigrationSource) []int64 {
|
||||
t.Helper()
|
||||
entries, err := fs.ReadDir(src.Migrations, src.Dir)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s migrations: %v", src.Name, err)
|
||||
}
|
||||
var versions []int64
|
||||
for _, e := range entries {
|
||||
if m := pendingVersionPrefix.FindStringSubmatch(e.Name()); m != nil {
|
||||
v, err := strconv.ParseInt(m[1], 10, 64)
|
||||
if err != nil {
|
||||
t.Fatalf("migration %s: %v", e.Name(), err)
|
||||
}
|
||||
versions = append(versions, v)
|
||||
}
|
||||
}
|
||||
slices.Sort(versions)
|
||||
return versions
|
||||
}
|
||||
|
||||
// migrateStreams migrates every stream of scratch in source order. A stream in
|
||||
// stop is migrated up to that version only; a stream in skip is not touched,
|
||||
// so it has no ledger.
|
||||
func migrateStreams(t *testing.T, scratch *sql.DB, stop map[string]int64, skip ...string) {
|
||||
t.Helper()
|
||||
for _, src := range migrate.Sources() {
|
||||
if slices.Contains(skip, src.Name) {
|
||||
continue
|
||||
}
|
||||
version, short := stop[src.Name]
|
||||
if !short {
|
||||
version = embeddedVersionsOf(t, src)[len(embeddedVersionsOf(t, src))-1]
|
||||
}
|
||||
dbtest.UpTo(t, scratch, src, version)
|
||||
}
|
||||
}
|
||||
|
||||
// asRuntimeConn pins a connection of scratch that reads as console_runtime
|
||||
// with the application's search_path, and closes it with the test.
|
||||
func asRuntimeConn(t *testing.T, scratch *sql.DB) *sql.Conn {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
conn, err := scratch.Conn(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("pin a connection: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = conn.Close() })
|
||||
for _, stmt := range []string{"SET ROLE " + db.RuntimeGroupRole, "SET search_path = core,public"} {
|
||||
if _, err := conn.ExecContext(ctx, stmt); err != nil {
|
||||
t.Fatalf("%s: %v", stmt, err)
|
||||
}
|
||||
}
|
||||
return conn
|
||||
}
|
||||
|
||||
// pendingText renders what PendingMigrations found as the detail of the start
|
||||
// refusal, which names the stream and each of its versions.
|
||||
func pendingText(t *testing.T, conn *sql.Conn) string {
|
||||
t.Helper()
|
||||
pending, err := db.PendingMigrations(context.Background(), conn, migrate.Sources())
|
||||
if err != nil {
|
||||
t.Fatalf("PendingMigrations: %v", err)
|
||||
}
|
||||
return db.PendingDetail(pending)
|
||||
}
|
||||
|
||||
// pendingRow is one scratch database and what the check must say about it.
|
||||
type pendingRow struct {
|
||||
name string
|
||||
// build migrates the scratch database and may change it further.
|
||||
build func(t *testing.T, scratch *sql.DB)
|
||||
// onConn runs on the checking connection before the check.
|
||||
onConn func(t *testing.T, conn *sql.Conn)
|
||||
// want is the detail PendingDetail renders; empty is nothing pending.
|
||||
want func(t *testing.T) string
|
||||
}
|
||||
|
||||
// temporaryLedger creates, on the checking connection, a temporary table named
|
||||
// like the stream's ledger. It sits in pg_temp, which an unqualified name
|
||||
// meets before any other schema.
|
||||
func temporaryLedger(stream string) func(*testing.T, *sql.Conn) {
|
||||
return func(t *testing.T, conn *sql.Conn) {
|
||||
t.Helper()
|
||||
ddl := `CREATE TEMPORARY TABLE goose_db_version_` + stream + ` (version_id bigint, is_applied boolean)`
|
||||
if _, err := conn.ExecContext(context.Background(), ddl); err != nil {
|
||||
t.Fatalf("create the temporary table: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func pendingRows() []pendingRow {
|
||||
stripe := func(t *testing.T) []int64 { return embeddedVersionsOf(t, sourceNamed(t, "stripe")) }
|
||||
return []pendingRow{
|
||||
{
|
||||
name: "migrated in full",
|
||||
build: func(t *testing.T, s *sql.DB) { migrateStreams(t, s, nil) },
|
||||
want: func(*testing.T) string { return "" },
|
||||
},
|
||||
{
|
||||
name: "a stream stopped short lists its missing version",
|
||||
build: func(t *testing.T, s *sql.DB) {
|
||||
v := stripe(t)
|
||||
migrateStreams(t, s, map[string]int64{"stripe": v[len(v)-2]})
|
||||
},
|
||||
want: func(t *testing.T) string {
|
||||
v := stripe(t)
|
||||
return fmt.Sprintf("pending migrations: stripe %d", v[len(v)-1])
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "a stream stopped short by several versions lists them all",
|
||||
build: func(t *testing.T, s *sql.DB) { migrateStreams(t, s, map[string]int64{"core": coreGroupRoleVersion}) },
|
||||
want: func(t *testing.T) string {
|
||||
var later []string
|
||||
for _, v := range embeddedVersionsOf(t, sourceNamed(t, "core")) {
|
||||
if v > coreGroupRoleVersion {
|
||||
later = append(later, strconv.FormatInt(v, 10))
|
||||
}
|
||||
}
|
||||
return "pending migrations: core " + strings.Join(later, ", ")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "no discourse ledger lists every discourse version",
|
||||
build: func(t *testing.T, s *sql.DB) { migrateStreams(t, s, nil, "discourse") },
|
||||
want: func(*testing.T) string { return "discourse has no migration ledger" },
|
||||
},
|
||||
{
|
||||
name: "a ledger console_runtime cannot read",
|
||||
build: func(t *testing.T, s *sql.DB) {
|
||||
migrateStreams(t, s, nil)
|
||||
dbtest.On(t, s).Exec("REVOKE SELECT ON public.goose_db_version_stripe FROM " + db.RuntimeGroupRole)
|
||||
},
|
||||
want: func(*testing.T) string { return "console_runtime cannot read goose_db_version_stripe" },
|
||||
},
|
||||
{
|
||||
name: "a temporary table named like a ledger changes nothing",
|
||||
build: func(t *testing.T, s *sql.DB) { migrateStreams(t, s, nil) },
|
||||
onConn: temporaryLedger("core"),
|
||||
want: func(*testing.T) string { return "" },
|
||||
},
|
||||
{
|
||||
name: "a temporary table named like a missing ledger does not stand in for it",
|
||||
build: func(t *testing.T, s *sql.DB) { migrateStreams(t, s, nil, "discourse") },
|
||||
onConn: temporaryLedger("discourse"),
|
||||
want: func(*testing.T) string { return "discourse has no migration ledger" },
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// TestPendingMigrations covers task 4.8: the pending check reports the
|
||||
// versions a ledger lacks, a missing ledger as every version, and an
|
||||
// unreadable ledger as unreadable, and reads the ledgers in public whatever
|
||||
// the connection's search_path holds.
|
||||
func TestPendingMigrations(t *testing.T) {
|
||||
for _, row := range pendingRows() {
|
||||
t.Run(row.name, func(t *testing.T) {
|
||||
scratch := dbtest.Connect(t, dbtest.Scratch(t))
|
||||
row.build(t, scratch)
|
||||
conn := asRuntimeConn(t, scratch)
|
||||
if row.onConn != nil {
|
||||
row.onConn(t, conn)
|
||||
}
|
||||
if got, want := pendingText(t, conn), row.want(t); got != want {
|
||||
t.Errorf("pending = %q, want %q", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestPendingMigrationsListsEveryVersionOfAMissingLedger requires a stream
|
||||
// with no ledger to list every embedded version, which the detail text does
|
||||
// not show.
|
||||
func TestPendingMigrationsListsEveryVersionOfAMissingLedger(t *testing.T) {
|
||||
scratch := dbtest.Connect(t, dbtest.Scratch(t))
|
||||
migrateStreams(t, scratch, nil, "discourse")
|
||||
conn := asRuntimeConn(t, scratch)
|
||||
pending, err := db.PendingMigrations(context.Background(), conn, migrate.Sources())
|
||||
if err != nil {
|
||||
t.Fatalf("PendingMigrations: %v", err)
|
||||
}
|
||||
if len(pending) != 1 || pending[0].Stream != "discourse" || !pending[0].NoLedger {
|
||||
t.Fatalf("pending = %+v, want the discourse stream with no ledger", pending)
|
||||
}
|
||||
if want := embeddedVersionsOf(t, sourceNamed(t, "discourse")); !slices.Equal(pending[0].Versions, want) {
|
||||
t.Errorf("discourse versions = %v, want every embedded version %v", pending[0].Versions, want)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,12 @@
|
||||
|
||||
package db
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// RuntimeGroupRole is the NOLOGIN group role the console's runtime user
|
||||
// holds. Core migration 00029 creates it and grants it core_writer and SELECT
|
||||
// on the core goose ledger, and core 00031 grants it EXECUTE on
|
||||
@@ -11,3 +17,89 @@ package db
|
||||
// stream's writer and ledger. No migration grants it to a login: the console
|
||||
// does that at start.
|
||||
const RuntimeGroupRole = "console_runtime"
|
||||
|
||||
// RuntimeUserRefusal is a start refusal about the runtime user or the schema
|
||||
// it meets: its Error is the sentence the design's copy table gives, which
|
||||
// start logs as the error of "refused to start".
|
||||
type RuntimeUserRefusal struct {
|
||||
// Message is the whole sentence.
|
||||
Message string
|
||||
}
|
||||
|
||||
func (r *RuntimeUserRefusal) Error() string { return r.Message }
|
||||
|
||||
// SchemaBehindRefusal is the refusal for a schema that is behind this
|
||||
// release (copy 4), with the detail PendingDetail renders.
|
||||
func SchemaBehindRefusal(pending []PendingStream) *RuntimeUserRefusal {
|
||||
return &RuntimeUserRefusal{Message: "the schema is behind this release (" + PendingDetail(pending) +
|
||||
"); run member-console migrate up with db-migrate-dsn set, then start again"}
|
||||
}
|
||||
|
||||
// queryPinSearchPath begins every transaction in which the console runs a
|
||||
// check query or the boot grant: with this path no schema a role could have
|
||||
// planted in is a candidate for an operator or function name (design D2, the
|
||||
// search-path pin). pg_temp is a schema name here, not a catalog relation.
|
||||
const queryPinSearchPath = `SET LOCAL search_path = pg_catalog, pg_temp`
|
||||
|
||||
// pinned opens a transaction that runs the search-path pin first.
|
||||
var pinned = FirstRuns(func(ctx context.Context, tx *sql.Tx) error {
|
||||
_, err := tx.ExecContext(ctx, queryPinSearchPath)
|
||||
return err
|
||||
})
|
||||
|
||||
// queryCurrentUser reads the login a connection runs as.
|
||||
const queryCurrentUser = `SELECT current_user::text`
|
||||
|
||||
// CurrentUser returns the login that q's connection runs as. The boot grant
|
||||
// takes its name from the runtime connection, not from the connection string,
|
||||
// so a string whose user part is URL-encoded, defaulted by the driver or
|
||||
// rewritten by a pooler still names the user the console serves as.
|
||||
func CurrentUser(ctx context.Context, q Beginner) (string, error) {
|
||||
tx, err := Begin(ctx, q, pinned)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
var user string
|
||||
if err := tx.QueryRowContext(ctx, queryCurrentUser).Scan(&user); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return user, nil
|
||||
}
|
||||
|
||||
// queryGrantSetting passes the runtime user's name into a setting that lasts
|
||||
// until the transaction ends, so no Go code concatenates SQL.
|
||||
const queryGrantSetting = `SELECT pg_catalog.set_config('member_console.runtime_user', $1, true)`
|
||||
|
||||
// queryGrantRuntimeRole builds the grant in the database and quotes the name
|
||||
// with %I. WITH INHERIT TRUE makes the grant pass privileges whatever the
|
||||
// login's own INHERIT attribute says: a login created NOINHERIT would
|
||||
// otherwise inherit nothing through it.
|
||||
const queryGrantRuntimeRole = `DO $$
|
||||
BEGIN
|
||||
EXECUTE pg_catalog.format('GRANT console_runtime TO %I WITH INHERIT TRUE',
|
||||
pg_catalog.current_setting('member_console.runtime_user'));
|
||||
END
|
||||
$$`
|
||||
|
||||
// GrantRuntimeRole grants console_runtime to the login name in tx, which
|
||||
// belongs to the migration connection (design D8). It takes a transaction
|
||||
// because the setting that carries the name lasts until the transaction ends;
|
||||
// cmd runs it through InTx. Its first statement is the search-path pin, and
|
||||
// every function it calls is qualified: the migration connection is a
|
||||
// superuser, and a role that can create objects in public could otherwise
|
||||
// plant a function with the argument types of format. Granting a membership
|
||||
// that exists from the same grantor changes nothing, so the grant is
|
||||
// idempotent.
|
||||
func GrantRuntimeRole(ctx context.Context, tx *sql.Tx, name string) error {
|
||||
if _, err := tx.ExecContext(ctx, queryPinSearchPath); err != nil {
|
||||
return fmt.Errorf("pinning the search path: %w", err)
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, queryGrantSetting, name); err != nil {
|
||||
return fmt.Errorf("passing the user's name: %w", err)
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, queryGrantRuntimeRole); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
|
||||
package db_test
|
||||
|
||||
// Test of the startup parameters both pools set (openspec change
|
||||
// runtime-database-user, design D2 rule 2, review finding on the string
|
||||
// rewrite). It runs on a real connection: what matters is the current_user and
|
||||
// the search_path the server reports, not the text handed to the driver.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/db"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest"
|
||||
)
|
||||
|
||||
// keywordValue returns the keyword/value form of a postgres:// address, with
|
||||
// extra appended, each value in single quotes.
|
||||
func keywordValue(t *testing.T, dsn, extra string) string {
|
||||
t.Helper()
|
||||
u, err := url.Parse(dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("parse %q: %v", dsn, err)
|
||||
}
|
||||
quote := func(v string) string {
|
||||
return "'" + strings.NewReplacer(`\`, `\\`, `'`, `\'`).Replace(v) + "'"
|
||||
}
|
||||
password, _ := u.User.Password()
|
||||
parts := []string{
|
||||
"host=" + quote(u.Hostname()), "port=" + quote(u.Port()),
|
||||
"user=" + quote(u.User.Username()), "password=" + quote(password),
|
||||
"dbname=" + quote(strings.TrimPrefix(u.Path, "/")),
|
||||
"sslmode=" + quote(u.Query().Get("sslmode")),
|
||||
}
|
||||
return strings.Join(parts, " ") + " " + extra
|
||||
}
|
||||
|
||||
// withQuery returns dsn with query appended to its query string.
|
||||
func withQuery(dsn, query string) string {
|
||||
if strings.Contains(dsn, "?") {
|
||||
return dsn + "&" + query
|
||||
}
|
||||
return dsn + "?" + query
|
||||
}
|
||||
|
||||
// startupCase is one connection string and what the server must report on
|
||||
// every connection of both pools.
|
||||
type startupCase struct {
|
||||
name string
|
||||
dsn string
|
||||
// runtimePath is the search_path the runtime pool keeps; the migration
|
||||
// pool always reports public.
|
||||
runtimePath string
|
||||
// setting is a server setting the string also carries, with its expected
|
||||
// value: it proves the rest of the string is not corrupted.
|
||||
setting, settingWant string
|
||||
}
|
||||
|
||||
func startupCases(t *testing.T, role string) []startupCase {
|
||||
base := dbtest.DSN(t)
|
||||
return []startupCase{
|
||||
{"mixed-case keys", withQuery(base, "ROLE="+role+"&Search_Path=decoy_a"), "decoy_a", "", ""},
|
||||
{"repeated keys", withQuery(base, "role=other&role="+role+"&search_path=decoy_b&search_path=decoy_c"), "decoy_c", "", ""},
|
||||
{"repeated keys in two cases", withQuery(base, "role="+role+"&ROLE="+role+"&search_path=decoy_d&SEARCH_PATH=decoy_d"), "decoy_d", "", ""},
|
||||
{"keys after a hash", withQuery(base, "application_name=a#&role="+role+"&search_path=decoy_e"), "decoy_e", "", ""},
|
||||
{"options with -c and a space", withQuery(base, "options=-c%20role%3D"+role+"%20-c%20search_path%3Ddecoy_f"), "core,public", "", ""},
|
||||
{"options with another switch", withQuery(base, "options=-c%20lock_timeout%3D5s%20-c%20role%3D"+role), "core,public", "lock_timeout", "5s"},
|
||||
{"options and mixed-case keys", withQuery(base, "options=-c%20role%3D"+role+"&Role="+role+"&search_path=decoy_g"), "decoy_g", "", ""},
|
||||
{"a space in a value", withQuery(base, "application_name=a%20role%3Db"), "core,public", "application_name", "a role=b"},
|
||||
{"keyword/value", keywordValue(t, base, "role="+role+" search_path=decoy_h"), "decoy_h", "", ""},
|
||||
{"keyword/value, mixed-case keys", keywordValue(t, base, "ROLE="+role+" Search_Path=decoy_i"), "decoy_i", "", ""},
|
||||
{"keyword/value, quoted value", keywordValue(t, base, "application_name='a role=b' options='-c role="+role+"'"), "core,public", "application_name", "a role=b"},
|
||||
{"no setting", base, "core,public", "", ""},
|
||||
}
|
||||
}
|
||||
|
||||
// checkStartupConnections holds several connections of pool at once, because
|
||||
// the startup message writes the parameters in map order and one connection
|
||||
// proves little, and checks what the server reports on each.
|
||||
func checkStartupConnections(t *testing.T, pool *sql.DB, wantPath string, tc startupCase) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
var conns []*sql.Conn
|
||||
defer func() {
|
||||
for _, c := range conns {
|
||||
_ = c.Close()
|
||||
}
|
||||
}()
|
||||
for range 8 {
|
||||
c, err := pool.Conn(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("Conn: %v", err)
|
||||
}
|
||||
conns = append(conns, c)
|
||||
}
|
||||
for i, c := range conns {
|
||||
var login, user, path string
|
||||
if err := c.QueryRowContext(ctx,
|
||||
"SELECT session_user::text, current_user::text, current_setting('search_path')").
|
||||
Scan(&login, &user, &path); err != nil {
|
||||
t.Fatalf("connection %d: %v", i, err)
|
||||
}
|
||||
if user != login {
|
||||
t.Errorf("connection %d: current_user = %q, want the login %q", i, user, login)
|
||||
}
|
||||
if path != wantPath {
|
||||
t.Errorf("connection %d: search_path = %q, want %q", i, path, wantPath)
|
||||
}
|
||||
if tc.setting == "" {
|
||||
continue
|
||||
}
|
||||
var setting string
|
||||
if err := c.QueryRowContext(ctx, "SELECT current_setting($1)", tc.setting).Scan(&setting); err != nil {
|
||||
t.Fatalf("connection %d: %v", i, err)
|
||||
}
|
||||
if setting != tc.settingWant {
|
||||
t.Errorf("connection %d: %s = %q, want %q", i, tc.setting, setting, tc.settingWant)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestStartupParamsWinOverTheString covers every spelling of a search_path or a
|
||||
// role in a connection string that the parser reads as a setting: the server
|
||||
// reports the login as current_user and the pool's search_path on every
|
||||
// connection of both pools.
|
||||
func TestStartupParamsWinOverTheString(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
role := poolTestRole(t)
|
||||
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
|
||||
pools := []struct {
|
||||
name string
|
||||
open func(dsn string) (*sql.DB, error)
|
||||
}{
|
||||
{"runtime", func(dsn string) (*sql.DB, error) {
|
||||
return db.Connect(ctx, logger, db.DefaultDBConfig(dsn))
|
||||
}},
|
||||
{"migration", func(dsn string) (*sql.DB, error) {
|
||||
return db.ConnectMigration(ctx, logger, db.DefaultDBConfig(dsn))
|
||||
}},
|
||||
}
|
||||
for _, tc := range startupCases(t, role) {
|
||||
for _, p := range pools {
|
||||
t.Run(p.name+"/"+tc.name, func(t *testing.T) {
|
||||
pool, err := p.open(tc.dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = pool.Close() })
|
||||
wantPath := "public"
|
||||
if p.name == "runtime" {
|
||||
wantPath = tc.runtimePath
|
||||
}
|
||||
checkStartupConnections(t, pool, wantPath, tc)
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -15,8 +15,9 @@
|
||||
base-url: "https://console.example.com"
|
||||
port: "8080"
|
||||
|
||||
# PostgreSQL connection string.
|
||||
# PostgreSQL connection string of the runtime user: not a superuser, owns nothing.
|
||||
db-dsn: "postgres://user:password@localhost:5432/member_console?sslmode=disable"
|
||||
#db-migrate-dsn: "" # the migration user's string; start runs migrations only when it is set
|
||||
|
||||
# Valkey/Redis address for the server-side session store.
|
||||
valkey-addr: "localhost:6379"
|
||||
|
||||
@@ -106,23 +106,23 @@ The plan-act obligation lock (items 3.10 to 3.14) lives in this section, not in
|
||||
|
||||
This section's `make test` passes only after the documented stack respin of item 4.9, which the implementer performs in the worktree before the section's suite runs: the guard refuses a `test/.env` without `MC_DB_MIGRATE_DSN`, and `postgres-seed` creates the runtime login only on `docker compose up`. "Every commit passes `make test`" therefore holds from a stack so respun, not from a fresh checkout beside an old `test/.env`.
|
||||
|
||||
- [ ] 4.1 Keys (design D1): `startCmd` registers `db-migrate-dsn` and `db-migrate-dsn-file` (copies 23, 24), and the help of `db-dsn` and `db-dsn-file` changes (copies 21, 22); `databaseSecretPairs` gains `db-migrate-dsn`/`db-migrate-dsn-file`; `ValidateStart` refuses a malformed `db-migrate-dsn` (copy 14) and treats it as optional.
|
||||
- [ ] 4.2 `internal/db`: `PendingMigrations(ctx, q, sources)` (design D10) in `migrations.go`, which begins its transaction with `SET LOCAL search_path = pg_catalog, pg_temp` and reads each ledger as `public.goose_db_version_<stream>` by qualified name and calls `pg_catalog.to_regclass` and `pg_catalog.has_table_privilege` (design D2, the qualification principle), with a doc comment that says why it does not use goose's provider and why the names are qualified; `ConnectMigration(ctx, logger, config)`, which opens the migration pool with the startup parameters `search_path=public` and `role=none` as runtime parameters of the connection string, replacing any `search_path` or `role` the string carries (`ensureSearchPath`, `internal/db/database.go:43-60`, keeps an operator's value and is not reused, so a `db-migrate-dsn` copied from the runtime string cannot open with `core,public`); the runtime pool's open gains `role=none`, always replacing the string's value, beside its `search_path=core,public` (`ConnectPlain` stays for `dbConfig`); `CurrentUser` and `GrantRuntimeRole(ctx, tx, name)` (design D8, which begins with `SET LOCAL search_path = pg_catalog, pg_temp`, and whose statement calls `pg_catalog.set_config`, `pg_catalog.format` and `pg_catalog.current_setting`; it takes a transaction because the setting lasts until the transaction ends, and `cmd` runs it through `db.InTx`) in `runtime_user.go`, and the refusal type `RuntimeUserRefusal`. Every SQL string of these functions is a constant whose name begins `query` (item 5.8 scans them). `ConnectAndMigrate` is deleted.
|
||||
- [ ] 4.3 `cmd/database.go` (design D11): modes `dbRuntime`, `dbMigration` and `dbConfig`. `dbMigration` opens `db-migrate-dsn` with `search_path=public` and `role=none` (design D2, rule 2) and refuses without it (copy 12). `dbConfig` is today's `dbPlain` on `db-dsn`. `dbRuntime` opens the runtime pool with `role=none` beside its `search_path`, and runs the boot sequence's steps 2, 3, 6 and 7, step 5's migrations, and the pending check in both modes (copies 4, 4b to 4d, 10, 18, 20); the other checks, and steps 4 and 8 in full, are section 5's, with copy 4a, which belongs to check 5. `start` and `seed-demo` open `dbRuntime`; `start` logs a refusal with copy 17; the `migrate` commands open `dbMigration`; the `config` commands open `dbConfig`. The comment on `openDB` says which mode each command uses and why.
|
||||
- [ ] 4.4 `internal/embeds/mc-config.yaml`: copies 27 and 27a.
|
||||
- [ ] 4.5 Test stack (design D14): `test/compose.yaml` gives `postgres` a health check and adds `postgres-seed`; `test/seed/postgres/seed-runtime-user.sh` (new, SPDX header, copies 30 and 31); `test/bootstrap-stack.sh` writes `MC_DB_DSN` and `MC_DB_MIGRATE_DSN`, prints copy 29, refuses a stale `.env` on a re-run with copy 28, and its header says what it writes; `test/reset-test-db.sh` refuses a stale `.env` (copy 28) after its existing guard and runs `MC_DB_MIGRATE_DSN="$dsn" go run . migrate up`; `test/reset-app-db.sh` refuses a stale `.env` and uses `MC_DB_MIGRATE_DSN` for `migrate up`, the pin, `pg_dump` and the snapshot's address, and its header says so; `test/seed-demo.sh` refuses a stale `.env`; `test/mc-config.yaml` carries both strings; `test/mockshot-nuke.sql`'s comment (`:111-114`) says that the stack's migration user is a superuser, which `SET ROLE core_owner` relies on; `test/seed/discourse/seed-discourse.sh`'s header says the runtime user needs `console_runtime`, which the first `start` or `reset-app-db.sh` grants.
|
||||
- [ ] 4.6 `test/e2e/browsertest/session.go`: `AppAdminDB` opens `MC_DB_MIGRATE_DSN` and skips with copy 32; `AppDB`'s comment says it is the runtime user. `test/e2e/screens/screens_test.go`'s `pinTimestamps` uses `AppAdminDB`.
|
||||
- [ ] 4.7 Docs: `docs/environment-reference.md` (rows for `db-migrate-dsn` and `db-migrate-dsn-file`; `db-dsn` and `db-dsn-file` name the runtime user); `docs/hosting.md` (five secrets); `docs/database-management.md` ("Migrations run automatically on startup" and the `ConnectAndMigrate` paragraph, design D17; the migration connection opens with `search_path=public`, and the `ConnectPlain` reason is restated for the application's `core,public` path); `docs/testing.md`, `test/AGENTS.md` (the `.env` keys block, the databases table's user column, the regeneration command at `:210-211`) and `test/README.md` (the Postgres row names both users).
|
||||
- [ ] 4.8 Tests:
|
||||
- [x] 4.1 Keys (design D1): `startCmd` registers `db-migrate-dsn` and `db-migrate-dsn-file` (copies 23, 24), and the help of `db-dsn` and `db-dsn-file` changes (copies 21, 22); `databaseSecretPairs` gains `db-migrate-dsn`/`db-migrate-dsn-file`; `ValidateStart` refuses a malformed `db-migrate-dsn` (copy 14) and treats it as optional.
|
||||
- [x] 4.2 `internal/db`: `PendingMigrations(ctx, q, sources)` (design D10) in `migrations.go`, which begins its transaction with `SET LOCAL search_path = pg_catalog, pg_temp` and reads each ledger as `public.goose_db_version_<stream>` by qualified name and calls `pg_catalog.to_regclass` and `pg_catalog.has_table_privilege` (design D2, the qualification principle), with a doc comment that says why it does not use goose's provider and why the names are qualified; `ConnectMigration(ctx, logger, config)`, which opens the migration pool with the startup parameters `search_path=public` and `role=none` as runtime parameters of the connection string, replacing any `search_path` or `role` the string carries (`ensureSearchPath`, `internal/db/database.go:43-60`, keeps an operator's value and is not reused, so a `db-migrate-dsn` copied from the runtime string cannot open with `core,public`); the runtime pool's open gains `role=none`, always replacing the string's value, beside its `search_path=core,public` (`ConnectPlain` stays for `dbConfig`); `CurrentUser` and `GrantRuntimeRole(ctx, tx, name)` (design D8, which begins with `SET LOCAL search_path = pg_catalog, pg_temp`, and whose statement calls `pg_catalog.set_config`, `pg_catalog.format` and `pg_catalog.current_setting`; it takes a transaction because the setting lasts until the transaction ends, and `cmd` runs it through `db.InTx`) in `runtime_user.go`, and the refusal type `RuntimeUserRefusal`. Every SQL string of these functions is a constant whose name begins `query` (item 5.8 scans them). `ConnectAndMigrate` is deleted.
|
||||
- [x] 4.3 `cmd/database.go` (design D11): modes `dbRuntime`, `dbMigration` and `dbConfig`. `dbMigration` opens `db-migrate-dsn` with `search_path=public` and `role=none` (design D2, rule 2) and refuses without it (copy 12). `dbConfig` is today's `dbPlain` on `db-dsn`. `dbRuntime` opens the runtime pool with `role=none` beside its `search_path`, and runs the boot sequence's steps 2, 3, 6 and 7, step 5's migrations, and the pending check in both modes (copies 4, 4b to 4d, 10, 18, 20); the other checks, and steps 4 and 8 in full, are section 5's, with copy 4a, which belongs to check 5. `start` and `seed-demo` open `dbRuntime`; `start` logs a refusal with copy 17; the `migrate` commands open `dbMigration`; the `config` commands open `dbConfig`. The comment on `openDB` says which mode each command uses and why.
|
||||
- [x] 4.4 `internal/embeds/mc-config.yaml`: copies 27 and 27a.
|
||||
- [x] 4.5 Test stack (design D14): `test/compose.yaml` gives `postgres` a health check and adds `postgres-seed`; `test/seed/postgres/seed-runtime-user.sh` (new, SPDX header, copies 30 and 31); `test/bootstrap-stack.sh` writes `MC_DB_DSN` and `MC_DB_MIGRATE_DSN`, prints copy 29, refuses a stale `.env` on a re-run with copy 28, and its header says what it writes; `test/reset-test-db.sh` refuses a stale `.env` (copy 28) after its existing guard and runs `MC_DB_MIGRATE_DSN="$dsn" go run . migrate up`; `test/reset-app-db.sh` refuses a stale `.env` and uses `MC_DB_MIGRATE_DSN` for `migrate up`, the pin, `pg_dump` and the snapshot's address, and its header says so; `test/seed-demo.sh` refuses a stale `.env`; `test/mc-config.yaml` carries both strings; `test/mockshot-nuke.sql`'s comment (`:111-114`) says that the stack's migration user is a superuser, which `SET ROLE core_owner` relies on; `test/seed/discourse/seed-discourse.sh`'s header says the runtime user needs `console_runtime`, which the first `start` or `reset-app-db.sh` grants.
|
||||
- [x] 4.6 `test/e2e/browsertest/session.go`: `AppAdminDB` opens `MC_DB_MIGRATE_DSN` and skips with copy 32; `AppDB`'s comment says it is the runtime user. `test/e2e/screens/screens_test.go`'s `pinTimestamps` uses `AppAdminDB`.
|
||||
- [x] 4.7 Docs: `docs/environment-reference.md` (rows for `db-migrate-dsn` and `db-migrate-dsn-file`; `db-dsn` and `db-dsn-file` name the runtime user); `docs/hosting.md` (five secrets); `docs/database-management.md` ("Migrations run automatically on startup" and the `ConnectAndMigrate` paragraph, design D17; the migration connection opens with `search_path=public`, and the `ConnectPlain` reason is restated for the application's `core,public` path); `docs/testing.md`, `test/AGENTS.md` (the `.env` keys block, the databases table's user column, the regeneration command at `:210-211`) and `test/README.md` (the Postgres row names both users).
|
||||
- [x] 4.8 Tests:
|
||||
- `TestPendingMigrations` (`internal/db`), on scratch databases: migrated in full, nothing pending; a stream stopped short with `dbtest.UpTo`, its missing versions; a database with no Discourse ledger, Discourse with every version; a ledger whose `SELECT` grant to `console_runtime` is revoked, the unreadable detail; a temporary table named like a ledger, created on the checking connection, changes no result; each read as `console_runtime` on a connection with `search_path=core,public`.
|
||||
- `TestMigrationPoolIgnoresADatabaseSearchPath` (`internal/db`, in a `db_test` file such as `internal/db/migration_pool_test.go`, because it exercises `db.ConnectMigration` and `schemaSnapshot` lives in package `db_test`, which `cmd` cannot import), on scratch databases the test creates: with `ALTER DATABASE` setting `search_path` to a schema the test created and then `public`, the migrations run through the migration pool, `SHOW search_path` on it returns `public`, and every ledger is in `public` and read by `PendingMigrations`; a row whose database sets `role` to a role the test creates, expecting `SELECT current_user` on the pool to be the migration login; and two fresh scratch databases with no extra schema (`schemaSnapshot` reads every user schema, so a database that carries the first row's extra schema never compares equal), one migrated through the migration pool and one by a plain replay (`dbtest`), with equal snapshots, so a dropped object fails the test.
|
||||
- `TestGrantRuntimeRole` (`internal/db`), in a transaction it rolls back, which `GrantRuntimeRole` takes because its setting is local to a transaction: a role named `Mc-Runtime-<n>`, created `LOGIN NOINHERIT`, inherits `console_runtime` after the grant; a second grant succeeds and changes nothing.
|
||||
- `TestOpenRuntimeDatabase` (`cmd/database_db_test.go`), on scratch databases with login roles the test creates with a password and drops in its cleanup: with both strings on a fresh scratch database, the runtime login inherits `console_runtime`, `SELECT current_user` on the returned pool is the runtime login, the ledgers read through the returned pool are at every embedded version, and `pg_stat_activity`, read from the shared test database, shows no session of the migration user on the scratch database; without `db-migrate-dsn` on a scratch database stopped short, copy 4 with 4d, and the ledgers unchanged; without it on a current database, the pool is returned and nothing ran; `migrate up` without `db-migrate-dsn`, copy 12; a runtime login that is a member of `console_runtime` and has run `ALTER ROLE CURRENT_USER SET role = 'console_runtime'` (its later sessions would report `current_user` as the group role), a normal boot with the login as `current_user` on the returned pool.
|
||||
- `TestValidateStart` gains a row for copy 14; `TestResolveSecretFiles_DBDSN` gains the `db-migrate-dsn` pair's two rows; `TestDatabaseCommandsReadTheFileKeys`'s `migrate` rows set `db-migrate-dsn-file` and expect copy 12 when neither key is set.
|
||||
- `TestScriptsRefuseAnEnvWithoutTheMigrationDSN` (`test/scripts_env_test.go`), with the shims of `TestScriptsRefuseUnknownArguments`, runs each of the four scripts beside a `.env` with every other key it reads and asserts copy 28 on stderr, exit status 1, and that no shim ran. `TestBootstrapChecksTheFarmName`'s fixture `.env` gains `MC_DB_MIGRATE_DSN`; its result does not change.
|
||||
- [ ] 4.9 Before this section's `make test`, regenerate this worktree's `test/.env` with copy 28's commands; `docker compose logs postgres-seed` shows copy 30. Then `cd test && ./reset-app-db.sh --rebuild`, start the application, and check that it boots, that the boot log shows copy 20, and that the operator and member pages load.
|
||||
- [ ] 4.10 Walkthroughs from the demo snapshot before and after (`test/AGENTS.md`, "Browser walkthroughs"), with `go test -count=1 ./test/e2e/operator-walkthroughs/` on the parent commit with the `test/.env` it writes, and on this one with the regenerated file: the test names, results and skip messages match. `make screens` lists no changed screen.
|
||||
- [ ] 4.11 Breaks (the migration-free ones need no reset):
|
||||
- [x] 4.9 Before this section's `make test`, regenerate this worktree's `test/.env` with copy 28's commands; `docker compose logs postgres-seed` shows copy 30. Then `cd test && ./reset-app-db.sh --rebuild`, start the application, and check that it boots, that the boot log shows copy 20, and that the operator and member pages load.
|
||||
- [x] 4.10 Walkthroughs from the demo snapshot before and after (`test/AGENTS.md`, "Browser walkthroughs"), with `go test -count=1 ./test/e2e/operator-walkthroughs/` on the parent commit with the `test/.env` it writes, and on this one with the regenerated file: the test names, results and skip messages match. `make screens` lists no changed screen.
|
||||
- [x] 4.11 Breaks (the migration-free ones need no reset):
|
||||
- `dbRuntime` runs the migrations on the runtime pool: caught by `TestOpenRuntimeDatabase`'s first row (the runtime login cannot create a schema);
|
||||
- `dbRuntime` skips the boot grant: caught by the first row's membership assertion;
|
||||
- `dbRuntime` leaves the migration pool open: caught by the first row's `pg_stat_activity` assertion;
|
||||
|
||||
+25
-10
@@ -34,7 +34,8 @@ The script:
|
||||
- Prints the URL/port table the developer should use, and which integration
|
||||
profiles are active.
|
||||
- **Idempotent**: re-running with `test/.env` already in place is a no-op
|
||||
that reprints the summary.
|
||||
that reprints the summary. A `test/.env` without `MC_DB_MIGRATE_DSN` is
|
||||
refused with the regeneration command under "After bootstrap".
|
||||
|
||||
## Which integrations the stack composes
|
||||
|
||||
@@ -174,13 +175,24 @@ because Go's test cache keys on code and environment, not database state, so
|
||||
after a reset a cached `ok` would describe a database that no longer exists.
|
||||
The whole suite takes a few minutes.
|
||||
|
||||
The stack's Postgres carries three databases:
|
||||
The stack's Postgres carries three databases and two users. `member_console`
|
||||
is the migration user, the cluster's bootstrap superuser, and owns the
|
||||
databases and their schemas. `member_console_runtime` is the runtime user,
|
||||
created by the `postgres-seed` compose service in new and existing volumes: not
|
||||
a superuser, owning nothing, and holding only `console_runtime`, which the
|
||||
application grants it at start.
|
||||
|
||||
| Database | Who writes it | Env var |
|
||||
|-----------------------|----------------------------------------|----------------------|
|
||||
| `member_console` | the running app, seeders, browser walkthroughs | `MC_DB_DSN` |
|
||||
| `member_console_test` | every DB-backed package test | `TEST_DATABASE_URL` |
|
||||
| `member_console_e2e` | `test/e2e/plan-management` | `E2E_DATABASE_URL` |
|
||||
| Database | User | Who writes it | Env var |
|
||||
|-----------------------|----------------------------|----------------------------------------|----------------------|
|
||||
| `member_console` | `member_console_runtime` | the running app, the demo seeder, browser walkthroughs | `MC_DB_DSN` |
|
||||
| `member_console` | `member_console` | migrations, the demo snapshot, the screens' timestamp pinning (`browsertest.AppAdminDB`) | `MC_DB_MIGRATE_DSN` |
|
||||
| `member_console_test` | `member_console` | every DB-backed package test | `TEST_DATABASE_URL` |
|
||||
| `member_console_e2e` | `member_console` | `test/e2e/plan-management` | `E2E_DATABASE_URL` |
|
||||
|
||||
`TEST_DATABASE_URL`, `E2E_DATABASE_URL` and `POSTGRES_ADMIN_DSN` stay the
|
||||
superuser: scratch databases, migration replays, `CREATE ROLE` and `SET ROLE`
|
||||
need one. The tests that prove a lock drop to `console_runtime` with `SET LOCAL
|
||||
ROLE` instead of connecting as the runtime user.
|
||||
|
||||
The split is what makes the suite trustworthy, and each boundary earns its
|
||||
place. Tests are off the app database because a running `member-console
|
||||
@@ -207,8 +219,11 @@ browser against the live app. Those skip wholesale when no app is listening
|
||||
failure there is about that database's contents, not about the isolation the
|
||||
reset provides.
|
||||
|
||||
If `reset-test-db.sh` reports a missing key, `test/.env` predates the split:
|
||||
`cd test && rm .env && ./bootstrap-stack.sh`. Containers and volumes survive.
|
||||
If `reset-test-db.sh` reports a missing key, `test/.env` is older than the
|
||||
stack: `cd test && docker compose down && rm .env && ./bootstrap-stack.sh &&
|
||||
docker compose up -d`. Volumes survive, and `postgres-seed` adds the runtime
|
||||
user to the existing cluster. `bootstrap-stack.sh`, `reset-app-db.sh` and `seed-demo.sh` refuse the
|
||||
same file with the same command.
|
||||
|
||||
## Browser walkthroughs (e2e) and stack age
|
||||
|
||||
@@ -356,7 +371,7 @@ TEMPORAL_PORT, TEMPORAL_UI_PORT, FEDWIKI_PORT, DISCOURSE_PORT
|
||||
|
||||
KC_HOSTNAME, DISCOURSE_HOSTNAME, MC_BASE_URL, TEMPORAL_UI_URL
|
||||
|
||||
MC_PORT, MC_DB_DSN, MC_VALKEY_ADDR,
|
||||
MC_PORT, MC_DB_DSN, MC_DB_MIGRATE_DSN, MC_VALKEY_ADDR,
|
||||
MC_OIDC_IDP_ISSUER_URL, MC_TEMPORAL_HOST,
|
||||
MC_TEMPORAL_OAUTH_TOKEN_URL
|
||||
|
||||
|
||||
+1
-1
@@ -85,7 +85,7 @@ Core services, always composed:
|
||||
| Member Console | http://localhost:8081 | (via Keycloak) |
|
||||
| Keycloak | http://keycloak.localhost:8080 | admin / admin |
|
||||
| Temporal UI | http://localhost:8233 | (via Keycloak) |
|
||||
| PostgreSQL | localhost:5432 | member_console / member_console |
|
||||
| PostgreSQL | localhost:5432 | runtime member_console_runtime / member_console_runtime; migration member_console / member_console |
|
||||
| Valkey | localhost:6379 | — |
|
||||
|
||||
Integration services, composed only when their profile is selected:
|
||||
|
||||
+27
-5
@@ -6,8 +6,15 @@
|
||||
# collisions until a free range is found.
|
||||
# - Copies test/secrets/ from the main worktree if missing locally.
|
||||
# - Writes test/.env with COMPOSE_PROJECT_NAME, host port assignments, and
|
||||
# MC_* env overrides for the application's Viper config.
|
||||
# - Idempotent: if test/.env exists, prints the summary and exits 0.
|
||||
# MC_* env overrides for the application's Viper config. The database keys
|
||||
# name two users: MC_DB_DSN is the runtime user member_console_runtime
|
||||
# (created by the postgres-seed compose service), MC_DB_MIGRATE_DSN is the
|
||||
# migration user member_console, and TEST_DATABASE_URL, E2E_DATABASE_URL and
|
||||
# POSTGRES_ADMIN_DSN stay member_console, which scratch databases, migration
|
||||
# replays and role statements need.
|
||||
# - Idempotent: if test/.env exists, prints the summary and exits 0. A test/.env
|
||||
# without MC_DB_MIGRATE_DSN predates the runtime user; it is refused and the
|
||||
# command that regenerates it is printed.
|
||||
#
|
||||
# Usage:
|
||||
# cd test && ./bootstrap-stack.sh # slug = basename of worktree
|
||||
@@ -81,7 +88,7 @@ print_summary() {
|
||||
Temporal gRPC: localhost:${TEMPORAL_PORT}
|
||||
${fedwiki_lines}
|
||||
|
||||
Postgres: localhost:${POSTGRES_PORT} (member_console / member_console)
|
||||
Postgres: localhost:${POSTGRES_PORT} (runtime member_console_runtime / member_console_runtime, migration member_console / member_console)
|
||||
databases: member_console (app), member_console_test and
|
||||
member_console_e2e (tests; reset by test/reset-test-db.sh)
|
||||
Valkey: localhost:${VALKEY_PORT}
|
||||
@@ -238,9 +245,23 @@ EOF
|
||||
# ---------- idempotent re-run ----------
|
||||
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
echo "test/.env already exists — using existing port assignments."
|
||||
# A shell that exports the key must not stand in for a stale file's missing one.
|
||||
unset MC_DB_MIGRATE_DSN
|
||||
# shellcheck disable=SC1090
|
||||
set -a; . "$ENV_FILE"; set +a
|
||||
if [[ -z "${MC_DB_MIGRATE_DSN:-}" ]]; then
|
||||
cat >&2 <<EOF
|
||||
error: MC_DB_MIGRATE_DSN is not set in ${ENV_FILE}.
|
||||
|
||||
This .env predates the stack's runtime database user. Regenerate it:
|
||||
|
||||
cd test && docker compose down && rm .env && ./bootstrap-stack.sh && docker compose up -d
|
||||
|
||||
Volumes survive, and postgres-seed adds the runtime user to the existing cluster.
|
||||
EOF
|
||||
exit 1
|
||||
fi
|
||||
echo "test/.env already exists — using existing port assignments."
|
||||
print_summary
|
||||
check_farm_name
|
||||
exit 0
|
||||
@@ -364,7 +385,8 @@ TEMPORAL_UI_URL=${TEMPORAL_UI_URL}
|
||||
# Viper overrides for mc-config.yaml (MC_ prefix, dash → underscore)
|
||||
MC_PORT=${MC_PORT}
|
||||
MC_BASE_URL=${MC_BASE_URL}
|
||||
MC_DB_DSN=postgres://member_console:member_console@localhost:${POSTGRES_PORT}/member_console?sslmode=disable
|
||||
MC_DB_DSN=postgres://member_console_runtime:member_console_runtime@localhost:${POSTGRES_PORT}/member_console?sslmode=disable
|
||||
MC_DB_MIGRATE_DSN=postgres://member_console:member_console@localhost:${POSTGRES_PORT}/member_console?sslmode=disable
|
||||
MC_VALKEY_ADDR=localhost:${VALKEY_PORT}
|
||||
MC_VALKEY_PASSWORD=${VALKEY_PASSWORD}
|
||||
MC_OIDC_IDP_ISSUER_URL=http://${KC_HOSTNAME}:${KEYCLOAK_PORT}/realms/test
|
||||
|
||||
@@ -24,6 +24,26 @@ services:
|
||||
- ./testdata/postgres:/var/lib/postgresql
|
||||
ports:
|
||||
- "${POSTGRES_PORT:-5432}:5432"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U member_console -d member_console"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
|
||||
# Creates the runtime login the application serves from (member_console is
|
||||
# the migration user). Runs on every `docker compose up`, so an existing
|
||||
# volume gains the login without a teardown.
|
||||
postgres-seed:
|
||||
image: postgres:18.1
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- PGPASSWORD=member_console
|
||||
volumes:
|
||||
- ./seed/postgres/seed-runtime-user.sh:/seed/seed-runtime-user.sh:ro
|
||||
restart: "no"
|
||||
entrypoint: ["/bin/bash", "/seed/seed-runtime-user.sh"]
|
||||
|
||||
# Identity Provider
|
||||
keycloak:
|
||||
|
||||
@@ -24,15 +24,32 @@ const (
|
||||
Diana = "diana"
|
||||
)
|
||||
|
||||
// AppDB opens the running application's database (MC_DB_DSN in test/.env)
|
||||
// and closes it when the test ends. It never migrates: the running app owns
|
||||
// its schema. Skips when the key is missing, which means test/.env predates
|
||||
// the key.
|
||||
// AppDB opens the running application's database as the runtime user
|
||||
// (MC_DB_DSN in test/.env) and closes it when the test ends. The fixtures
|
||||
// that use it write with the application's privileges. It never migrates: the
|
||||
// running app owns its schema. Skips when the key is missing, which means
|
||||
// test/.env predates the key.
|
||||
func AppDB(t testing.TB) *sql.DB {
|
||||
t.Helper()
|
||||
dsn := EnvFromTestDir(t)["MC_DB_DSN"]
|
||||
return openAppDatabase(t, "MC_DB_DSN", "MC_DB_DSN not set in test/.env; cd test && rm .env && ./bootstrap-stack.sh")
|
||||
}
|
||||
|
||||
// AppAdminDB opens the application's database as the migration user
|
||||
// (MC_DB_MIGRATE_DSN in test/.env), for work that needs a superuser, such as
|
||||
// the screens' timestamp pinning. It closes when the test ends and skips when
|
||||
// the key is missing, which means test/.env predates the runtime user.
|
||||
func AppAdminDB(t testing.TB) *sql.DB {
|
||||
t.Helper()
|
||||
return openAppDatabase(t, "MC_DB_MIGRATE_DSN", "MC_DB_MIGRATE_DSN not set in test/.env; cd test && docker compose down && rm .env && ./bootstrap-stack.sh && docker compose up -d")
|
||||
}
|
||||
|
||||
// openAppDatabase opens the address test/.env holds under key, or skips the
|
||||
// test with skip when the key is missing.
|
||||
func openAppDatabase(t testing.TB, key, skip string) *sql.DB {
|
||||
t.Helper()
|
||||
dsn := EnvFromTestDir(t)[key]
|
||||
if dsn == "" {
|
||||
t.Skip("MC_DB_DSN not set in test/.env; cd test && rm .env && ./bootstrap-stack.sh")
|
||||
t.Skip(skip)
|
||||
}
|
||||
database, err := sql.Open("pgx", dsn)
|
||||
if err != nil {
|
||||
|
||||
@@ -824,10 +824,11 @@ func checkTableWrappers(page *rod.Page) tableWrapperCheck {
|
||||
|
||||
// pinTimestamps runs test/pin-timestamps.sql against the application
|
||||
// database named in test/.env (the same pass test/reset-app-db.sh runs
|
||||
// after seeding), with the updated_at triggers bypassed.
|
||||
// after seeding), with the updated_at triggers bypassed. Only a superuser sets
|
||||
// session_replication_role, so it runs as the migration user.
|
||||
func pinTimestamps(t *testing.T) {
|
||||
t.Helper()
|
||||
db := browsertest.AppDB(t)
|
||||
db := browsertest.AppAdminDB(t)
|
||||
script, err := os.ReadFile(filepath.Join("..", "..", "pin-timestamps.sql"))
|
||||
if err != nil {
|
||||
t.Fatalf("read pin-timestamps.sql: %v", err)
|
||||
|
||||
+2
-1
@@ -15,7 +15,8 @@ oidc-idp-issuer-url: "http://keycloak.localhost:8080/realms/test"
|
||||
|
||||
# Application settings
|
||||
base-url: "http://localhost:8081"
|
||||
db-dsn: "postgres://member_console:member_console@localhost:5432/member_console?sslmode=disable"
|
||||
db-dsn: "postgres://member_console_runtime:member_console_runtime@localhost:5432/member_console?sslmode=disable"
|
||||
db-migrate-dsn: "postgres://member_console:member_console@localhost:5432/member_console?sslmode=disable"
|
||||
valkey-addr: "localhost:6379"
|
||||
|
||||
# Federated Wiki settings — the farm connection is OPT-IN and lives in
|
||||
|
||||
@@ -109,8 +109,10 @@ DELETE FROM core.products WHERE key LIKE 'mockshot\_%';
|
||||
|
||||
-- core_writer holds no DML on core.entitlement_set_rules (migration 00018):
|
||||
-- core.commit_rule_change writes a rule and nothing deletes one, so a teardown
|
||||
-- deletes as the owner role. This file is a test-stack tool, run by a login
|
||||
-- role that is a member of core_owner.
|
||||
-- deletes as the owner role. This file is a test-stack tool, run as
|
||||
-- member_console, the stack's migration user. That user is the cluster's
|
||||
-- bootstrap superuser, which is what lets SET ROLE core_owner and the direct
|
||||
-- deletes below run; no role is a member of core_owner.
|
||||
SET ROLE core_owner;
|
||||
DELETE FROM core.entitlement_set_rules
|
||||
WHERE set_id IN (SELECT set_id FROM core.entitlement_sets WHERE key = 'mockshot_hosting');
|
||||
|
||||
+30
-7
@@ -25,6 +25,13 @@
|
||||
# reconnects on its next query; browser sessions that referenced the old
|
||||
# rows are invalid until their next sign-in.
|
||||
#
|
||||
# The work that needs a superuser (migrate up, the clock pin, pg_dump and the
|
||||
# snapshot's address) runs as the migration user, MC_DB_MIGRATE_DSN; the
|
||||
# database's name comes from MC_DB_DSN, the runtime user's string. seed-demo
|
||||
# runs with both keys, so it grants console_runtime to the runtime user and
|
||||
# seeds as that user. A test/.env without MC_DB_MIGRATE_DSN is refused before
|
||||
# anything runs.
|
||||
#
|
||||
# Usage: cd test && ./reset-app-db.sh [--rebuild]
|
||||
# Any other argument, or a second one, is refused before anything runs.
|
||||
set -euo pipefail
|
||||
@@ -39,6 +46,8 @@ if [[ ! -f "$env_file" ]]; then
|
||||
echo "error: $env_file not found. Run ./bootstrap-stack.sh first." >&2
|
||||
exit 1
|
||||
fi
|
||||
# A shell that exports the key must not stand in for a stale file's missing one.
|
||||
unset MC_DB_MIGRATE_DSN
|
||||
set -a
|
||||
# shellcheck source=/dev/null
|
||||
source "$env_file"
|
||||
@@ -49,6 +58,18 @@ for var in POSTGRES_ADMIN_DSN MC_DB_DSN; do
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
if [[ -z "${MC_DB_MIGRATE_DSN:-}" ]]; then
|
||||
cat >&2 <<EOF
|
||||
error: MC_DB_MIGRATE_DSN is not set in $env_file.
|
||||
|
||||
This .env predates the stack's runtime database user. Regenerate it:
|
||||
|
||||
cd test && docker compose down && rm .env && ./bootstrap-stack.sh && docker compose up -d
|
||||
|
||||
Volumes survive, and postgres-seed adds the runtime user to the existing cluster.
|
||||
EOF
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The database reset re-creates every row from the demo seed (or, on
|
||||
# rebuild, from a fresh seed run), and ids are UUIDv7 -- never reused -- so
|
||||
@@ -80,11 +101,13 @@ if [[ -z "$name" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
snapshot="${name}_demo"
|
||||
# The snapshot's DSN is the application's with the database name (the last
|
||||
# The snapshot's DSN is the migration user's with the database name (the last
|
||||
# path segment) swapped; the user name may equal the database name, so a
|
||||
# plain substitution would rename the wrong part.
|
||||
dsn_prefix="${MC_DB_DSN%/*}"
|
||||
dsn_tail="${MC_DB_DSN##*/}"
|
||||
# plain substitution would rename the wrong part. The restore creates schemas
|
||||
# and the clock pin sets session_replication_role, which only a superuser
|
||||
# sets, so the snapshot is built and read as the migration user.
|
||||
dsn_prefix="${MC_DB_MIGRATE_DSN%/*}"
|
||||
dsn_tail="${MC_DB_MIGRATE_DSN##*/}"
|
||||
dsn_query="${dsn_tail#"${name}"}"
|
||||
snapshot_dsn="${dsn_prefix}/${snapshot}${dsn_query}"
|
||||
|
||||
@@ -112,9 +135,9 @@ if [[ "$rebuild" == 1 ]]; then
|
||||
DROP DATABASE IF EXISTS ${name} WITH (FORCE);
|
||||
CREATE DATABASE ${name} OWNER member_console;
|
||||
SQL
|
||||
(cd "$repo_root" && MC_DB_DSN="$MC_DB_DSN" go run . migrate up --config test/mc-config.yaml >/dev/null)
|
||||
(cd "$repo_root" && MC_DB_MIGRATE_DSN="$MC_DB_MIGRATE_DSN" go run . migrate up --config test/mc-config.yaml >/dev/null)
|
||||
"$here/seed-demo.sh" --with-operator >/dev/null
|
||||
psql "$MC_DB_DSN" -v ON_ERROR_STOP=1 --quiet <<SQL
|
||||
psql "$MC_DB_MIGRATE_DSN" -v ON_ERROR_STOP=1 --quiet <<SQL
|
||||
SET session_replication_role = replica;
|
||||
\\i $here/pin-timestamps.sql
|
||||
SQL
|
||||
@@ -126,7 +149,7 @@ SQL
|
||||
# server, and without them the snapshot's SECURITY DEFINER functions
|
||||
# run as the superuser and PUBLIC may execute them, unlike a migrated
|
||||
# database.
|
||||
pg_dump "$MC_DB_DSN" | psql "$snapshot_dsn" -v ON_ERROR_STOP=1 --quiet >/dev/null
|
||||
pg_dump "$MC_DB_MIGRATE_DSN" | psql "$snapshot_dsn" -v ON_ERROR_STOP=1 --quiet >/dev/null
|
||||
psql "$POSTGRES_ADMIN_DSN" -v ON_ERROR_STOP=1 --quiet -c "COMMENT ON DATABASE ${snapshot} IS '${fingerprint}'"
|
||||
echo "Demo snapshot '${snapshot}' built; application database '${name}' is at the demo state."
|
||||
flush_sessions
|
||||
|
||||
+20
-2
@@ -14,6 +14,9 @@
|
||||
# of. The browser walkthroughs (test/e2e/operator-walkthroughs) drive the live
|
||||
# app and stay on the app database.
|
||||
#
|
||||
# The migration runs as the migration user: each database's own address, which
|
||||
# is the superuser that owns the databases, is passed as MC_DB_MIGRATE_DSN.
|
||||
#
|
||||
# Migrating here is load-bearing, not a convenience: no test migrates the
|
||||
# shared test databases, and dbtest.Open refuses one whose core ledger is
|
||||
# behind the migrations, so a database this script has not migrated fails every
|
||||
@@ -38,6 +41,8 @@ if [[ ! -f "$env_file" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# A shell that exports the key must not stand in for a stale file's missing one.
|
||||
unset MC_DB_MIGRATE_DSN
|
||||
set -a
|
||||
# shellcheck source=/dev/null
|
||||
source "$env_file"
|
||||
@@ -53,7 +58,7 @@ error: $var is not set in $env_file.
|
||||
|
||||
This .env predates the test-database split. Regenerate it:
|
||||
|
||||
cd test && rm .env && ./bootstrap-stack.sh
|
||||
cd test && docker compose down && rm .env && ./bootstrap-stack.sh && docker compose up -d
|
||||
|
||||
Existing containers and volumes survive; only the env file is rewritten.
|
||||
EOF
|
||||
@@ -61,6 +66,19 @@ EOF
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ -z "${MC_DB_MIGRATE_DSN:-}" ]]; then
|
||||
cat >&2 <<EOF
|
||||
error: MC_DB_MIGRATE_DSN is not set in $env_file.
|
||||
|
||||
This .env predates the stack's runtime database user. Regenerate it:
|
||||
|
||||
cd test && docker compose down && rm .env && ./bootstrap-stack.sh && docker compose up -d
|
||||
|
||||
Volumes survive, and postgres-seed adds the runtime user to the existing cluster.
|
||||
EOF
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Extract the database name from a DSN's path component.
|
||||
db_name_from_dsn() {
|
||||
local dsn="$1"
|
||||
@@ -88,7 +106,7 @@ CREATE DATABASE ${name} OWNER member_console;
|
||||
SQL
|
||||
|
||||
echo " migrating..."
|
||||
(cd "$repo_root" && MC_DB_DSN="$dsn" go run . migrate up --config test/mc-config.yaml >/dev/null)
|
||||
(cd "$repo_root" && MC_DB_MIGRATE_DSN="$dsn" go run . migrate up --config test/mc-config.yaml >/dev/null)
|
||||
}
|
||||
|
||||
reset_one "$TEST_DATABASE_URL" "package-test"
|
||||
|
||||
@@ -132,6 +132,7 @@ func (c argRow) check(t *testing.T, sb scriptBox) {
|
||||
// shims in place of every command that reaches the stack.
|
||||
type scriptBox struct {
|
||||
bash, dir, script, bin, calls string
|
||||
env []string // added to the script's environment
|
||||
}
|
||||
|
||||
type scriptRun struct {
|
||||
@@ -172,7 +173,7 @@ func (sb scriptBox) run(t *testing.T, args ...string) scriptRun {
|
||||
_ = os.Remove(sb.calls)
|
||||
cmd := exec.Command(sb.bash, append([]string{sb.script}, args...)...)
|
||||
cmd.Dir = sb.dir
|
||||
cmd.Env = []string{"PATH=" + sb.bin + ":/usr/bin:/bin", "SHIM_LOG=" + sb.calls}
|
||||
cmd.Env = append([]string{"PATH=" + sb.bin + ":/usr/bin:/bin", "SHIM_LOG=" + sb.calls}, sb.env...)
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &stdout, &stderr
|
||||
r := scriptRun{}
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
|
||||
package test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// staleEnvRefusal is the stack scripts' refusal of a test/.env that lacks
|
||||
// MC_DB_MIGRATE_DSN (design copy 28), for the .env at envFile.
|
||||
func staleEnvRefusal(envFile string) string {
|
||||
return "error: MC_DB_MIGRATE_DSN is not set in " + envFile + ".\n" +
|
||||
"\n" +
|
||||
"This .env predates the stack's runtime database user. Regenerate it:\n" +
|
||||
"\n" +
|
||||
" cd test && docker compose down && rm .env && ./bootstrap-stack.sh && docker compose up -d\n" +
|
||||
"\n" +
|
||||
"Volumes survive, and postgres-seed adds the runtime user to the existing cluster.\n"
|
||||
}
|
||||
|
||||
// exportedMigrateDSN is the entry of a shell that sourced another worktree's
|
||||
// test/.env.
|
||||
const exportedMigrateDSN = "MC_DB_MIGRATE_DSN=postgres://x@h/db"
|
||||
|
||||
// TestScriptsRefuseAnEnvWithoutTheMigrationDSN runs each script that reads
|
||||
// test/.env beside a .env that holds every other key the script reads and
|
||||
// lacks MC_DB_MIGRATE_DSN, with the shims of TestScriptsRefuseUnknownArguments.
|
||||
// Each must print the regeneration command on stderr, exit 1, print nothing on
|
||||
// stdout, and reach no database or container: a guard that runs after the
|
||||
// script's first psql, go or docker call shows as a shim that ran. The
|
||||
// bootstrap script asks git for the repository's top first, so its one
|
||||
// permitted shim is git. Each also runs in a shell that exports
|
||||
// MC_DB_MIGRATE_DSN: the file is what the script reads, so the exported value
|
||||
// must not stand in for the key the file lacks.
|
||||
func TestScriptsRefuseAnEnvWithoutTheMigrationDSN(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
script string
|
||||
env string
|
||||
ran string
|
||||
}{
|
||||
{"reset-test-db.sh",
|
||||
"POSTGRES_ADMIN_DSN=postgres://u:p@localhost:5432/postgres\n" +
|
||||
"TEST_DATABASE_URL=postgres://u:p@localhost:5432/member_console_test\n" +
|
||||
"E2E_DATABASE_URL=postgres://u:p@localhost:5432/member_console_e2e\n", ""},
|
||||
{"reset-app-db.sh",
|
||||
"POSTGRES_ADMIN_DSN=postgres://u:p@localhost:5432/postgres\n" +
|
||||
"MC_DB_DSN=postgres://r:p@localhost:5432/member_console\n", ""},
|
||||
{"seed-demo.sh", "MC_DB_DSN=postgres://r:p@localhost:5432/member_console\n", ""},
|
||||
} {
|
||||
t.Run(c.script, func(t *testing.T) {
|
||||
sb := newScriptBox(t, c.script)
|
||||
envFile := filepath.Join(sb.dir, ".env")
|
||||
if err := os.WriteFile(envFile, []byte(c.env), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkStaleEnvRefused(t, sb, envFile, c.ran)
|
||||
sb.env = []string{exportedMigrateDSN}
|
||||
checkStaleEnvRefused(t, sb, envFile, c.ran)
|
||||
})
|
||||
}
|
||||
t.Run("bootstrap-stack.sh", func(t *testing.T) {
|
||||
sb := newFarmNameBox(t, "", "")
|
||||
envFile := filepath.Join(sb.dir, "repo", "test", ".env")
|
||||
// The farm-name fixture holds every key bootstrap reads once it has
|
||||
// found the .env; drop the one under test.
|
||||
env, err := os.ReadFile(envFile)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var kept []string
|
||||
for _, line := range strings.Split(string(env), "\n") {
|
||||
if !strings.HasPrefix(line, "MC_DB_MIGRATE_DSN=") {
|
||||
kept = append(kept, line)
|
||||
}
|
||||
}
|
||||
if err := os.WriteFile(envFile, []byte(strings.Join(kept, "\n")), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkStaleEnvRefused(t, sb, envFile, "git")
|
||||
sb.env = []string{exportedMigrateDSN}
|
||||
checkStaleEnvRefused(t, sb, envFile, "git")
|
||||
})
|
||||
}
|
||||
|
||||
func checkStaleEnvRefused(t *testing.T, sb scriptBox, envFile, ran string) {
|
||||
t.Helper()
|
||||
r := sb.run(t)
|
||||
if r.code != 1 {
|
||||
t.Errorf("exit status %d, want 1 (stderr %q)", r.code, r.stderr)
|
||||
}
|
||||
if want := staleEnvRefusal(envFile); r.stderr != want {
|
||||
t.Errorf("stderr %q, want %q", r.stderr, want)
|
||||
}
|
||||
if r.stdout != "" {
|
||||
t.Errorf("stdout %q, want nothing", r.stdout)
|
||||
}
|
||||
if r.ran != ran {
|
||||
t.Errorf("the script ran %q, want %q", r.ran, ran)
|
||||
}
|
||||
}
|
||||
@@ -78,7 +78,8 @@ func newFarmNameBox(t *testing.T, profiles, getent string) scriptBox {
|
||||
env := "COMPOSE_PROJECT_NAME=farm-name\nCOMPOSE_PROFILES=" + profiles + "\n" +
|
||||
"MC_BASE_URL=http://farm-name.localhost:8081\nKC_HOSTNAME=keycloak-farm-name.localhost\n" +
|
||||
"KEYCLOAK_PORT=8080\nTEMPORAL_UI_URL=http://temporal-ui-farm-name.localhost:8233\n" +
|
||||
"TEMPORAL_PORT=7233\nFEDWIKI_PORT=8090\nPOSTGRES_PORT=5432\nVALKEY_PORT=6379\n"
|
||||
"TEMPORAL_PORT=7233\nFEDWIKI_PORT=8090\nPOSTGRES_PORT=5432\nVALKEY_PORT=6379\n" +
|
||||
"MC_DB_MIGRATE_DSN=postgres://member_console:member_console@localhost:5432/member_console\n"
|
||||
if err := os.WriteFile(filepath.Join(testDir, ".env"), []byte(env), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
+16
-1
@@ -2,7 +2,7 @@
|
||||
#
|
||||
# test/seed-demo.sh — host-side invocation of the member-console demo seeder.
|
||||
#
|
||||
# Loads the worktree-isolated env (DB port, DSN) from test/.env and runs
|
||||
# Loads the worktree-isolated env (DB port, both DSNs) from test/.env and runs
|
||||
# `go run . seed-demo --config test/mc-config.yaml` from the repo root.
|
||||
#
|
||||
# Opt-in only: the default bootstrap-stack.sh + docker compose up workflow
|
||||
@@ -29,10 +29,25 @@ if [[ ! -f "$env_file" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# A shell that exports the key must not stand in for a stale file's missing one.
|
||||
unset MC_DB_MIGRATE_DSN
|
||||
set -a
|
||||
# shellcheck source=/dev/null
|
||||
source "$env_file"
|
||||
set +a
|
||||
|
||||
if [[ -z "${MC_DB_MIGRATE_DSN:-}" ]]; then
|
||||
cat >&2 <<EOF
|
||||
error: MC_DB_MIGRATE_DSN is not set in $env_file.
|
||||
|
||||
This .env predates the stack's runtime database user. Regenerate it:
|
||||
|
||||
cd test && docker compose down && rm .env && ./bootstrap-stack.sh && docker compose up -d
|
||||
|
||||
Volumes survive, and postgres-seed adds the runtime user to the existing cluster.
|
||||
EOF
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cd "$repo_root"
|
||||
exec go run . seed-demo --config test/mc-config.yaml "$@"
|
||||
|
||||
@@ -16,6 +16,10 @@
|
||||
# whose external id matches the person's OIDC subject (seed one via the
|
||||
# walkthrough or the fakeserver's -seed-group + client calls).
|
||||
#
|
||||
# MC_DB_DSN names the stack's runtime user, which writes through the writer
|
||||
# tables and core functions only and needs console_runtime first: the first
|
||||
# `start` or a test/reset-app-db.sh run that rebuilds the snapshot grants it.
|
||||
#
|
||||
# Usage: MC_DB_DSN=... ./seed-discourse.sh (or source test/.env first)
|
||||
set -euo pipefail
|
||||
|
||||
|
||||
Executable
+42
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env bash
|
||||
# SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
# SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
#
|
||||
# Creates the test stack's runtime login, member_console_runtime, beside
|
||||
# member_console, the migration user the official image makes the cluster's
|
||||
# bootstrap superuser. The application serves from the runtime login, which is
|
||||
# not a superuser and owns nothing; the application grants it console_runtime
|
||||
# at start. This script grants nothing.
|
||||
#
|
||||
# Idempotent: it creates the role when it is missing and sets LOGIN NOSUPERUSER
|
||||
# and the password on every run, so the role converges whatever an earlier run
|
||||
# left. Runs as the postgres-seed compose service on every `docker compose up`.
|
||||
#
|
||||
# TEST-ONLY: the password is a throwaway local-development value.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
psql_admin=(psql -h postgres -U member_console -d member_console -v ON_ERROR_STOP=1 --quiet)
|
||||
|
||||
# The image's first start runs initdb behind a temporary server that refuses
|
||||
# TCP, so retry until the real server answers.
|
||||
ready=0
|
||||
for _ in $(seq 1 60); do
|
||||
if "${psql_admin[@]}" -c 'SELECT 1' >/dev/null 2>&1; then
|
||||
ready=1
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
if [[ "$ready" != 1 ]]; then
|
||||
echo "ERROR: postgres-seed gave up waiting for postgres" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
"${psql_admin[@]}" <<'SQL'
|
||||
SELECT 'CREATE ROLE member_console_runtime LOGIN'
|
||||
WHERE NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'member_console_runtime')
|
||||
\gexec
|
||||
ALTER ROLE member_console_runtime LOGIN NOSUPERUSER PASSWORD 'member_console_runtime';
|
||||
SQL
|
||||
echo "runtime user member_console_runtime ready"
|
||||
Reference in New Issue
Block a user