Serve before Temporal connects, and connect it in the background

Sixth section of #141, which closes the issue's defect: start no longer
waits on Temporal before it serves. Start derives the Stripe key's mode
and fingerprint, sets the SDK's key, opens the database, applies the
operator overrides and the boot data state, binds the listener, and only
then starts the Temporal connector and the identity provider's discovery
beside the server. A bind error returns before any dial or discovery
request. While Temporal is unreachable the console serves: sign-in, the
domain ask, the static files and every page that needs no workflow
answer as usual, and the Temporal-backed actions answer as the previous
section made them.

The connector runs the retry loop. Each attempt dials once (NewClient
no longer retries on its own; dialWithRetry and its budget are gone),
builds the worker and runs its precondition. A precondition that
refuses keeps the client and retries at the cap, so FedWiki's refusal
over open runs of a removed workflow type is read again on the same
connection; one that cannot read what it checks, a worker that fails to
build or start, closes the client and dials again. The attempt ends when
the worker starts: "connected to Temporal server" is logged with the
attempts and the time it took, then the holder is published and the
integrations' Startup hooks and the schedules run as before. "Temporal not ready yet, retrying" is a Warn until
temporal-connect-timeout (90 seconds by default), whose help now says
so, and an Error from then on, under the loop's decay. A Temporal OAuth
configuration that cannot be built still refuses at start.

start exits 1 when serving fails, at any point, and returns only after
the connector and discovery have returned. A failure before the
listener keeps the message it had (the overrides, the boot data state,
the integration connections); a serving failure logs "server failed to
start".

The Stripe webhook activity marks an event processing only from
received, processing or failed, so a run of an event already completed,
skipped or dead-lettered returns at once and dispatches nothing.

The handlers and drains that found no Temporal client logged that it
was not configured; with Temporal still connecting that was false, so
they log "Temporal is not connected" (the six plan-act goldens carry the
new drain line). Copies 13, 15 and 16 also carry the host and namespace.

Checked by hand on the test stack: with Temporal stopped, the console
served /login and the ask at once; the connector logged Warn, then
Error from the first attempt after 90 seconds; when Temporal came back
the same process logged "connected to Temporal server" after 14
attempts. With the issuer refusing, /login answered the styled 503 with
Retry-After: 5 and the ask answered.

Full suite on reset databases: 4513 pass, 15 skip, none failed. go test -race -p 1 over cmd,
auth, server, retryloop, workflows and the three integrations after
reset-test-db.sh: every package passes, no data race. make lint passes. Breaks, each caught
(section 1's UseKey break among them). Two depart from item 6.10's
first wording, which the item now carries: the synchronous-fetch break
sits between listen and the server's start (before listen the row would
fail waiting for the address, not at its one-second assertions), and the
"returns without waiting" break is caught by the fourth boot row, whose
fake dial holds for 200 ms of wall-clock time past the context's end,
the least that makes an early return observable.
- serveConsole without UseKey
- UseKey after the bound server's Serve
- Serve without go Discover
- Serve fetches the discovery document before serving
- serveConsole waits for connectTemporal before Serve
- connectTemporal started before Bind returns
- serveConsole returns without waiting for its goroutines
- serveConsole returns without waiting for its goroutines (race)
- skipSession without the ask's path
- connectTemporal publishes after afterConnect
- connectTemporal publishes before worker.Start
- connectTemporal returns after the escalation time
- the open-runs refusal closes the client
- ErrCannotCheck treated as a refusal
- worker.Start failure leaves the client open
- listRunsError without its Is method
- worker.Start failure logged from inside the attempt
- ProcessWebhookEvent without its status guard
- the guard without failed
- the guard without processing
- NewClient without the deadline
- connectTemporal under serveConsole's own context
- the attempt runs the hooks before copy 14
- Run logs copy 21 for a step failure
- the overrides failure is a plain error
- a Temporal authorization that cannot be built is not refused
This commit is contained in:
2026-10-23 20:45:22 -04:00
parent 3bfd0f67f4
commit bfc24e2c74
28 changed files with 2421 additions and 658 deletions
+783
View File
@@ -0,0 +1,783 @@
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package cmd
import (
"bytes"
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net"
"net/http"
"net/http/httptest"
"os"
"strings"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
"github.com/alexedwards/scs/redisstore"
"github.com/alexedwards/scs/v2"
"github.com/gomodule/redigo/redis"
"github.com/spf13/viper"
"github.com/stripe/stripe-go/v81/webhook"
"go.temporal.io/sdk/client"
"git.coopcloud.tech/wiki-cafe/member-console/internal/auth"
"git.coopcloud.tech/wiki-cafe/member-console/internal/domains"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integrations"
stripeintegration "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/stripe"
"git.coopcloud.tech/wiki-cafe/member-console/internal/stripetest"
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/authtest"
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest"
"git.coopcloud.tech/wiki-cafe/member-console/internal/workflows"
)
// bootWait bounds the time a boot test gives the console to answer once its
// address is known (design: within one second of the address being received).
const bootWait = time.Second
// setConfig sets a viper key for the test and restores what it held.
func setConfig(t *testing.T, key string, value any) {
t.Helper()
prev, had := viper.Get(key), viper.IsSet(key)
viper.Set(key, value)
t.Cleanup(func() {
if had {
viper.Set(key, prev)
} else {
viper.Set(key, nil)
}
})
}
// countingListener accepts every connection, counts it and closes it, so a
// count that stays at zero could fail: a closed port records nothing.
type countingListener struct {
l net.Listener
count atomic.Int64
}
func newCountingListener(t *testing.T) *countingListener {
t.Helper()
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
c := &countingListener{l: l}
done := make(chan struct{})
go func() {
defer close(done)
for {
conn, err := l.Accept()
if err != nil {
return
}
c.count.Add(1)
_ = conn.Close()
}
}()
t.Cleanup(func() {
_ = l.Close()
<-done
})
return c
}
func (c *countingListener) addr() string { return c.l.Addr().String() }
// bootIdP is the identity provider the console discovers: it answers 503 until
// the test switches it to a discovery document whose issuer is its own URL.
type bootIdP struct {
srv *httptest.Server
usable atomic.Bool
requests atomic.Int64
}
func newBootIdP(t *testing.T) *bootIdP {
t.Helper()
p := &bootIdP{}
p.srv = httptest.NewServer(http.HandlerFunc(p.serve))
t.Cleanup(p.srv.Close)
return p
}
func (p *bootIdP) serve(w http.ResponseWriter, r *http.Request) {
p.requests.Add(1)
if !p.usable.Load() || r.URL.Path != "/.well-known/openid-configuration" {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{
"issuer": p.srv.URL,
"authorization_endpoint": p.srv.URL + "/auth",
"token_endpoint": p.srv.URL + "/token",
"jwks_uri": p.srv.URL + "/jwks",
"end_session_endpoint": p.srv.URL + "/logout",
"revocation_endpoint": p.srv.URL + "/revoke",
})
}
// bootEnv is the configuration, the fakes and the seeded rows one boot test
// serves the console over.
type bootEnv struct {
t *testing.T
database *sql.DB
temporal *countingListener
idp *bootIdP
valkeyAddr, valkeyPassword string
stripeKey, webhookSecret string
servable, unknown string
member *http.Cookie
}
// newBootEnv configures viper as a deployment would, seeds a servable
// placement under the test's suffix and shortens the connector's waits. It
// skips when the stack's session store is not available.
func newBootEnv(t *testing.T) *bootEnv {
t.Helper()
valkeyAddr, valkeyPassword := os.Getenv("MC_VALKEY_ADDR"), os.Getenv("MC_VALKEY_PASSWORD")
if valkeyAddr == "" || valkeyPassword == "" {
t.Skip("MC_VALKEY_ADDR or MC_VALKEY_PASSWORD is not set; the test stack's session store is not available")
}
database := dbtest.Open(t)
suffix := fmt.Sprintf("%d", time.Now().UnixNano())
e := &bootEnv{
t: t,
database: database,
temporal: newCountingListener(t),
idp: newBootIdP(t),
valkeyAddr: valkeyAddr,
valkeyPassword: valkeyPassword,
stripeKey: "sk_test_serve" + suffix,
webhookSecret: "whsec_serve" + suffix,
}
// The SDK key starts empty and is restored afterwards, so the test shows
// that serveConsole sets it.
stripetest.ClearSDKKey(t)
setConfig(t, "env", "development")
setConfig(t, "base-url", "http://127.0.0.1")
setConfig(t, "valkey-addr", valkeyAddr)
setConfig(t, "valkey-password", valkeyPassword)
setConfig(t, "oidc-idp-issuer-url", e.idp.srv.URL)
setConfig(t, "oidc-sp-client-id", "boot-test")
setConfig(t, "oidc-sp-client-secret", "boot-test-secret")
setConfig(t, "stripe-api-key", e.stripeKey)
setConfig(t, "stripe-webhook-secret", e.webhookSecret)
setConfig(t, "temporal-host", e.temporal.addr())
setConfig(t, "temporal-namespace", "default")
prevFirst, prevMax := temporalFirstWait, temporalMaxWait
temporalFirstWait, temporalMaxWait = 50*time.Millisecond, 50*time.Millisecond
t.Cleanup(func() { temporalFirstWait, temporalMaxWait = prevFirst, prevMax })
e.servable, e.unknown = e.seedServablePlacement("s"+suffix), "unknown-"+suffix+".test"
e.member = e.signedInCookie()
return e
}
// seedServablePlacement seeds a servable placement the way the domains
// registry's own database tests seed one: a workspace chain, an operator root,
// a carved name and a placement of it, all under token.
func (e *bootEnv) seedServablePlacement(token string) string {
e.t.Helper()
ctx := context.Background()
if _, err := e.database.ExecContext(ctx,
`INSERT INTO core.providers (provider, provider_kind, display_name)
VALUES ('testprovider', 'provisioning', 'Test Provider')
ON CONFLICT (provider) DO NOTHING`); err != nil {
e.t.Fatalf("fixture provider: %v", err)
}
workspaces := [2]string{}
for i := range workspaces {
workspaces[i] = e.newWorkspace(fmt.Sprintf("%s-%d", token, i))
}
registry := domains.NewRegistry(e.database)
root := token + ".test"
if _, err := registry.EnsureOperatorRoot(ctx, workspaces[0], root); err != nil {
e.t.Fatalf("ensure operator root: %v", err)
}
name := "alpha." + root
if _, err := registry.ClaimCarved(ctx, workspaces[1], name); err != nil {
e.t.Fatalf("carve: %v", err)
}
if _, err := registry.Place(ctx, domains.PlaceParams{
WorkspaceID: workspaces[1], FQDN: name, Provider: "testprovider", ResourceRef: "res-" + token, Servable: true,
}); err != nil {
e.t.Fatalf("place: %v", err)
}
return name
}
// newWorkspace builds the user, person, organization, workspace chain a claim
// needs and returns the workspace id.
func (e *bootEnv) newWorkspace(token string) string {
e.t.Helper()
ctx := context.Background()
scan := func(query string, args ...any) string {
var id string
if err := e.database.QueryRowContext(ctx, query, args...).Scan(&id); err != nil {
e.t.Fatalf("fixture %q: %v", query, err)
}
return id
}
userID := scan(`INSERT INTO core.users (oidc_subject) VALUES ($1) RETURNING user_id`, "sub-"+token)
personID := scan(`INSERT INTO core.persons (user_id, display_name, primary_email) VALUES ($1,$2,$3) RETURNING person_id`,
userID, "Tester", "t-"+token+"@example.com")
orgID := scan(`INSERT INTO core.organizations (name, org_type, owner_person_id) VALUES ($1,'personal',$2) RETURNING org_id`,
"Org "+token, personID)
return scan(`INSERT INTO core.workspaces (org_id, name) VALUES ($1,$2) RETURNING workspace_id`, orgID, "Workspace")
}
// signedInCookie writes a signed-in session into the stack's session store,
// through a session manager of the same shape the console builds, and returns
// the cookie a browser would send.
func (e *bootEnv) signedInCookie() *http.Cookie {
e.t.Helper()
pool := &redis.Pool{
MaxIdle: 1,
Dial: func() (redis.Conn, error) {
return redis.Dial("tcp", e.valkeyAddr, redis.DialPassword(e.valkeyPassword))
},
}
e.t.Cleanup(func() { _ = pool.Close() })
sm := scs.New()
sm.Store = redisstore.New(pool)
ctx := authtest.SignIn(e.t, sm, auth.SignIn{
PersonID: "person-boot", OrgID: "org-boot", WorkspaceID: "workspace-boot", Name: "Member",
})
return authtest.Cookie(e.t, sm, ctx)
}
// runningConsole is a serveConsole in its own goroutine, serving on a loopback
// listener on a free port.
type runningConsole struct {
e *bootEnv
addr string
cancel context.CancelFunc
done chan error
once sync.Once
client *http.Client
}
// start runs serveConsole with a listener that sends its address on a channel,
// which is the happens-before edge the read of the SDK key needs.
func (e *bootEnv) start() *runningConsole {
e.t.Helper()
ctx, cancel := context.WithCancel(context.Background())
addrs := make(chan string, 1)
rc := &runningConsole{
e: e,
cancel: cancel,
done: make(chan error, 1),
client: &http.Client{
Transport: &http.Transport{DisableKeepAlives: true},
CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
},
}
listen := func() (net.Listener, error) {
l, err := net.Listen("tcp", "127.0.0.1:0")
if err == nil {
addrs <- l.Addr().String()
}
return l, err
}
go func() { rc.done <- e.serve(ctx, listen) }()
e.t.Cleanup(rc.stop)
select {
case rc.addr = <-addrs:
case err := <-rc.done:
e.t.Fatalf("serveConsole returned before it listened: %v", err)
case <-time.After(30 * time.Second):
e.t.Fatal("waited 30s for serveConsole to listen")
}
return rc
}
// serve runs serveConsole over the environment with the Stripe integration.
func (e *bootEnv) serve(ctx context.Context, listen func() (net.Listener, error)) error {
adapter := stripeintegration.New()
boot := bootState{
stripeMode: "test",
stripeKeyFingerprint: stripeintegration.FingerprintForKey(e.stripeKey),
specs: adapter.ConfigSpec(),
}
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
return serveConsole(ctx, logger, boot, e.database, []integrations.Integration{adapter}, listen)
}
// stop cancels the console and waits for serveConsole to return.
func (rc *runningConsole) stop() {
rc.once.Do(func() {
rc.cancel()
select {
case <-rc.done:
case <-time.After(15 * time.Second):
rc.e.t.Error("waited 15s for serveConsole to return after its context ended")
}
})
}
// answer is the status and headers of one response.
type answer struct {
status int
header http.Header
location string
}
// do sends a request and names what it waited for when it fails.
func (rc *runningConsole) do(ctx context.Context, what, method, path string, body []byte, header http.Header, cookie *http.Cookie) (answer, error) {
req, err := http.NewRequestWithContext(ctx, method, "http://"+rc.addr+path, bytes.NewReader(body))
if err != nil {
return answer{}, fmt.Errorf("%s: %w", what, err)
}
for k, v := range header {
req.Header[k] = v
}
if cookie != nil {
req.AddCookie(cookie)
}
resp, err := rc.client.Do(req)
if err != nil {
return answer{}, fmt.Errorf("%s: %w", what, err)
}
defer resp.Body.Close()
_, _ = io.Copy(io.Discard, resp.Body)
return answer{status: resp.StatusCode, header: resp.Header, location: resp.Header.Get("Location")}, nil
}
func (rc *runningConsole) get(ctx context.Context, what, path string, cookie *http.Cookie) answer {
rc.e.t.Helper()
a, err := rc.do(ctx, what, http.MethodGet, path, nil, nil, cookie)
if err != nil {
rc.e.t.Fatalf("waited for %v", err)
}
return a
}
// expect fails the test when an answer's status is not want.
func expect(t *testing.T, what string, got answer, want int) {
t.Helper()
if got.status != want {
t.Errorf("%s answered %d, want %d", what, got.status, want)
}
}
// signedEvent returns a signed Stripe event of the environment's key mode and
// the header that signs it.
func (e *bootEnv) signedEvent(eventID string) ([]byte, http.Header) {
payload := []byte(fmt.Sprintf(
`{"id": %q, "type": "customer.created", "livemode": false, "created": 1757400000, "data": {"object": {"id": "cus_boot"}}}`, eventID))
signed := webhook.GenerateTestSignedPayload(&webhook.UnsignedPayload{Payload: payload, Secret: e.webhookSecret})
return payload, http.Header{"Stripe-Signature": {signed.Header}}
}
func (e *bootEnv) webhookRowStatus(eventID string) string {
e.t.Helper()
var status string
err := e.database.QueryRowContext(context.Background(),
`SELECT status FROM core.webhook_events WHERE provider = 'stripe' AND provider_event_id = $1`, eventID).Scan(&status)
if err != nil {
e.t.Fatalf("read the webhook row of %s: %v", eventID, err)
}
return status
}
// waitFor polls cond until it holds or the deadline passes, and fails naming
// what it waited for.
func waitFor(t *testing.T, what string, within time.Duration, cond func() bool) {
t.Helper()
deadline := time.Now().Add(within)
for !cond() {
if time.Now().After(deadline) {
t.Fatalf("waited %v for %s", within, what)
}
time.Sleep(10 * time.Millisecond)
}
}
// TestServeConsoleBeforeDependencies serves the real handler with a Temporal
// address that drops every connection and an identity provider that answers
// 503, and shows that everything which needs only the database answers within
// a second, that sign-in recovers when the provider does, and that nothing
// outlives the console.
func TestServeConsoleBeforeDependencies(t *testing.T) {
rows := []struct {
name string
check func(t *testing.T)
}{
{"serves before Temporal and the identity provider answer", checkServesBeforeDependencies},
{"a port in use starts nothing", checkPortInUseStartsNothing},
{"a session store that refuses the connection", checkServesWithoutASessionStore},
{"returns only after the connector has", checkReturnsAfterTheConnector},
{"serving that fails while Temporal is unreachable returns", checkReturnsWhenServingFails},
}
for _, row := range rows {
t.Run(row.name, row.check)
}
}
func checkServesBeforeDependencies(t *testing.T) {
e := newBootEnv(t)
rc := e.start()
// Within one second of the address: the routes that need the database.
ctx, cancel := context.WithTimeout(context.Background(), bootWait)
defer cancel()
expect(t, "the ask for a seeded name", rc.get(ctx, "the ask for a seeded name", "/domains/ask?domain="+e.servable, nil), http.StatusOK)
if got := stripetest.SDKKey(); got != e.stripeKey {
t.Errorf("the SDK key after the first answer = %q, want the configured key: serveConsole sets it before it listens", got)
}
expect(t, "the ask for an unknown name", rc.get(ctx, "the ask for an unknown name", "/domains/ask?domain="+e.unknown, nil), http.StatusNotFound)
expect(t, "/static/app.css", rc.get(ctx, "/static/app.css", "/static/app.css", nil), http.StatusOK)
login := rc.get(ctx, "/login before discovery", "/login", nil)
expect(t, "/login before the identity provider answers", login, http.StatusServiceUnavailable)
if got := login.header.Get("Retry-After"); got != "5" {
t.Errorf("/login Retry-After = %q, want 5", got)
}
eventID := "evt_serve_" + fmt.Sprint(time.Now().UnixNano())
t.Cleanup(func() {
_, _ = e.database.Exec(`DELETE FROM core.webhook_events WHERE provider = 'stripe' AND provider_event_id = $1`, eventID)
})
payload, header := e.signedEvent(eventID)
event, err := rc.do(ctx, "a signed Stripe event", http.MethodPost, "/webhooks/stripe", payload, header, nil)
if err != nil {
t.Fatalf("waited for %v", err)
}
expect(t, "a signed Stripe event before Temporal connects", event, http.StatusServiceUnavailable)
if got := e.webhookRowStatus(eventID); got != "received" {
t.Errorf("the event's row reads %q, want received", got)
}
// The connector has dialled the address that drops its connections, so the
// count below can show a stray dial after the console stops.
waitFor(t, "Temporal to be dialled", 10*time.Second, func() bool { return e.temporal.count.Load() > 0 })
// After the switch the loop publishes the document without a restart.
e.idp.usable.Store(true)
switched := time.Now()
waitFor(t, "/login to redirect to the identity provider after the switch", 5*time.Second, func() bool {
a, err := rc.do(context.Background(), "/login after the switch", http.MethodGet, "/login", nil, nil, nil)
return err == nil && strings.HasPrefix(a.location, e.idp.srv.URL+"/auth")
})
t.Logf("sign-in recovered %v after the switch", time.Since(switched))
rc.stop()
assertNothingOutlives(t, e)
}
// assertNothingOutlives compares the count the Temporal address saw when
// serveConsole returned with the count 500 milliseconds later: a connector
// that outlived it and dialled again within its shortened wait is seen. It does
// not cover the discovery loop: a loop that outlives serveConsole cannot be
// told from one that returned, because its HTTP client returns as soon as its
// context ends.
func assertNothingOutlives(t *testing.T, e *bootEnv) {
t.Helper()
temporalAtReturn := e.temporal.count.Load()
time.Sleep(500 * time.Millisecond)
if got := e.temporal.count.Load(); got != temporalAtReturn {
t.Errorf("Temporal was dialled %d times after serveConsole returned", got-temporalAtReturn)
}
}
func checkPortInUseStartsNothing(t *testing.T) {
e := newBootEnv(t)
occupied, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
t.Cleanup(func() { _ = occupied.Close() })
listen := func() (net.Listener, error) { return net.Listen("tcp", occupied.Addr().String()) }
err = e.serve(context.Background(), listen)
if !errors.Is(err, syscall.EADDRINUSE) {
t.Fatalf("serveConsole returned %v, want the error of binding an address in use", err)
}
time.Sleep(500 * time.Millisecond)
if got := e.temporal.count.Load(); got != 0 {
t.Errorf("Temporal was dialled %d times after a failed bind, want none", got)
}
if got := e.idp.requests.Load(); got != 0 {
t.Errorf("the identity provider got %d requests after a failed bind, want none", got)
}
}
// checkReturnsAfterTheConnector holds the connector's dial open, with a dial
// that outlasts its context by 200 milliseconds, and shows that serveConsole
// does not return before that dial has: a console that returned early would
// leave the connector running into its caller's cleanup.
func checkReturnsAfterTheConnector(t *testing.T) {
e := newBootEnv(t)
var started, finished atomic.Bool
prev := temporalDial
temporalDial = func(ctx context.Context, _ workflows.ClientConfig) (client.Client, error) {
started.Store(true)
<-ctx.Done()
time.Sleep(200 * time.Millisecond)
finished.Store(true)
return nil, ctx.Err()
}
t.Cleanup(func() { temporalDial = prev })
rc := e.start()
waitFor(t, "the connector to dial", 10*time.Second, started.Load)
rc.stop()
if !finished.Load() {
t.Error("serveConsole returned before connectTemporal returned")
}
// A console that returned early is seen by -race once the held dial
// returns and the connector writes its result: the test waits for it.
waitFor(t, "the held dial to return", 5*time.Second, finished.Load)
}
// checkReturnsWhenServingFails closes the listener once the connector is
// dialling an address that drops every connection, which makes the server
// fail, and shows that serveConsole returns that error and that no dial follows
// it: serving that ends for any reason ends the connector (design D18).
func checkReturnsWhenServingFails(t *testing.T) {
e := newBootEnv(t)
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
listeners := make(chan net.Listener, 1)
listen := func() (net.Listener, error) {
l, err := net.Listen("tcp", "127.0.0.1:0")
if err == nil {
listeners <- l
}
return l, err
}
done := make(chan error, 1)
go func() { done <- e.serve(ctx, listen) }()
var l net.Listener
select {
case l = <-listeners:
case err := <-done:
t.Fatalf("serveConsole returned before it listened: %v", err)
case <-time.After(30 * time.Second):
t.Fatal("waited 30s for serveConsole to listen")
}
waitFor(t, "Temporal to be dialled", 10*time.Second, func() bool { return e.temporal.count.Load() > 0 })
_ = l.Close()
select {
case err := <-done:
if !errors.Is(err, net.ErrClosed) {
t.Errorf("serveConsole returned %v, want the error of serving on a closed listener", err)
}
case <-time.After(5 * time.Second):
t.Fatal("serveConsole did not return after serving failed")
}
assertNothingOutlives(t, e)
}
func checkServesWithoutASessionStore(t *testing.T) {
e := newBootEnv(t)
refused, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
setConfig(t, "valkey-addr", refused.Addr().String())
_ = refused.Close()
rc := e.start()
ctx, cancel := context.WithTimeout(context.Background(), bootWait)
defer cancel()
for _, cookie := range []*http.Cookie{nil, e.member} {
label := "without a session cookie"
if cookie != nil {
label = "with a session cookie"
}
expect(t, "the ask "+label, rc.get(ctx, "the ask "+label, "/domains/ask?domain="+e.servable, cookie), http.StatusOK)
expect(t, "the ask for an unknown name "+label, rc.get(ctx, "the ask for an unknown name "+label, "/domains/ask?domain="+e.unknown, cookie), http.StatusNotFound)
expect(t, "/static/app.css "+label, rc.get(ctx, "/static/app.css "+label, "/static/app.css", cookie), http.StatusOK)
}
root := rc.get(ctx, "an anonymous /", "/", nil)
expect(t, "an anonymous /", root, http.StatusFound)
if root.location != "/login" {
t.Errorf("an anonymous / redirects to %q, want /login", root.location)
}
}
// TestServeConsoleUnderLoad runs the first row with twenty goroutines
// requesting /login, the ask and a signed-in page across the switch, under
// -race: the publication of the discovery document races nothing a request
// reads. The Temporal holder stays empty, because the address drops every
// connection; TestHolderUnderConcurrentReads covers its publication.
func TestServeConsoleUnderLoad(t *testing.T) {
e := newBootEnv(t)
rc := e.start()
var failures struct {
sync.Mutex
list []string
}
fail := func(msg string) {
failures.Lock()
defer failures.Unlock()
failures.list = append(failures.list, msg)
}
var served atomic.Int64
stop := make(chan struct{})
var wg sync.WaitGroup
for range 20 {
wg.Add(1)
go func() {
defer wg.Done()
for {
select {
case <-stop:
return
default:
}
checkLoadRound(rc, e, fail)
served.Add(1)
}
}()
}
waitFor(t, "requests to be served before the switch", 10*time.Second, func() bool { return served.Load() >= 20 })
e.idp.usable.Store(true)
waitFor(t, "/login to redirect after the switch", 10*time.Second, func() bool {
a, err := rc.do(context.Background(), "/login", http.MethodGet, "/login", nil, nil, nil)
return err == nil && strings.HasPrefix(a.location, e.idp.srv.URL+"/auth")
})
after := served.Load()
waitFor(t, "requests to be served after the switch", 20*time.Second, func() bool { return served.Load() >= after+100 })
close(stop)
wg.Wait()
rc.stop()
for _, msg := range failures.list {
t.Error(msg)
}
}
// checkLoadRound makes the three requests of one round and reports an answer
// that is not one the state allows.
func checkLoadRound(rc *runningConsole, e *bootEnv, fail func(string)) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
login, err := rc.do(ctx, "/login", http.MethodGet, "/login", nil, nil, nil)
if err != nil {
fail(err.Error())
} else if login.status != http.StatusServiceUnavailable && login.status != http.StatusFound {
fail(fmt.Sprintf("/login answered %d, want 503 or 302", login.status))
}
ask, err := rc.do(ctx, "the ask", http.MethodGet, "/domains/ask?domain="+e.servable, nil, nil, nil)
if err != nil {
fail(err.Error())
} else if ask.status != http.StatusOK {
fail(fmt.Sprintf("the ask answered %d, want 200", ask.status))
}
page, err := rc.do(ctx, "a signed-in page", http.MethodGet, "/partials/x", nil, nil, e.member)
if err != nil {
fail(err.Error())
} else if page.status != http.StatusNotFound {
fail(fmt.Sprintf("the signed-in page answered %d, want its styled 404", page.status))
}
}
// TestServeConsoleStepFailures shows that a step before the listener returns
// its failure as the step's own message, which Run logs, and that a Temporal
// authorization that cannot be built refuses before the listener opens.
func TestServeConsoleStepFailures(t *testing.T) {
rows := []struct {
name string
check func(t *testing.T)
}{
{"overrides that cannot be loaded", checkOverridesCannotBeLoaded},
{"a Temporal authorization that cannot be built", checkTemporalAuthorizationRefuses},
}
for _, row := range rows {
t.Run(row.name, row.check)
}
}
// serveRefused runs serveConsole over a listener that records being asked, and
// returns what it returned and whether it listened.
func (e *bootEnv) serveRefused() (listened bool, err error) {
var asked atomic.Bool
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
err = e.serve(ctx, func() (net.Listener, error) {
asked.Store(true)
return net.Listen("tcp", "127.0.0.1:0")
})
return asked.Load(), err
}
// assertStepFailure checks that err is a step failure with the message and
// that the report Run makes of it is one Error line with that message.
func assertStepFailure(t *testing.T, err error, message string) {
t.Helper()
var step *startStepError
if !errors.As(err, &step) {
t.Fatalf("serveConsole returned %v, want a step failure", err)
}
if step.message != message {
t.Errorf("the step failure's message = %q, want %q", step.message, message)
}
logs := &connectLogs{}
reportServeFailure(slog.New(logs), err)
lines := logs.withMessage(message)
if len(lines) != 1 || lines[0].level != slog.LevelError || logs.count() != 1 {
t.Errorf("the report logged %d lines, %d of them %q, want one Error line with the step's own message", logs.count(), len(lines), message)
}
}
func checkOverridesCannotBeLoaded(t *testing.T) {
e := newBootEnv(t)
if err := e.database.Close(); err != nil {
t.Fatalf("close the database: %v", err)
}
_, err := e.serveRefused()
assertStepFailure(t, err, "failed to load integration config overrides")
}
func checkTemporalAuthorizationRefuses(t *testing.T) {
e := newBootEnv(t)
setConfig(t, "temporal-oauth-scopes", []string{"temporal"})
listened, err := e.serveRefused()
if listened {
t.Error("serveConsole listened before it refused the Temporal authorization")
}
assertStepFailure(t, err, "failed to connect to Temporal")
}
// A failure that is not a step's reads as the server failing to start.
func TestReportServeFailure(t *testing.T) {
for _, tc := range []struct {
name string
err error
}{
{"an error from serving", errors.New("use of closed network connection")},
{"no error", nil},
} {
t.Run(tc.name, func(t *testing.T) {
logs := &connectLogs{}
reportServeFailure(slog.New(logs), tc.err)
lines := logs.withMessage("server failed to start")
if len(lines) != 1 || lines[0].level != slog.LevelError || logs.count() != 1 {
t.Errorf("the report logged %d lines, %d of them copy 21, want one Error line", logs.count(), len(lines))
}
})
}
}
+580 -388
View File
File diff suppressed because it is too large Load Diff
+171
View File
@@ -0,0 +1,171 @@
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package cmd
import (
"context"
"errors"
"log/slog"
"strings"
"time"
"git.coopcloud.tech/wiki-cafe/member-console/internal/retryloop"
"git.coopcloud.tech/wiki-cafe/member-console/internal/workflows"
"git.coopcloud.tech/wiki-cafe/member-console/internal/workflows/clientholder"
"github.com/spf13/viper"
"go.temporal.io/sdk/client"
)
// The connector's first wait and its cap (design D17). Package variables so
// that the boot test shortens them to milliseconds.
var (
temporalFirstWait = 500 * time.Millisecond
temporalMaxWait = 15 * time.Second
)
// temporalDial is the dial the console's connector uses. A package variable so
// that the boot test replaces it with a dial it can hold open.
var temporalDial = workflows.NewClient
// temporalWorker is the part of *workflows.Worker the connector uses.
type temporalWorker interface {
Start() error
Stop()
}
// temporalConnect is everything connectTemporal needs, read from
// configuration once, before the listener opens (design D13). The three
// functions are injected so that a test drives the connector without a
// Temporal server.
type temporalConnect struct {
clientConfig workflows.ClientConfig
workerConfig workflows.WorkerConfig
holder *clientholder.Holder
logger *slog.Logger
// escalation is temporal-connect-timeout: the time after which the
// connector's lines are Error. Zero means 90 seconds.
escalation time.Duration
firstWait time.Duration
maxWait time.Duration
// clock and rand are the retry loop's, the real ones when nil.
clock retryloop.Clock
rand retryloop.Source
dial func(ctx context.Context, cfg workflows.ClientConfig) (client.Client, error)
newWorker func(c client.Client, cfg workflows.WorkerConfig) (temporalWorker, error)
// afterConnect runs once, after the successful attempt and the client's
// publication: the integrations' startup hooks, the pollers and the
// schedules.
afterConnect func(ctx context.Context, c client.Client, w temporalWorker)
}
// temporalHost is the configured Temporal address, trimmed, so a value of
// only whitespace counts as unset.
func temporalHost() string {
return strings.TrimSpace(viper.GetString("temporal-host"))
}
// connectTemporal connects to Temporal for the life of the process: it
// retries the attempts of D6 with the policy of D17 until one succeeds or ctx
// ends. After a successful attempt, which the loop logs as copy 14, it
// publishes the client to the holder, runs afterConnect and returns the client
// and the started worker; when ctx ends first it returns nil, nil, having
// closed whatever it dialled, and has published nothing.
//
// One attempt dials, runs every provider's precondition, builds the worker and
// starts it, and ends there. An attempt that fails closes the client it dialled and is retried,
// except a precondition that refuses the start (design D10): that attempt keeps
// the client, starts no worker, publishes nothing, and reads the precondition
// again at the cap. The attempt logs nothing itself: every failure is returned,
// so that every line goes through the loop's decay and an hour of failures
// writes the bounded count.
func connectTemporal(ctx context.Context, t temporalConnect) (client.Client, temporalWorker) {
k := &temporalAttempts{t: t}
policy := retryloop.Policy{
FirstWait: t.firstWait,
Cap: t.maxWait,
Escalation: t.escalation,
FailureMessage: "Temporal not ready yet, retrying",
SuccessMessage: "connected to Temporal server",
Attrs: []slog.Attr{
slog.String("host", t.clientConfig.HostPort),
slog.String("namespace", t.clientConfig.Namespace),
},
Logger: t.logger,
Clock: t.clock,
Rand: t.rand,
}
if err := retryloop.Run(ctx, policy, k.attempt); err != nil {
k.closeClient()
return nil, nil
}
// Publish after the worker has started, so a workflow a handler starts has
// a worker, and before the first Startup hook, so the sweeps the hooks run
// see every row recorded while the holder was empty (design D7). Copy 14 has
// been logged by now, as the attempt ended at worker.Start.
t.holder.Publish(k.c)
t.afterConnect(ctx, k.c, k.w)
return k.c, k.w
}
// temporalAttempts is the state the attempts of one connectTemporal share: the
// client a refusal keeps, and the worker of the attempt that succeeded.
type temporalAttempts struct {
t temporalConnect
c client.Client
w temporalWorker
}
// closeClient closes the client the attempts hold, if any.
func (k *temporalAttempts) closeClient() {
if k.c != nil {
k.c.Close()
k.c = nil
}
}
// attempt is one connection attempt (design D6).
func (k *temporalAttempts) attempt(ctx context.Context) error {
if k.c == nil {
c, err := k.t.dial(ctx, k.t.clientConfig)
if err != nil {
return err
}
k.c = c
}
w, err := newCheckedWorker(ctx, k.c, k.t.workerConfig, k.t.newWorker)
if err != nil {
return k.workerFailure(err)
}
if err := w.Start(); err != nil {
k.closeClient()
return retryloop.Failure{Message: "failed to start Temporal worker", Level: slog.LevelError, Err: err}
}
k.w = w
return nil
}
// workerFailure is the attempt's error for a worker that was not built. A
// precondition that could not read what it checks (workflows.ErrCannotCheck)
// closes the client and is retried like a failed dial. A precondition that
// refuses keeps the client and is read again at the cap. Anything else is a
// worker that failed to build: the client is closed and the attempt retried.
func (k *temporalAttempts) workerFailure(err error) error {
var refused *workerRefusedError
switch {
case errors.Is(err, workflows.ErrCannotCheck):
k.closeClient()
return err
case errors.As(err, &refused):
return retryloop.AtCap(retryloop.Failure{
Message: "refused to start the Temporal worker",
Level: slog.LevelError,
Err: err,
})
default:
k.closeClient()
return retryloop.Failure{Message: "failed to create Temporal worker", Level: slog.LevelError, Err: err}
}
}
+535
View File
@@ -0,0 +1,535 @@
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
package cmd
import (
"context"
"errors"
"fmt"
"log/slog"
"slices"
"sync"
"testing"
"time"
"github.com/spf13/viper"
"go.temporal.io/api/workflowservice/v1"
"go.temporal.io/sdk/client"
"git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/fedwiki"
"git.coopcloud.tech/wiki-cafe/member-console/internal/retryloop"
"git.coopcloud.tech/wiki-cafe/member-console/internal/workflows"
"git.coopcloud.tech/wiki-cafe/member-console/internal/workflows/clientholder"
)
// connectClock is the retry loop's clock for a connector test. A timer fires
// at once and moves the clock by its duration, unless onWait returns false,
// in which case it never fires and the wait ends only with the context.
type connectClock struct {
mu sync.Mutex
now time.Time
waits []time.Duration
onWait func(d time.Duration) bool
}
type connectTimer struct {
c chan time.Time
}
func (t connectTimer) C() <-chan time.Time { return t.c }
func (t connectTimer) Stop() bool { return true }
func (c *connectClock) Now() time.Time {
c.mu.Lock()
defer c.mu.Unlock()
return c.now
}
func (c *connectClock) NewTimer(d time.Duration) retryloop.Timer {
c.mu.Lock()
c.waits = append(c.waits, d)
hook := c.onWait
c.mu.Unlock()
timer := connectTimer{c: make(chan time.Time, 1)}
if hook != nil && !hook(d) {
return timer
}
c.mu.Lock()
c.now = c.now.Add(d)
now := c.now
c.mu.Unlock()
timer.c <- now
return timer
}
// noJitter draws zero every time, so a wait is exactly its base.
type noJitter struct{}
func (noJitter) Float64() float64 { return 0 }
// connectLine is one captured log record.
type connectLine struct {
level slog.Level
message string
attrs map[string]slog.Value
}
// connectLogs is a slog handler that keeps what the connector logged.
type connectLogs struct {
mu sync.Mutex
lines []connectLine
}
func (*connectLogs) Enabled(context.Context, slog.Level) bool { return true }
func (h *connectLogs) WithAttrs([]slog.Attr) slog.Handler { return h }
func (h *connectLogs) WithGroup(string) slog.Handler { return h }
func (h *connectLogs) Handle(_ context.Context, r slog.Record) error {
line := connectLine{level: r.Level, message: r.Message, attrs: map[string]slog.Value{}}
r.Attrs(func(a slog.Attr) bool {
line.attrs[a.Key] = a.Value
return true
})
h.mu.Lock()
defer h.mu.Unlock()
h.lines = append(h.lines, line)
return nil
}
func (h *connectLogs) withMessage(message string) []connectLine {
h.mu.Lock()
defer h.mu.Unlock()
var out []connectLine
for _, l := range h.lines {
if l.message == message {
out = append(out, l)
}
}
return out
}
func (h *connectLogs) count() int {
h.mu.Lock()
defer h.mu.Unlock()
return len(h.lines)
}
// connectFixture is the fakes of one connector run: a dial that fails as many
// times as dialErrs says, a worker whose Start fails as startErrs says, and
// the record of what happened and in what order.
type connectFixture struct {
t *testing.T
holder *clientholder.Holder
clock *connectClock
logs *connectLogs
providers []workflows.WorkflowProvider
dialErrs []error
startErrs []error
// clientFor builds the client a successful dial returns.
clientFor func(f *connectFixture) client.Client
dials int
closed int
starts int
events []string
// connectedAtHooks is the number of copy 14 lines logged when the first
// afterConnect ran.
connectedAtHooks *int
}
func newConnectFixture(t *testing.T) *connectFixture {
t.Helper()
f := &connectFixture{
t: t,
holder: clientholder.New(),
clock: &connectClock{now: time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)},
logs: &connectLogs{},
}
f.clientFor = func(f *connectFixture) client.Client { return &fakeTemporalClient{f: f} }
return f
}
// fakeTemporalClient counts its closes. Its other methods are the nil
// interface's, which a connector must not call.
type fakeTemporalClient struct {
client.Client
f *connectFixture
}
func (c *fakeTemporalClient) Close() { c.f.closed++ }
// fakeTemporalWorker records its start with the state of the holder.
type fakeTemporalWorker struct{ f *connectFixture }
func (w fakeTemporalWorker) Start() error {
w.f.starts++
if i := w.f.starts - 1; i < len(w.f.startErrs) && w.f.startErrs[i] != nil {
return w.f.startErrs[i]
}
w.f.events = append(w.f.events, "start:"+holderState(w.f.holder))
return nil
}
func (fakeTemporalWorker) Stop() {}
func holderState(h *clientholder.Holder) string {
if h.Client() == nil {
return "empty"
}
return "published"
}
func (f *connectFixture) connect() temporalConnect {
return temporalConnect{
clientConfig: workflows.ClientConfig{HostPort: "temporal.test:7233", Namespace: "default"},
workerConfig: workflows.WorkerConfig{WorkflowProviders: f.providers},
holder: f.holder,
logger: slog.New(f.logs),
clock: f.clock,
rand: noJitter{},
dial: func(context.Context, workflows.ClientConfig) (client.Client, error) {
f.dials++
if i := f.dials - 1; i < len(f.dialErrs) && f.dialErrs[i] != nil {
return nil, f.dialErrs[i]
}
return f.clientFor(f), nil
},
newWorker: func(client.Client, workflows.WorkerConfig) (temporalWorker, error) {
return fakeTemporalWorker{f: f}, nil
},
afterConnect: func(context.Context, client.Client, temporalWorker) {
f.events = append(f.events, "afterConnect:"+holderState(f.holder))
if f.connectedAtHooks == nil {
n := len(f.logs.withMessage("connected to Temporal server"))
f.connectedAtHooks = &n
}
},
}
}
// sequencedPrecondition is a WorkflowProvider whose precondition answers errs
// in turn and passes after them, recording the holder's state at each reading.
type sequencedPrecondition struct {
plainProvider
f *connectFixture
errs []error
states *[]string
calls *int
}
func (p sequencedPrecondition) CheckBeforeWorker(context.Context, client.Client) error {
*p.states = append(*p.states, holderState(p.f.holder))
i := *p.calls
*p.calls++
if i < len(p.errs) {
return p.errs[i]
}
return nil
}
func (f *connectFixture) sequence(errs ...error) (states *[]string, calls *int) {
states, calls = new([]string), new(int)
f.providers = []workflows.WorkflowProvider{sequencedPrecondition{f: f, errs: errs, states: states, calls: calls}}
return states, calls
}
func TestConnectTemporal(t *testing.T) {
rows := []struct {
name string
check func(t *testing.T)
}{
{"refused dials, then a dial that succeeds", checkRefusedDialsThenSuccess},
{"a precondition that refuses twice and then passes", checkPreconditionRefusesTwice},
{"a precondition that cannot check", checkPreconditionCannotCheck},
{"the real FedWiki precondition that cannot list", checkFedWikiListFailure},
{"worker start that fails once", checkWorkerStartFailsOnce},
{"worker start that fails for an hour", checkWorkerStartFailsForAnHour},
{"a context that ends during a wait", checkContextEndsDuringAWait},
{"failures past the escalation time", checkFailuresPastEscalation},
}
for _, row := range rows {
t.Run(row.name, row.check)
}
}
func checkRefusedDialsThenSuccess(t *testing.T) {
f := newConnectFixture(t)
refused := errors.New("connection refused")
f.dialErrs = []error{refused, refused, refused}
c, w := connectTemporal(t.Context(), f.connect())
if c == nil || w == nil {
t.Fatal("connectTemporal returned no connection after the fourth dial succeeded")
}
if f.holder.Client() == nil {
t.Error("the holder is empty after a successful attempt")
}
want := []string{"start:empty", "afterConnect:published"}
if !slices.Equal(f.events, want) {
t.Errorf("events = %v, want %v: the worker starts before the client is published, and afterConnect runs after", f.events, want)
}
if f.dials != 4 {
t.Errorf("dials = %d, want 4", f.dials)
}
wantWaits := []time.Duration{500 * time.Millisecond, time.Second, 2 * time.Second}
if !slices.Equal(f.clock.waits, wantWaits) {
t.Errorf("waits = %v, want %v", f.clock.waits, wantWaits)
}
failed := f.logs.withMessage("Temporal not ready yet, retrying")
if len(failed) != 1 || failed[0].level != slog.LevelWarn {
t.Errorf("failure lines = %v, want one at Warn: the same error is logged once", failed)
}
connected := f.logs.withMessage("connected to Temporal server")
if len(connected) != 1 {
t.Fatalf("connected lines = %d, want 1", len(connected))
}
if f.connectedAtHooks == nil || *f.connectedAtHooks != 1 {
t.Errorf("connected lines when the hooks first ran = %v, want 1: copy 14 is logged before the Startup hooks run", f.connectedAtHooks)
}
if got := connected[0].attrs["attempts"].Int64(); got != 4 {
t.Errorf("connected line attempts = %d, want 4", got)
}
if connected[0].attrs["host"].String() != "temporal.test:7233" || connected[0].attrs["namespace"].String() != "default" {
t.Errorf("connected line host/namespace = %v/%v", connected[0].attrs["host"], connected[0].attrs["namespace"])
}
}
func checkPreconditionRefusesTwice(t *testing.T) {
f := newConnectFixture(t)
refusal := errors.New("1 FedWiki workflow run of types this release removed is still open")
states, calls := f.sequence(refusal, refusal)
c, _ := connectTemporal(t.Context(), f.connect())
if c == nil {
t.Fatal("connectTemporal returned no connection once the precondition passed")
}
if f.dials != 1 {
t.Errorf("dials = %d, want 1: a refusal keeps the client", f.dials)
}
if *calls != 3 {
t.Errorf("the precondition was read %d times, want 3", *calls)
}
if f.starts != 1 {
t.Errorf("the worker started %d times, want once the precondition passed", f.starts)
}
if !slices.Equal(*states, []string{"empty", "empty", "empty"}) {
t.Errorf("holder at each reading = %v, want empty throughout", *states)
}
capWait := []time.Duration{15 * time.Second, 15 * time.Second}
if !slices.Equal(f.clock.waits, capWait) {
t.Errorf("waits = %v, want %v: the next reading is a cap's wait after the last", f.clock.waits, capWait)
}
if f.closed != 0 {
t.Errorf("the client was closed %d times, want 0", f.closed)
}
lines := f.logs.withMessage("refused to start the Temporal worker")
if len(lines) != 1 || lines[0].level != slog.LevelError {
t.Errorf("refusal lines = %v, want one at Error from the first line", lines)
}
}
func checkPreconditionCannotCheck(t *testing.T) {
f := newConnectFixture(t)
states, _ := f.sequence(fmt.Errorf("list the runs: %w", workflows.ErrCannotCheck))
c, _ := connectTemporal(t.Context(), f.connect())
if c == nil {
t.Fatal("connectTemporal returned no connection once the precondition passed")
}
if f.closed != 1 {
t.Errorf("the client was closed %d times, want once after the unreadable precondition", f.closed)
}
if f.dials != 2 {
t.Errorf("dials = %d, want 2: the next attempt dials again", f.dials)
}
if !slices.Equal(f.clock.waits, []time.Duration{500 * time.Millisecond}) {
t.Errorf("waits = %v, want the backoff's first wait", f.clock.waits)
}
if len(*states) != 2 {
t.Errorf("the precondition was read %d times, want 2", len(*states))
}
}
// listingClient is a client whose visibility list fails as many times as
// listErrs says and then answers an empty page: the input of FedWiki's real
// precondition.
type listingClient struct {
fakeTemporalClient
listErrs []error
lists int
}
func (c *listingClient) ListWorkflow(context.Context, *workflowservice.ListWorkflowExecutionsRequest) (*workflowservice.ListWorkflowExecutionsResponse, error) {
i := c.lists
c.lists++
if i < len(c.listErrs) {
return nil, c.listErrs[i]
}
return &workflowservice.ListWorkflowExecutionsResponse{}, nil
}
func checkFedWikiListFailure(t *testing.T) {
f := newConnectFixture(t)
f.providers = []workflows.WorkflowProvider{fedwiki.Adapter{}}
var listing *listingClient
f.clientFor = func(f *connectFixture) client.Client {
listing = &listingClient{fakeTemporalClient: fakeTemporalClient{f: f}}
if f.dials == 1 {
listing.listErrs = []error{errors.New("visibility unavailable")}
}
return listing
}
c, _ := connectTemporal(t.Context(), f.connect())
if c == nil {
t.Fatal("connectTemporal returned no connection once the list could be read")
}
if f.dials != 2 {
t.Errorf("dials = %d, want 2: an unreadable list is retried on a new connection", f.dials)
}
if f.closed != 1 {
t.Errorf("the client was closed %d times, want once", f.closed)
}
}
func checkWorkerStartFailsOnce(t *testing.T) {
f := newConnectFixture(t)
f.startErrs = []error{errors.New("namespace call failed")}
c, _ := connectTemporal(t.Context(), f.connect())
if c == nil {
t.Fatal("connectTemporal returned no connection after the second start succeeded")
}
if f.closed != 1 {
t.Errorf("the client was closed %d times, want once after the failed start", f.closed)
}
if f.dials != 2 {
t.Errorf("dials = %d, want 2: the next attempt dials again", f.dials)
}
if !slices.Equal(f.clock.waits, []time.Duration{500 * time.Millisecond}) {
t.Errorf("waits = %v, want the backoff's first wait", f.clock.waits)
}
lines := f.logs.withMessage("failed to start Temporal worker")
if len(lines) != 1 || lines[0].level != slog.LevelError {
t.Errorf("start-failure lines = %v, want one at Error from the first line", lines)
}
}
func checkWorkerStartFailsForAnHour(t *testing.T) {
f := newConnectFixture(t)
f.startErrs = slices.Repeat([]error{errors.New("namespace call failed")}, 10000)
start := f.clock.now
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
f.clock.onWait = func(time.Duration) bool {
if f.clock.now.Sub(start) >= time.Hour {
cancel()
return false
}
return true
}
c, _ := connectTemporal(ctx, f.connect())
if c != nil || f.holder.Client() != nil {
t.Fatal("a worker that never starts published a client")
}
lines := f.logs.withMessage("failed to start Temporal worker")
if n := len(lines); n > 14 || n < 2 {
t.Errorf("an hour of failed starts wrote %d lines, want at most 14: the attempt logs nothing outside the decay", n)
}
for _, l := range lines {
if l.level != slog.LevelError {
t.Errorf("a start-failure line is at %v, want Error", l.level)
}
}
if total := f.logs.count(); total != len(lines) {
t.Errorf("the log holds %d lines, %d of them copy 16: nothing else is written", total, len(lines))
}
if f.starts < 100 {
t.Errorf("the worker was started %d times in an hour of fake time, want a few hundred", f.starts)
}
}
func checkContextEndsDuringAWait(t *testing.T) {
f := newConnectFixture(t)
f.dialErrs = slices.Repeat([]error{errors.New("connection refused")}, 100)
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
f.clock.onWait = func(time.Duration) bool {
cancel()
return false
}
c, w := connectTemporal(ctx, f.connect())
if c != nil || w != nil {
t.Error("connectTemporal returned a connection after its context ended")
}
if f.holder.Client() != nil {
t.Error("connectTemporal published a client after its context ended")
}
if f.dials != 1 {
t.Errorf("dials = %d, want 1: no attempt follows the wait that ended", f.dials)
}
}
func checkFailuresPastEscalation(t *testing.T) {
f := newConnectFixture(t)
// Fifteen refused dials pass two minutes of fake time at the cap.
f.dialErrs = slices.Repeat([]error{errors.New("connection refused")}, 15)
connect := f.connect()
connect.escalation = 30 * time.Second
c, _ := connectTemporal(t.Context(), connect)
if c == nil {
t.Fatal("connectTemporal gave up past the escalation time, as the old budget did")
}
lines := f.logs.withMessage("Temporal not ready yet, retrying")
if len(lines) < 2 {
t.Fatalf("failure lines = %d, want the first and the first past the escalation time", len(lines))
}
if lines[0].level != slog.LevelWarn {
t.Errorf("the first line is at %v, want Warn", lines[0].level)
}
last := lines[len(lines)-1]
if last.level != slog.LevelError {
t.Errorf("a line past the escalation time is at %v, want Error", last.level)
}
if got := last.attrs["elapsed"].Duration(); got < 30*time.Second {
t.Errorf("the first Error line was written at %v, before the escalation time", got)
}
}
// temporal-host of only whitespace is unset: the connector does not start.
func TestTemporalHostIsTrimmed(t *testing.T) {
prev, had := viper.Get("temporal-host"), viper.IsSet("temporal-host")
t.Cleanup(func() {
if had {
viper.Set("temporal-host", prev)
} else {
viper.Set("temporal-host", nil)
}
})
for _, tc := range []struct{ configured, want string }{
{" \t\n", ""},
{"", ""},
{" temporal.test:7233\n", "temporal.test:7233"},
} {
viper.Set("temporal-host", tc.configured)
if got := temporalHost(); got != tc.want {
t.Errorf("temporalHost() with %q = %q, want %q", tc.configured, got, tc.want)
}
if _, ok, err := newTemporalConnect(slog.New(&connectLogs{}), nil, bootState{}, nil, clientholder.New()); err != nil || ok != (tc.want != "") {
t.Errorf("newTemporalConnect with %q reported %t, want %t", tc.configured, ok, tc.want != "")
}
}
}
+7 -2
View File
@@ -441,11 +441,11 @@ type openRunsError struct {
func (e *openRunsError) Error() string {
if e.total == 1 {
return fmt.Sprintf("1 FedWiki workflow run of types this release removed is still open: %s. "+
"Let it finish on the previous release, or terminate it, before starting this release.",
"Let it finish on the previous release, or terminate it; the worker starts once none is open.",
strings.Join(e.counts, ", "))
}
return fmt.Sprintf("%d FedWiki workflow runs of types this release removed are still open: %s. "+
"Let them finish on the previous release, or terminate them, before starting this release.",
"Let them finish on the previous release, or terminate them; the worker starts once none is open.",
e.total, strings.Join(e.counts, ", "))
}
@@ -458,6 +458,11 @@ func (e *listRunsError) Error() string {
func (e *listRunsError) Unwrap() error { return e.err }
// Is reports workflows.ErrCannotCheck: the list could not be read, so the
// connector retries the connection; an openRunsError is a refusal and does
// not report it.
func (e *listRunsError) Is(target error) bool { return target == corework.ErrCannotCheck }
// CheckBeforeWorker refuses the start while a run of a workflow type this
// release removed is open (workflows.WorkerPrecondition): a worker without
// the type fails every task of such a run and leaves its farm call and row
+2 -2
View File
@@ -420,7 +420,7 @@ func (h *FedWikiHandler) CreateSite(w http.ResponseWriter, r *http.Request) {
// Check if Temporal client is available
if tc == nil {
h.Logger.Error("Temporal client not configured")
h.Logger.Error("Temporal is not connected")
respondError(w, http.StatusServiceUnavailable, msgServiceAvailable, h.SupportURL)
return
}
@@ -545,7 +545,7 @@ func (h *FedWikiHandler) DeleteSite(w http.ResponseWriter, r *http.Request) {
}
if tc == nil {
h.Logger.Error("Temporal client not configured")
h.Logger.Error("Temporal is not connected")
respondError(w, http.StatusServiceUnavailable, msgServiceAvailable, h.SupportURL)
return
}
@@ -149,7 +149,7 @@ func startClaimVerification(
connectTarget string,
) (*customDomainInitiation, string) {
if temporalClient == nil {
logger.Error("Temporal client not configured")
logger.Error("Temporal is not connected")
return nil, "Domain verification is not available right now."
}
@@ -149,7 +149,7 @@ func (h *FedWikiPartialsHandler) CreateSite(w http.ResponseWriter, r *http.Reque
}
if tc == nil {
h.Logger.Error("Temporal client not configured")
h.Logger.Error("Temporal is not connected")
errs.Form(msgServiceAvailable)
h.renderCreateFormRefusal(w, r, session.WorkspaceID, values, errs, false)
return
@@ -343,7 +343,7 @@ func (h *FedWikiPartialsHandler) beginRowAction(w http.ResponseWriter, r *http.R
return rowAction{}, false
}
if h.Temporal.Client() == nil {
h.Logger.Error("Temporal client not configured")
h.Logger.Error("Temporal is not connected")
h.siteActionResult(w, msgServiceAvailable)
return rowAction{}, false
}
@@ -60,7 +60,7 @@ func TestCheckNoRemovedRunsNamesOpenTypesWithCounts(t *testing.T) {
}
want := "3 FedWiki workflow runs of types this release removed are still open: " +
"CreateFedWikiSiteWorkflow (2), SwapActiveSiteWorkflow (1). " +
"Let them finish on the previous release, or terminate them, before starting this release."
"Let them finish on the previous release, or terminate them; the worker starts once none is open."
if err.Error() != want {
t.Errorf("err = %q\nwant %q", err, want)
}
@@ -82,7 +82,7 @@ func TestCheckNoRemovedRunsSingularForOneRun(t *testing.T) {
}
want := "1 FedWiki workflow run of types this release removed is still open: " +
"SetSiteStatusWorkflow (1). " +
"Let it finish on the previous release, or terminate it, before starting this release."
"Let it finish on the previous release, or terminate it; the worker starts once none is open."
if err.Error() != want {
t.Errorf("err = %q\nwant %q", err, want)
}
+6 -4
View File
@@ -132,8 +132,9 @@ func (Adapter) ConfigSpec() []config.ConfigKey {
}
}
// RegisterRoutes mirrors today's server.Start conditional: the webhook
// route is only mounted when a webhook secret is configured. The secret
// RegisterRoutes is called by NewHandler's registration of the route
// mounts: the webhook route is only mounted when a webhook secret is
// configured. The secret
// is read directly from viper — Stripe's own concern, not threaded
// through server.Deps. ConfigSpec above declares this key.
//
@@ -297,8 +298,9 @@ func ModeForKey(apiKey string) (string, error) {
// UseKey assigns the stripe-go package-level API key when the trimmed key is
// not empty and does nothing otherwise. start calls it once, after the mode
// and the fingerprint are derived and before any goroutine starts; nothing
// assigns the key afterwards, so every later reader sees one value.
// and the fingerprint are derived, before the listener and before any
// goroutine that reads the key; nothing assigns the key afterwards, so every
// later reader sees one value.
func UseKey(apiKey string) {
if key := strings.TrimSpace(apiKey); key != "" {
stripego.Key = key
@@ -149,21 +149,34 @@ func NewWebhookActivities(db *sql.DB, logger *slog.Logger) *WebhookActivities {
}
// ProcessWebhookEvent transitions an event to 'processing', dispatches it,
// and marks it 'completed' or 'skipped'. A failed attempt is recorded on the
// and marks it 'completed' or 'skipped'. An event whose row is already
// finished (completed, skipped, refused or dead_letter) is returned nil
// without dispatching. A failed attempt is recorded on the
// row (`failed`, the attempt number Temporal reports, the error) before the
// error goes back to Temporal to schedule the next one, so the operator
// page and the row agree with the Temporal UI about where an event stands.
func (a *WebhookActivities) ProcessWebhookEvent(ctx context.Context, evt WebhookEvent) error {
// Transition to processing
_, err := a.DB.ExecContext(ctx,
// Transition to processing, only from a status that still owes work. The
// boot sweep and a redelivery can each start a finished event's workflow
// again: the workflow id is the event id and its reuse policy lets a
// completed id run once more, and the sweep's status read and the
// workflow start are two steps. A row that is already finished touches no
// row here, and the activity returns without dispatching (design F34).
marked, err := a.DB.ExecContext(ctx,
`UPDATE core.webhook_events
SET status = 'processing', updated_at = NOW()
WHERE id = $1 AND received_at = (SELECT received_at FROM core.webhook_events WHERE id = $1)`,
WHERE id = $1 AND received_at = (SELECT received_at FROM core.webhook_events WHERE id = $1)
AND status IN ('received','processing','failed')`,
evt.ID,
)
if err != nil {
return fmt.Errorf("mark processing: %w", err)
}
if n, err := marked.RowsAffected(); err != nil {
return fmt.Errorf("mark processing: %w", err)
} else if n == 0 {
return nil
}
// Dispatch to event-type handler.
finalStatus, err := a.dispatchEvent(ctx, evt)
@@ -9,6 +9,7 @@ import (
"io"
"log/slog"
"testing"
"time"
"github.com/google/uuid"
"go.temporal.io/sdk/testsuite"
@@ -64,6 +65,73 @@ func TestProcessWebhookEventRecordsAFailedAttemptOnTheRow(t *testing.T) {
}
}
// finishedEventCase is one status a webhook row can rest at when the activity
// runs, and whether the activity may dispatch it.
type finishedEventCase struct {
status string
dispatch bool
}
// The boot sweep and a redelivery can start a finished event's workflow again
// (design F34). The activity's first update applies only to a row that still
// owes work and returns when it touches none. The event type is one whose
// handler fails on an empty payload, so a dispatch shows as an error and as a
// row at 'failed'.
func TestProcessWebhookEventSkipsAFinishedEvent(t *testing.T) {
db := dbtest.Open(t)
acts := NewWebhookActivities(db, slog.New(slog.NewTextHandler(io.Discard, nil)))
cases := []finishedEventCase{
{"completed", false}, {"skipped", false}, {"refused", false}, {"dead_letter", false},
{"received", true}, {"processing", true}, {"failed", true},
}
for _, tc := range cases {
t.Run(tc.status, func(t *testing.T) { checkFinishedEventCase(t, db, acts, tc) })
}
}
func checkFinishedEventCase(t *testing.T, db *sql.DB, acts *WebhookActivities, tc finishedEventCase) {
t.Helper()
evt := insertEventRow(t, db, "invoice.paid", `{}`)
if _, err := db.ExecContext(context.Background(),
`UPDATE core.webhook_events SET status = $1 WHERE id = $2`, tc.status, evt.ID); err != nil {
t.Fatalf("set status %s: %v", tc.status, err)
}
var before time.Time
if err := db.QueryRowContext(context.Background(),
`SELECT updated_at FROM core.webhook_events WHERE id = $1`, evt.ID).Scan(&before); err != nil {
t.Fatalf("read updated_at: %v", err)
}
var suite testsuite.WorkflowTestSuite
env := suite.NewTestActivityEnvironment()
env.RegisterActivity(acts)
_, err := env.ExecuteActivity(acts.ProcessWebhookEvent, evt)
status, attempts, msg := eventRow(t, db, evt.ID)
if !tc.dispatch {
var after time.Time
if err := db.QueryRowContext(context.Background(),
`SELECT updated_at FROM core.webhook_events WHERE id = $1`, evt.ID).Scan(&after); err != nil {
t.Fatalf("read updated_at: %v", err)
}
if err != nil {
t.Errorf("a row at %s: activity returned %v, want nil without dispatching", tc.status, err)
}
if status != tc.status || attempts != 0 || msg != "" || !after.Equal(before) {
t.Errorf("a row at %s changed: status %q, retry_count %d, error %q, updated_at moved %t; want it untouched",
tc.status, status, attempts, msg, !after.Equal(before))
}
return
}
if err == nil {
t.Errorf("a row at %s: activity returned nil, want the dispatch error", tc.status)
}
if status != "failed" || attempts != 1 || msg == "" {
t.Errorf("a row at %s after the dispatch: status %q, retry_count %d, error %q; want failed/1/<the error>",
tc.status, status, attempts, msg)
}
}
func TestMarkEventDeadLetter(t *testing.T) {
db := dbtest.Open(t)
acts := NewWebhookActivities(db, slog.New(slog.NewTextHandler(io.Discard, nil)))
+1 -1
View File
@@ -432,7 +432,7 @@ func (h *MemberDomainsHandler) renderClaimStatus(w http.ResponseWriter, r *http.
func (h *MemberDomainsHandler) startClaimVerification(ctx context.Context, workspaceID, root string) (*domains.Claim, string) {
tc := h.Temporal.Client()
if tc == nil {
h.Logger.Error("Temporal client not configured")
h.Logger.Error("Temporal is not connected")
return nil, "Domain verification is not available right now."
}
@@ -39,7 +39,7 @@ func (h *OperatorPartialsHandler) startEntitlementRecomputeDrain(ctx context.Con
}
tc := h.Temporal.Client()
if tc == nil {
h.Logger.Warn("no Temporal client configured; the entitlement recompute drain was not started",
h.Logger.Warn("Temporal is not connected; the entitlement recompute drain was not started",
slog.String("change_id", result.ChangeID))
return
}
+1 -1
View File
@@ -198,7 +198,7 @@ func (h *OperatorPartialsHandler) startPlanActDrain(ctx context.Context, actID s
log := logging.FromContext(ctx).With(slog.String("act_id", actID))
tc := h.Temporal.Client()
if tc == nil {
log.Warn("no Temporal client configured; the plan act drain was not started")
log.Warn("Temporal is not connected; the plan act drain was not started")
return
}
if _, err := tc.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
+91 -36
View File
@@ -15,6 +15,7 @@ import (
"net/url"
"slices"
"strings"
"sync"
"time"
"git.coopcloud.tech/wiki-cafe/member-console/internal/auth"
@@ -36,7 +37,6 @@ import (
// Config holds the configuration for the server.
type Config struct {
Port string
Env string
Logger *slog.Logger
Database *sql.DB // Raw DB connection
@@ -328,68 +328,123 @@ func orgHasEntitlement(ctx context.Context, entitlementsQ entitlements.Querier,
return false
}
// Start sets up authentication, builds the console's handler with NewHandler
// and serves it on cfg.Port until the server fails. Authentication setup
// fetches nothing from the identity provider: Start runs the discovery loop
// (auth.Config.Discover) in the background after NewHandler and before it
// listens, and sign-in answers 503 until the loop has published a usable
// document.
func Start(ctx context.Context, cfg Config) error {
// Bound is a server whose authentication is set up, whose handler is built
// and whose listener is open. Bind returns it; Serve runs it.
type Bound struct {
cfg Config
auth *auth.Config
server *http.Server
listener net.Listener
}
// shutdownTimeout bounds the wait for in-flight requests when Serve's context
// ends.
const shutdownTimeout = 5 * time.Second
// Bind sets up authentication, builds the console's handler with NewHandler
// and opens the listener with listen, in that order, and returns the bound
// server (design D19). It returns the first error having started nothing: no
// goroutine runs until Serve. Nothing before listen waits on Temporal or the
// identity provider: the authentication setup probes the session store once
// and fetches nothing from the identity provider, so a console whose
// dependencies are down binds after one bounded probe of the session store. A failure of
// Bind, such as a port in use or a session store that refuses the console's
// credentials, starts no background work.
func Bind(cfg Config, listen func() (net.Listener, error)) (*Bound, error) {
// The registry answers /domains/ask below, on the TLS handshake path of
// every proxy in front of the deployment. A missing querier would mean
// certificate issuance silently stops, so it is a required dependency
// rather than a conditional feature.
if cfg.DomainsQ == nil {
cfg.Logger.Error("domains registry queries are required to serve /domains/ask")
return fmt.Errorf("server: Config.DomainsQ is required")
return nil, fmt.Errorf("server: Config.DomainsQ is required")
}
// Set up authentication with identity and organization query interfaces.
authConfig, err := auth.Setup(cfg.Database, cfg.IdentityQ, cfg.OrgQ)
if err != nil {
cfg.Logger.Error("failed to set up authentication", slog.Any("error", err))
return err
return nil, err
}
handler, err := NewHandler(cfg, authConfig)
if err != nil {
return err
return nil, err
}
// The identity provider's discovery document is fetched in the
// background, so the routes that need only the database are served
// before the provider answers; sign-in answers 503 until it has.
go authConfig.Discover(ctx)
// Create HTTP server
server := http.Server{
Addr: ":" + cfg.Port,
Handler: handler,
ReadTimeout: 4 * time.Second,
WriteTimeout: 8 * time.Second,
IdleTimeout: 16 * time.Second,
MaxHeaderBytes: 1024 * 1024, // 1MB
BaseContext: func(_ net.Listener) context.Context { return ctx }, // Pass base context to all requests
listener, err := listen()
if err != nil {
return nil, err
}
return &Bound{
cfg: cfg,
auth: authConfig,
listener: listener,
server: &http.Server{
Handler: handler,
ReadTimeout: 4 * time.Second,
WriteTimeout: 8 * time.Second,
IdleTimeout: 16 * time.Second,
MaxHeaderBytes: 1024 * 1024, // 1MB
},
}, nil
}
// Log server startup with structured logging
cfg.Logger.Info("starting server",
slog.String("port", cfg.Port),
slog.String("environment", cfg.Env),
slog.String("address", "http://localhost:"+cfg.Port))
// Serve starts the discovery loop (auth.Config.Discover) in the background
// and serves on the bound listener until the server fails or ctx ends. The
// discovery document is fetched in the background, so the routes that need
// only the database are served before the provider answers; sign-in answers
// 503 until the loop has published a usable document. When ctx ends Serve
// calls Shutdown with a bounded context, waits for Discover to return and
// returns nil; a server failure returns its error, after the same wait. Serve
// returns once the listener is closed and the discovery loop has returned; a
// request still running after the five-second bound is cut off by Close.
func (b *Bound) Serve(ctx context.Context) error {
b.server.BaseContext = func(_ net.Listener) context.Context { return ctx } // pass the base context to all requests
// Start server and log any errors
if err := server.ListenAndServe(); err != nil {
cfg.Logger.Error("server error", slog.Any("error", err))
return err
discoverCtx, stopDiscovery := context.WithCancel(ctx)
var discovery sync.WaitGroup
discovery.Add(1)
go func() {
defer discovery.Done()
b.auth.Discover(discoverCtx)
}()
// The address comes from a bound TCP listener, so it always splits.
_, port, _ := net.SplitHostPort(b.listener.Addr().String())
b.cfg.Logger.Info("starting server",
slog.String("port", port),
slog.String("environment", b.cfg.Env),
slog.String("address", "http://localhost:"+port))
served := make(chan error, 1)
go func() { served <- b.server.Serve(b.listener) }()
var result error
select {
case err := <-served:
b.cfg.Logger.Error("server error", slog.Any("error", err))
_ = b.server.Close()
result = err
case <-ctx.Done():
shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), shutdownTimeout)
if err := b.server.Shutdown(shutdownCtx); err != nil {
b.cfg.Logger.Error("server shutdown", slog.Any("error", err))
_ = b.server.Close()
result = err
}
cancel()
<-served
}
return nil
stopDiscovery()
discovery.Wait()
return result
}
// NewHandler builds the console's HTTP handler from cfg: the auth routes,
// each mount in cfg.RouteMounts and cfg.UIMounts, every core route, and the
// middleware that runs before them. It sets authConfig.Failures to the
// member template set. cfg.DomainsQ must be set; Start refuses a Config
// member template set. cfg.DomainsQ must be set; Bind refuses a Config
// without it before it connects to anything.
func NewHandler(cfg Config, authConfig *auth.Config) (http.Handler, error) {
rt, err := newRouter(cfg, authConfig)
@@ -7,7 +7,7 @@
"checked": [],
"logs": [
"level=ERROR msg=\"a plan act move failed\" request_id=<uuid:1> act_id=<act:1> org_id=<org:Planless> pool_id=<pool:Planless> step=initiate error=\"reapply-defaults: confer default grant: confer grant: ERROR: confer: product <product:NoSet> has no entitlement set and confers nothing (SQLSTATE P0001)\"",
"level=WARN msg=\"no Temporal client configured; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
"level=WARN msg=\"Temporal is not connected; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
],
"statements": [
"begin",
@@ -7,7 +7,7 @@
"checked": [],
"logs": [
"level=ERROR msg=\"the plan act's listing batch failed\" request_id=<uuid:1> act_id=<act:1> error=\"list a batch of plan act <act:1>: read failed\"",
"level=WARN msg=\"no Temporal client configured; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
"level=WARN msg=\"Temporal is not connected; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
],
"statements": [
"begin",
@@ -7,7 +7,7 @@
"checked": [],
"logs": [
"level=ERROR msg=\"a plan act move failed\" request_id=<uuid:1> act_id=<act:1> org_id=<org:Holder> pool_id=<pool:Holder> step=restore error=\"reapply-defaults: confer default grant: confer grant: ERROR: confer: product <product:NoSet> has no entitlement set and confers nothing (SQLSTATE P0001)\"",
"level=WARN msg=\"no Temporal client configured; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
"level=WARN msg=\"Temporal is not connected; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
],
"statements": [
"begin",
@@ -6,7 +6,7 @@
],
"checked": [],
"logs": [
"level=WARN msg=\"no Temporal client configured; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
"level=WARN msg=\"Temporal is not connected; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
],
"statements": [
"begin",
@@ -7,7 +7,7 @@
"checked": [],
"logs": [
"level=ERROR msg=\"a plan act move failed\" request_id=<uuid:1> act_id=<act:1> org_id=<org:Planless> pool_id=<pool:Planless> step=initiate error=\"reapply-defaults: confer default grant: confer grant: ERROR: confer: product <product:NoSet> has no entitlement set and confers nothing (SQLSTATE P0001)\"",
"level=WARN msg=\"no Temporal client configured; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
"level=WARN msg=\"Temporal is not connected; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
],
"statements": [
"begin",
@@ -7,7 +7,7 @@
"checked": [],
"logs": [
"level=ERROR msg=\"the plan act's listing batch failed\" request_id=<uuid:1> act_id=<act:1> error=\"list a batch of plan act <act:1>: read failed\"",
"level=WARN msg=\"no Temporal client configured; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
"level=WARN msg=\"Temporal is not connected; the plan act drain was not started\" request_id=<uuid:1> act_id=<act:1>"
],
"statements": [
"begin",
+15
View File
@@ -18,6 +18,7 @@ import (
"strconv"
"strings"
"sync"
"testing"
stripe "github.com/stripe/stripe-go/v81"
"github.com/stripe/stripe-go/v81/form"
@@ -460,3 +461,17 @@ func Install(m *MockBackend) func() {
func KeyMode() string {
return stripestore.ModeForKey(stripe.Key)
}
// ClearSDKKey empties the Stripe SDK's API key and restores the previous value
// in t's cleanup, so a test can show that code under test sets the key.
func ClearSDKKey(t testing.TB) {
t.Helper()
prev := stripe.Key
stripe.Key = ""
t.Cleanup(func() { stripe.Key = prev })
}
// SDKKey returns the API key the Stripe SDK holds.
func SDKKey() string {
return stripe.Key
}
+12 -86
View File
@@ -7,7 +7,6 @@ import (
"context"
"fmt"
"log/slog"
"math/rand/v2"
"time"
"go.temporal.io/sdk/client"
@@ -19,22 +18,18 @@ type ClientConfig struct {
Namespace string
Logger *slog.Logger
OAuthTokenProvider *OAuthTokenProviderConfig
// ConnectTimeout bounds how long NewClient retries a failed initial Dial
// before giving up. Zero uses defaultConnectTimeout.
ConnectTimeout time.Duration
}
const (
// defaultConnectTimeout bounds the initial-Dial retry budget. It comfortably
// exceeds Temporal's ~1-minute JWKS refresh window, so a cold-start JWT
// rejection (server reachable, signing keys not yet loaded) self-heals
// without the operator having to re-run `start`.
defaultConnectTimeout = 90 * time.Second
connectInitialBackoff = 500 * time.Millisecond
connectMaxBackoff = 5 * time.Second
)
// dialTimeout is the deadline of one dial. It covers the token request's own
// ten-second timeout and the SDK's five-second system-info call. A package
// variable so a test can shorten it.
var dialTimeout = 20 * time.Second
// NewClient creates a new Temporal client.
// NewClient makes one attempt to connect to Temporal under a deadline of
// dialTimeout. It does not retry: the connector in cmd drives the attempts
// through internal/retryloop (design D6, D17). client.Dial issues an
// authenticated GetSystemInfo call, so a server that is reachable but rejects
// the token fails here too.
func NewClient(ctx context.Context, cfg ClientConfig) (client.Client, error) {
if cfg.HostPort == "" {
cfg.HostPort = "localhost:7233"
@@ -60,84 +55,15 @@ func NewClient(ctx context.Context, cfg ClientConfig) (client.Client, error) {
opts.Logger = newSlogAdapter(cfg.Logger)
}
timeout := cfg.ConnectTimeout
if timeout <= 0 {
timeout = defaultConnectTimeout
}
c, err := dialWithRetry(ctx, func() (client.Client, error) { return client.Dial(opts) },
timeout, connectInitialBackoff, connectMaxBackoff, cfg.Logger)
ctx, cancel := context.WithTimeout(ctx, dialTimeout)
defer cancel()
c, err := client.DialContext(ctx, opts)
if err != nil {
return nil, fmt.Errorf("failed to create Temporal client: %w", err)
}
if cfg.Logger != nil {
cfg.Logger.Info("connected to Temporal server",
slog.String("host", cfg.HostPort),
slog.String("namespace", cfg.Namespace))
}
return c, nil
}
// dialWithRetry dials Temporal, retrying transient boot-time failures with
// bounded exponential backoff + jitter. client.Dial eagerly issues an
// authenticated GetSystemInfo RPC, so a cold server — reachable but with a JWKS
// cache that has not yet loaded the identity provider's signing keys, briefly
// rejecting an otherwise-valid token — surfaces here as a Dial error. Rather
// than make that fatal (forcing the operator to re-run `start`), we retry within
// a bounded budget so the first boot after `docker compose up` succeeds on its
// own; the same loop rides out Temporal or IdP restarts in real deployments.
//
// dial is injected so the retry policy can be unit-tested without a live server.
func dialWithRetry(ctx context.Context, dial func() (client.Client, error), budget, initialBackoff, maxBackoff time.Duration, logger *slog.Logger) (client.Client, error) {
deadline := time.Now().Add(budget)
backoff := initialBackoff
for attempt := 1; ; attempt++ {
c, err := dial()
if err == nil {
if logger != nil && attempt > 1 {
logger.Info("connected to Temporal after retry", slog.Int("attempts", attempt))
}
return c, nil
}
if !time.Now().Before(deadline) {
return nil, fmt.Errorf("could not connect to Temporal after %d attempt(s) within %s: %w", attempt, budget, err)
}
sleep := jitter(backoff)
if until := time.Until(deadline); sleep > until {
sleep = until
}
if logger != nil {
logger.Warn("Temporal not ready yet, retrying",
slog.Int("attempt", attempt),
slog.Duration("retry_in", sleep),
slog.Any("error", err))
}
select {
case <-ctx.Done():
return nil, fmt.Errorf("connect to Temporal canceled after %d attempt(s): %w", attempt, ctx.Err())
case <-time.After(sleep):
}
if backoff *= 2; backoff > maxBackoff {
backoff = maxBackoff
}
}
}
// jitter returns d plus a random fraction up to d/2, so simultaneous clients
// don't reconnect in lockstep.
func jitter(d time.Duration) time.Duration {
if d <= 0 {
return 0
}
half := d / 2
if half <= 0 {
return d
}
return d + rand.N(half)
}
// slogAdapter adapts slog.Logger to Temporal's log.Logger interface.
type slogAdapter struct {
logger *slog.Logger
+62 -83
View File
@@ -5,95 +5,74 @@ package workflows
import (
"context"
"errors"
"net"
"sync"
"testing"
"time"
"go.temporal.io/sdk/client"
)
// TestDialWithRetry_SucceedsAfterTransientFailures models the cold-start JWKS
// race: the first attempts fail with an "unauthorized" error, then the server's
// key cache refreshes and the dial succeeds.
func TestDialWithRetry_SucceedsAfterTransientFailures(t *testing.T) {
calls := 0
dial := func() (client.Client, error) {
calls++
if calls < 3 {
return nil, errors.New("failed reaching server: Request unauthorized.")
}
return nil, nil
}
c, err := dialWithRetry(context.Background(), dial, time.Second, time.Millisecond, 5*time.Millisecond, nil)
// acceptAndHold opens a listener that accepts every connection and never
// answers, and returns its address. The connections close with the test.
func acceptAndHold(t *testing.T) string {
t.Helper()
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("expected success after retries, got: %v", err)
t.Fatalf("listen: %v", err)
}
if c != nil {
t.Fatalf("expected the stub's nil client, got non-nil")
}
if calls != 3 {
t.Fatalf("expected 3 dial attempts, got %d", calls)
}
}
// TestDialWithRetry_ExhaustsBudget asserts a persistently failing dial returns the
// last error (wrapped) once the budget elapses.
func TestDialWithRetry_ExhaustsBudget(t *testing.T) {
calls := 0
wantErr := errors.New("failed reaching server: Request unauthorized.")
dial := func() (client.Client, error) {
calls++
return nil, wantErr
}
_, err := dialWithRetry(context.Background(), dial, 20*time.Millisecond, time.Millisecond, 5*time.Millisecond, nil)
if err == nil {
t.Fatal("expected an error after exhausting the budget")
}
if !errors.Is(err, wantErr) {
t.Fatalf("expected the wrapped last error, got: %v", err)
}
if calls < 2 {
t.Fatalf("expected multiple attempts within the budget, got %d", calls)
}
}
// TestDialWithRetry_HealthySucceedsFirstTry asserts a healthy server connects on
// the first attempt with no backoff delay.
func TestDialWithRetry_HealthySucceedsFirstTry(t *testing.T) {
calls := 0
dial := func() (client.Client, error) {
calls++
return nil, nil
}
start := time.Now()
if _, err := dialWithRetry(context.Background(), dial, time.Second, 50*time.Millisecond, time.Second, nil); err != nil {
t.Fatalf("expected immediate success, got: %v", err)
}
if calls != 1 {
t.Fatalf("expected exactly 1 attempt, got %d", calls)
}
if elapsed := time.Since(start); elapsed > 20*time.Millisecond {
t.Fatalf("expected no backoff delay on first-try success, took %s", elapsed)
}
}
// TestDialWithRetry_RespectsContextCancellation asserts a canceled context aborts
// the retry loop promptly.
func TestDialWithRetry_RespectsContextCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
calls := 0
dial := func() (client.Client, error) {
calls++
if calls == 1 {
cancel() // cancel before the first backoff sleep
var mu sync.Mutex
var held []net.Conn
done := make(chan struct{})
go func() {
defer close(done)
for {
c, err := l.Accept()
if err != nil {
return
}
mu.Lock()
held = append(held, c)
mu.Unlock()
}
return nil, errors.New("still starting")
}
_, err := dialWithRetry(ctx, dial, 5*time.Second, 50*time.Millisecond, time.Second, nil)
if err == nil {
t.Fatal("expected a cancellation error")
}
if !errors.Is(err, context.Canceled) {
t.Fatalf("expected context.Canceled, got: %v", err)
}()
t.Cleanup(func() {
_ = l.Close()
<-done
mu.Lock()
defer mu.Unlock()
for _, c := range held {
_ = c.Close()
}
})
return l.Addr().String()
}
// TestNewClientAttemptEnds proves that one dial ends at its deadline: the
// server accepts the connection and never answers, so only the deadline of
// the attempt can end it. The test shortens the deadline to 200 milliseconds
// and allows two seconds: that separates the attempt's deadline from the SDK's
// own five-second GetSystemInfo timeout, which is what makes the break without
// the deadline fail.
func TestNewClientAttemptEnds(t *testing.T) {
old := dialTimeout
dialTimeout = 200 * time.Millisecond
t.Cleanup(func() { dialTimeout = old })
addr := acceptAndHold(t)
returned := make(chan error, 1)
go func() {
c, err := NewClient(context.Background(), ClientConfig{HostPort: addr})
if c != nil {
c.Close()
}
returned <- err
}()
select {
case err := <-returned:
if err == nil {
t.Fatal("NewClient connected to a server that never answers")
}
case <-time.After(2 * time.Second):
t.Fatal("NewClient outlasted the attempt deadline")
}
}
+14 -8
View File
@@ -6,6 +6,7 @@ package workflows
import (
"context"
"database/sql"
"errors"
"fmt"
"log/slog"
@@ -92,10 +93,20 @@ type WorkflowProvider interface {
Startup(ctx context.Context, c client.Client, taskQueue string, database *sql.DB, logger *slog.Logger) error
}
// ErrCannotCheck is the sentinel a WorkerPrecondition wraps when it could not
// read what it checks, as opposed to reading it and finding it refuses the
// start. errors.Is(err, ErrCannotCheck) is how the connector tells the two
// apart (design D10).
var ErrCannotCheck = errors.New("could not check the worker's precondition")
// WorkerPrecondition is implemented by a WorkflowProvider that must refuse
// to start the console while some state of the Temporal namespace would
// break the worker it is about to start. CheckBeforeWorker runs after the
// client connects and before the worker starts; an error stops the console.
// to start the worker while some state of the Temporal namespace would break
// it. CheckBeforeWorker runs after the client connects and before the worker
// starts. An error that wraps ErrCannotCheck means the state could not be
// read: the connector closes the client and connects again. Any other error is
// a refusal: the connector starts no worker, publishes no client, keeps the
// client it dialled, and reads the state again at the backoff cap until the
// precondition passes. Neither stops the console.
type WorkerPrecondition interface {
CheckBeforeWorker(ctx context.Context, c client.Client) error
}
@@ -195,11 +206,6 @@ func NewWorker(c client.Client, cfg WorkerConfig) (*Worker, error) {
desiredstate.Register(w, cfg.Database, cfg.Logger, transports,
desiredstate.TemporalPushStarter{Client: c, TaskQueue: cfg.TaskQueue})
if cfg.Logger != nil {
cfg.Logger.Info("created Temporal worker",
slog.String("taskQueue", cfg.TaskQueue))
}
return &Worker{
worker: w,
logger: cfg.Logger,
@@ -71,7 +71,9 @@ Reason: the existing guarantee is that leaving the provider's page still signs o
### D6. Temporal connects in the background
When `temporal-host` is set, `start` hands an empty holder (D7) to the router and, once the listener's port is bound (D19), starts `connectTemporal` in a goroutine. One attempt is: `client.DialContext` under a 20-second deadline, the worker preconditions (`newCheckedWorker`), the worker's construction and `worker.Start`. An attempt that fails at any step closes the client it dialled and is retried with the policy of D17, except the open-runs refusal (D10). After a successful attempt the connector publishes the client, then runs each integration's `Startup` hook, the recompute and plan-act pollers, the desired-state relay and read-back schedules, the Stripe environment check and public-details read, and the three sweep schedules, in today's order, and logs copy row 14 once.
When `temporal-host` is set, `start` hands an empty holder (D7) to the router and, once the listener's port is bound (D19), starts `connectTemporal` in a goroutine on a context that ends when serving ends, whether the context `start` was given ended or the server failed. One attempt is: `client.DialContext` under a 20-second deadline, the worker preconditions (`newCheckedWorker`), the worker's construction and `worker.Start`, and it ends there. An attempt that fails at any step closes the client it dialled and is retried with the policy of D17, except the open-runs refusal (D10). The loop logs copy row 14 when an attempt succeeds. Then the connector publishes the client and runs each integration's `Startup` hook, the recompute and plan-act pollers, the desired-state relay and read-back schedules, the Stripe environment check and public-details read, and the three sweep schedules, in today's order.
The Temporal authorization is the exception to retrying: a configuration that cannot be built (a token URL that does not parse, scopes without the other three keys, a client id of only whitespace) is refused before the listener opens, with the message `start` gave it before this change, because no attempt can fix it. Only connection failures are retried.
`temporal-connect-timeout` keeps its name and its default of 90 seconds and becomes the time after which the connector's log lines are Error instead of Warn; the first line after it passes is written at once. Retrying continues.
@@ -174,7 +176,7 @@ Reason: one policy, one test of it, and one place where the log bound is proved.
### D18. `start` exits non-zero when serving fails
When the serving half returns, `Run` logs `server failed to start` as today and calls `os.Exit(1)`, as every other failure in the command does. The serving half closes the server when its context ends and returns, which only the tests use.
When the serving half returns, `Run` logs `server failed to start` as today and calls `os.Exit(1)`, as every other failure in the command does. A failure of a step before the listener (the overrides, the boot data state, the connection binding, a Temporal authorization that cannot be built) keeps the message it had before this change, through a small error type that `Run` logs with its own message; copy row 21 is for a serving failure only. The serving half returns when the server fails at any time, including while Temporal is unreachable: the connector and the discovery loop run on a context that ends when serving ends, and the serving half returns after both have returned. It closes the server when its context ends and returns, which only the tests use.
Reason: an orchestrator that restarts only on failure would leave a process that exited 0 stopped. `RunE` would make cobra print the command's usage on an error unless silenced, and the command's failures are already logged as structured lines.
@@ -222,16 +224,16 @@ After:
3. `config.ValidateStart`, now also refusing a configured group that needs Temporal without `temporal-host`; exit 1.
4. The Stripe mode and fingerprint; exit 1 on an unknown prefix.
5. `openDB(dbRuntime)`, unchanged; exit 1. `Run` then calls the serving half, `serveConsole`, passing the mode, the fingerprint and the specs it derived; its steps follow.
6. `stripeintegration.UseKey`, then the overrides (the last `viper.Set`); exit 1.
7. `ensureBootDataState`; exit 1.
8. `bindIntegrationConnections`; exit 1.
6. `stripeintegration.UseKey`, then the overrides (the last `viper.Set`); exit 1 with the message of today's step.
7. `ensureBootDataState`; exit 1 with today's message.
8. `bindIntegrationConnections`; exit 1 with today's message.
9. The mounts.
10. An empty Temporal holder.
10. An empty Temporal holder, and, when `temporal-host` is set, the connector's inputs, read once; a Temporal authorization that cannot be built exits 1 here with today's message.
11. `server.Bind`: `auth.Setup` (the session store probe: an authentication refusal exits 1, any other probe error logs and continues; the tagged store and the session manager with its `ErrorFunc`; a `Config` with empty discovery holders; nothing is fetched from the identity provider), `NewHandler`, and the bound listener. A failure at any of these exits 1 with nothing started.
12. When `temporal-host` is set, `connectTemporal` starts in a goroutine; without it, the Warn and the backlog as today.
13. `(*Bound).Serve`: the discovery loop starts in a goroutine and the server serves. When serving stops, `start` exits 1.
12. When `temporal-host` is set, `connectTemporal` starts in a goroutine on a context that ends when serving ends; without it, the Warn and the backlog as today.
13. `(*Bound).Serve`: the discovery loop starts in a goroutine and the server serves. When serving stops for any reason, the connector and the loop are stopped and waited for, and `start` exits 1.
In the background, `connectTemporal`: the attempts of D6 until one succeeds, then `Publish`, then today's steps 12 to 15 in today's order. The discovery loop: the attempts of D3 until the verifier and the end-session endpoint are published.
In the background, `connectTemporal`: the attempts of D6 until one succeeds, then `Publish`, then today's steps 12 to 15 in today's order, which the loop's copy 14 precedes. The discovery loop: the attempts of D3 until the verifier and the end-session endpoint are published.
## The readiness holders
@@ -360,15 +362,15 @@ Removed: `Grant issued, but expiration scheduling FAILED (no scheduler); it will
| 10 | discovery loop, a document that cannot be used (Error from the first line) | `identity provider discovery document cannot be used; sign-in is unavailable`, attributes `issuer`, `retry_in`, `error` (the text of `issuerMismatchError` or of `providerEndpoints`' missing-endpoint error) |
| 11 | discovery loop, both values published (Info) | `identity provider discovery succeeded; sign-in is available`, attributes `issuer`, `attempts`, `elapsed` |
| 12 | discovery loop at publication, reused (Warn) | `the identity provider publishes no revocation_endpoint; refresh tokens will not be revoked at sign-out` |
| 13 | connector, a failed attempt (Warn, Error after `temporal-connect-timeout`), reused | `Temporal not ready yet, retrying`, attributes `attempts`, `elapsed`, `retry_in`, `error` |
| 13 | connector, a failed attempt (Warn, Error after `temporal-connect-timeout`), reused | `Temporal not ready yet, retrying`, attributes `host`, `namespace`, `attempts`, `elapsed`, `retry_in`, `error` |
| 14 | connector, connected (Info), reused | `connected to Temporal server`, attributes `host`, `namespace`, `attempts`, `elapsed`; `connected to Temporal after retry` goes |
| 15 | connector, open runs of a removed type (Error from the first line), message reused, the error text changed | message `refused to start the Temporal worker`, attributes `error`, `retry_in`; the error reads `1 FedWiki workflow run of types this release removed is still open: <types>. Let it finish on the previous release, or terminate it; the worker starts once none is open.`, and in the plural `<n> FedWiki workflow runs of types this release removed are still open: <types>. Let them finish on the previous release, or terminate them; the worker starts once none is open.` (`internal/integrations/fedwiki/fedwiki.go`, `openRunsError`; the old advice, "before starting this release", is wrong once the console is serving) |
| 16 | connector, the worker fails to build or start (Error from the first line), reused | `failed to create Temporal worker` and `failed to start Temporal worker`, attributes `error`, `retry_in` |
| 15 | connector, open runs of a removed type (Error from the first line), message reused, the error text changed | message `refused to start the Temporal worker`, attributes `host`, `namespace`, `error`, `retry_in`; the error reads `1 FedWiki workflow run of types this release removed is still open: <types>. Let it finish on the previous release, or terminate it; the worker starts once none is open.`, and in the plural `<n> FedWiki workflow runs of types this release removed are still open: <types>. Let them finish on the previous release, or terminate them; the worker starts once none is open.` (`internal/integrations/fedwiki/fedwiki.go`, `openRunsError`; the old advice, "before starting this release", is wrong once the console is serving) |
| 16 | connector, the worker fails to build or start (Error from the first line), reused | `failed to create Temporal worker` and `failed to start Temporal worker`, attributes `host`, `namespace`, `error`, `retry_in` |
| 17 | `auth.Setup`, the session store gives no answer, or an error reply that is not an authentication refusal such as `LOADING` (Error) | `session store not ready at startup; requests that need a session answer 503`, attributes `addr`, `error` |
| 18 | `start`, the session store refuses authentication (Error), reused | `server failed to start` with the error `failed to connect to Valkey: <error>`, then exit 1 |
| 19 | `ErrorFunc` (Error) | `session store request failed` for a store failure; `session could not be decoded or encoded` otherwise; attributes `method`, `path`, `error` |
| 20 | `POST /logout` while the end-session endpoint is not published (Info) | `session ended; the identity provider's session was not ended because its end-session endpoint is not known` |
| 21 | `start`, serving stopped (Error), reused | `server failed to start`, then exit 1 |
| 21 | `start`, serving stopped (Error), reused | `server failed to start`, then exit 1; a step before the listener logs its own message instead, all reused: `failed to load integration config overrides`, `invalid integration config override`, `boot data state`, `bind integration connections`, and `failed to connect to Temporal` for a Temporal authorization that cannot be built |
| 22 | Stripe webhook before Temporal connects (Warn) | `stripe webhook event recorded before Temporal connected; answering 503 so Stripe redelivers it`, attributes `event_id`, `event_type` |
### Flag help
@@ -385,8 +387,8 @@ Removed: `Grant issued, but expiration scheduling FAILED (no scheduler); it will
- **The not-ready golden** (`internal/server/testdata/routes-not-ready.golden`): the route table with the discovery holder empty and, from section 5, the Temporal holder empty, every row's anonymous and member answers; its diff from `routes.golden` is the list of routes that change.
- **The session store** (`internal/auth`, `internal/server`): an authentication refusal fails `Setup`; a refused address and a store that answers `-LOADING` do not; an `ErrorFunc` 503 for an error the tagged store returns (a network error, a `tls.RecordHeaderError`) and 500 for a decode error, including a store that returns an empty value; a request without a cookie whose commit fails gets the 503 alone, with no `Location`, no `HX-` header (`HX-Redirect`, `HX-Refresh`) and no handler body; `/static/`, `/favicon.ico`, the ask and a declared webhook path are served while the store refuses.
- **The holder and the handlers**: a handler built with an empty holder answers its "not available" answer, and the same handler answers normally once the holder is published, without being rebuilt; the shared operator harnesses and the plan-management harness publish a holder over a fake whose `ExecuteWorkflow` succeeds, so every bounded save in the existing tests keeps its result; the Stripe webhook records and answers 503, then 200 on the redelivery after publication; the Discourse webhook records `failed` with copy row 5; the bounded grant refusals write nothing and clearing an end date is saved; the structural test; a Stripe event at a finished status is not dispatched again by a second run of its activity.
- **The connector** (`cmd`): injected dial, worker and hooks; refused attempts, then a success; `Publish` before the first hook, every hook after; the removed-types refusal starts no worker and publishes nothing, then the worker starts once the fake list is empty; a list error and a worker start error are retried.
- **The boot test** (`cmd`, a database test): `serveConsole` on `127.0.0.1:0` with a Temporal address that accepts, counts and closes each connection and a switchable identity provider; within one second, the ask for a seeded placement is 200, an unknown name 404, `/static/app.css` 200, `/login` 503, a signed Stripe webhook 503 with its row recorded; `stripe.Key` holds the configured key before the first answer; after the switch, `/login` 302 within the loop's wait; a port already in use makes `serveConsole` return an error with no dial recorded in the 500 milliseconds after it returns; after a normal stop, no dial is recorded in those 500 milliseconds either; with `valkey-addr` naming a refused port, the ask answers 200 and 404 with and without a cookie, `/static/app.css` 200 and an anonymous `/` 302 to `/login`.
- **The connector** (`cmd`): injected dial, worker and hooks; refused attempts, then a success; copy 14 logged before the first hook, `Publish` before the first hook, every hook after; the removed-types refusal starts no worker and publishes nothing, then the worker starts once the fake list is empty; a list error and a worker start error are retried.
- **The boot test** (`cmd`, a database test): `serveConsole` on `127.0.0.1:0` with a Temporal address that accepts, counts and closes each connection and a switchable identity provider; within one second, the ask for a seeded placement is 200, an unknown name 404, `/static/app.css` 200, `/login` 503, a signed Stripe webhook 503 with its row recorded; `stripe.Key` holds the configured key before the first answer; after the switch, `/login` 302 within the loop's wait; a port already in use makes `serveConsole` return an error with no dial recorded in the 500 milliseconds after it returns; after a normal stop, no dial is recorded in those 500 milliseconds either; a server that fails while Temporal is unreachable makes `serveConsole` return that error within a few seconds, with no dial in those 500 milliseconds; a dial held open 200 milliseconds past the context's end is waited for; a step before the listener returns its own message, and a Temporal authorization that cannot be built returns before `listen` is called; with `valkey-addr` naming a refused port, the ask answers 200 and 404 with and without a cookie, `/static/app.css` 200 and an anonymous `/` 302 to `/login`.
- **Races**: `go test -race -p 1` after `test/reset-test-db.sh` on every touched package (the packages share one test database, and parallel packages make count-based tests claim each other's rows), with requests in flight across the discovery publication and the Temporal publication.
## Risks / Trade-offs
@@ -147,37 +147,43 @@ Sections 1 to 7 are one commit each, and their items are that commit's parts; se
## 6. Temporal connects in the background, and start serves before it
- [ ] 6.1 `internal/workflows/client.go` (design D17): delete `dialWithRetry`, its constants and `ClientConfig.ConnectTimeout`. `NewClient(ctx, cfg)` becomes one `client.DialContext` under a 20-second deadline, with the OAuth headers provider and the logger as today. `client_test.go`'s four `TestDialWithRetry_*` tests are deleted with the function; a test of the single attempt's deadline replaces them (a listener that accepts and never answers; the dial returns within the shortened deadline the test sets through a package variable).
- [ ] 6.2 `internal/workflows`: `ErrCannotCheck`, a sentinel a `WorkerPrecondition` wraps when it could not read what it checks. FedWiki's `listRunsError` reports `errors.Is(err, workflows.ErrCannotCheck)`; `openRunsError` does not. The doc comment of `WorkerPrecondition` says what the connector does with each (design D10). `openRunsError.Error()` (`internal/integrations/fedwiki/fedwiki.go`) takes copy 15's text in both forms: `... is still open: <types>. Let it finish on the previous release, or terminate it; the worker starts once none is open.` and the plural; its existing test, if it compares the sentence, compares the new one.
- [ ] 6.3 `cmd/temporal_connect.go` (new, design D6, D7, D10): `connectTemporal(ctx, t temporalConnect)`, where `temporalConnect` carries the client configuration, the worker configuration, the holder, the logger, the escalation time (`temporal-connect-timeout`, 90 seconds when zero), the policy's first wait and cap (two package variables, 0.5 and 15 seconds, which the boot test of item 6.8 shortens to 50 milliseconds), and the injected `dial`, `newWorker` and `afterConnect` functions. It runs `retryloop.Run` with D17's numbers and copies 13, 14, 15 and 16, and the attempt logs nothing itself: every failure is returned as a `retryloop.Failure`. One attempt: dial (a plain error, copy 13's message); `newCheckedWorker`; on a refusal that is not `ErrCannotCheck`, keep the client and return a `Failure` with copy 15's message and level Error, marked `AtCap`, so that the next attempt reads the list again on the same client; on `ErrCannotCheck`, a build error or a `worker.Start` error, close the client and return the error (a `Failure` with copy 16's message and level Error for the worker's two failures). On success, `Publish`, then `afterConnect(ctx, c, worker)`. `connectTemporal` and the `temporal-host` test that moves into it read the host trimmed, so a host of only whitespace counts as unset (a row of item 6.8 covers it). `afterConnect` in production is today's block of `cmd/start.go` from the `Startup` hooks to the webhook partition schedule, moved unchanged into `startTemporalWork`.
- [ ] 6.4 `cmd/start.go` (design D1, D13, D18, D19): `Run` keeps steps 1 to 5 of the design's boot sequence (the key derivations without `UseKey`), then calls `serveConsole(ctx, logger, boot, database, integs, listenTCP(":"+port))`, where `boot` carries the Stripe mode, the key's fingerprint and the integrations' specs `Run` derived, logs copy 21 when it returns and exits 1. `serveConsole` calls `stripeintegration.UseKey` first, then runs the overrides, the boot data state, the connection binding, the mounts and the holder, then `server.Bind(serverConfig, listen)`, then `go connectTemporal(...)` when `temporal-host` is set (the Warn and the backlog otherwise, as today), then the bound server's `Serve(ctx)`. When `ctx` ends it waits for `connectTemporal` and `Discover` to return (a `sync.WaitGroup`) before it returns. A comment above the first `go` statement says that every configuration write is above it and that the goroutines take their keys once (design D13). The words "before any goroutine starts" are no longer true once `UseKey` runs inside `serveConsole`, after `openDB` has started database/sql's connection opener, so they become "before the listener, and before any goroutine that reads the key" in the `UseKey` doc comment (`stripe.go:288-291`) and in the comment above the call in `cmd/start.go`. The block comments of the old Temporal section and `startStripeEnvironmentCheck`'s "boots at the same speed" paragraph are rewritten to the new order.
- [ ] 6.5 `internal/server/server.go` (design D19): `Start(ctx, cfg)` becomes two steps. `Bind(cfg, listen func() (net.Listener, error))` runs `auth.Setup`, `NewHandler` and `listen` and returns the bound server, or the first error having started nothing. `(*Bound).Serve(ctx)` starts the discovery loop (`go authConfig.Discover(ctx)`) and serves on the listener; when `ctx` ends it calls `Shutdown` with a bounded context (five seconds), waits for `Discover` to return, and returns. Their doc comments say that nothing before `listen` waits on Temporal or the identity provider and that a failure of `Bind` starts no background work.
- [ ] 6.6 `--temporal-connect-timeout`'s help is copy 23; the viper default stays 90 seconds. The log lines of copy 25 change as that row says (`internal/integrations/fedwiki/web/api.go`, `customdomain.go`, `site_actions.go`, `internal/server/member_domains.go`, `operator_entitlement_recompute.go`, `operator_plan_acts.go`), each at its current level; a test that asserts one of the old texts asserts the new one.
- [ ] 6.7 `internal/integrations/stripe/workflows/webhook.go` (design F34): `ProcessWebhookEvent`'s first `UPDATE` adds `AND status IN ('received','processing','failed')` and the activity checks the rows affected; when none is, it returns nil without dispatching, with a comment that says the sweep and a redelivery can start a completed event's id again. The guard is in the activity because the sweep's status read and the workflow start are two steps, and the workflow id's reuse policy lets a completed id run again. Test (`TestProcessWebhookEventSkipsAFinishedEvent`, a database test, one row per status through a named function): a row at `completed`, `skipped`, `refused` and `dead_letter`, with an event type whose handler fails on the empty payload, is returned nil, unchanged and not dispatched; a row at `received`, one at `processing` (a worker that died mid-attempt, which Temporal then retries) and one at `failed` are each dispatched: the activity returns the dispatch error, and the row reads `failed` with `retry_count` 1 and the error, because the handler fails on the payload and the row does not rest at `processing`.
- [ ] 6.8 Tests. `TestConnectTemporal` (`cmd/temporal_connect_test.go`), injected fakes and a fake clock, one row per case through a named function: three refused dials, then a dial that succeeds: the holder is published, `afterConnect` runs once after `Publish` (the fakes record the order of `Start`, `Publish` and `afterConnect`), and copy 14 carries `attempts` 4; a precondition that refuses twice and then passes: one dial, no worker until it passes, the holder empty while it refuses, copy 15 at Error, the second reading a cap's wait after the first; a precondition that returns `ErrCannotCheck`: the client is closed and the next attempt dials again after the backoff's first wait; `worker.Start` that fails once: the client is closed, the next attempt dials again, copy 16; `worker.Start` that fails on every attempt for an hour of fake time: at most fourteen lines carry copy 16, all at Error, because the attempt logs nothing outside the decay; a context that ends during a wait: `connectTemporal` returns and publishes nothing; failures past the escalation time: Error lines. `TestServeConsoleBeforeDependencies` (`cmd/serve_db_test.go`, a database test that also needs the stack's session store), with `temporal-host` naming a listener that accepts, counts and closes each connection (a closed port records nothing, so a count that stays zero could not fail), the connector's first wait and cap shortened to 50 milliseconds through the package variables of item 6.3, `oidc-idp-issuer-url` naming an `httptest` server that answers 503 until the test switches it to a discovery document whose issuer is its own URL, `stripe-api-key` `sk_test_serve<suffix>` with a webhook secret, the SDK key empty beforehand, and a servable placement seeded the way the domains registry's own database tests seed one, under the test's suffix. The test clears and reads the SDK key through `stripetest.ClearSDKKey(t)` (which restores it in `Cleanup`) and `stripetest.SDKKey()`, both new in `internal/stripetest`, because `.golangci.yml` lints tests and forbids `stripe.Key` outside `internal/(integrations/stripe|stripetest)/`. `serveConsole` runs in a goroutine with a `listen` that opens `127.0.0.1:0` and sends the address on a channel, which is the happens-before edge the key read needs; within one second of the address being received: the ask for the seeded name answers 200 and for an unknown name 404; `/static/app.css` 200; `/login` 503 with `Retry-After: 5`; a signed Stripe event (`webhook.GenerateTestSignedPayload`) 503 with its row `received`; and `stripetest.SDKKey()`, read after the first answer, is the configured key. After the switch, `/login` answers 302 to the fake's authorization endpoint within five seconds. Cancelling the context makes `serveConsole` return, and it returns only after `connectTemporal` and `Discover` have: the listener's count and the identity provider fake's count are equal when `serveConsole` returns and 500 milliseconds later, so a loop that outlived it and dialled again within its shortened wait is seen. A second row runs `serveConsole` with a `listen` that returns the error of binding an address already in use, and asserts that `serveConsole` returns that error and that, 500 milliseconds after it returned, the counting listener recorded no Temporal dial and the identity provider fake no discovery request. A third row sets `valkey-addr` to a refused port and asserts, with and without a session cookie, that the ask answers 200 and 404, `/static/app.css` 200, and an anonymous `/` 302 to `/login`. `TestServeConsoleUnderLoad` runs the first row with twenty goroutines requesting `/login`, the ask and a signed-in page across the switch, under `-race` (item 6.11).
- [ ] 6.9 Start this worktree's application on the stack (`go run . start` with `test/.env` loaded) and check that it serves; stop the stack's Temporal container and start the application again: the ask and `/login` answer while the boot log shows copy 13 at Warn, then Error after ninety seconds; start the container again: copy 14 appears and the process has not restarted. Point `oidc-idp-issuer-url` at an address that refuses and check that `/login` answers the styled 503 page and that the ask answers.
- [ ] 6.10 Breaks:
- [x] 6.1 `internal/workflows/client.go` (design D17): delete `dialWithRetry`, its constants and `ClientConfig.ConnectTimeout`. `NewClient(ctx, cfg)` becomes one `client.DialContext` under a 20-second deadline, with the OAuth headers provider and the logger as today. `client_test.go`'s four `TestDialWithRetry_*` tests are deleted with the function; a test of the single attempt's deadline replaces them (a listener that accepts and never answers; the dial returns within the shortened deadline the test sets through a package variable).
- [x] 6.2 `internal/workflows`: `ErrCannotCheck`, a sentinel a `WorkerPrecondition` wraps when it could not read what it checks. FedWiki's `listRunsError` reports `errors.Is(err, workflows.ErrCannotCheck)`; `openRunsError` does not. The doc comment of `WorkerPrecondition` says what the connector does with each (design D10). `openRunsError.Error()` (`internal/integrations/fedwiki/fedwiki.go`) takes copy 15's text in both forms: `... is still open: <types>. Let it finish on the previous release, or terminate it; the worker starts once none is open.` and the plural; its existing test, if it compares the sentence, compares the new one.
- [x] 6.3 `cmd/temporal_connect.go` (new, design D6, D7, D10): `connectTemporal(ctx, t temporalConnect)`, where `temporalConnect` carries the client configuration, the worker configuration, the holder, the logger, the escalation time (`temporal-connect-timeout`, 90 seconds when zero), the policy's first wait and cap (two package variables, 0.5 and 15 seconds, which the boot test of item 6.8 shortens to 50 milliseconds), and the injected `dial`, `newWorker` and `afterConnect` functions. It runs `retryloop.Run` with D17's numbers and copies 13, 14, 15 and 16, and the attempt logs nothing itself: every failure is returned as a `retryloop.Failure`. One attempt: dial (a plain error, copy 13's message); `newCheckedWorker`; on a refusal that is not `ErrCannotCheck`, keep the client and return a `Failure` with copy 15's message and level Error, marked `AtCap`, so that the next attempt reads the list again on the same client; on `ErrCannotCheck`, a build error or a `worker.Start` error, close the client and return the error (a `Failure` with copy 16's message and level Error for the worker's two failures). The attempt ends at `worker.Start`, so the loop logs copy 14 when it succeeds; `connectTemporal` then calls `Publish` and `afterConnect(ctx, c, worker)`, once the loop has returned nil. The failure lines carry `host` and `namespace`, the policy's base attributes (copies 13, 15 and 16 list them). `workflows.NewWorker` no longer logs `created Temporal worker` on every call, which would write outside the decay when `worker.Start` fails on every attempt. `connectTemporal` and the `temporal-host` test that moves into it read the host trimmed, so a host of only whitespace counts as unset (a row of item 6.8 covers it). `afterConnect` in production is today's block of `cmd/start.go` from the `Startup` hooks to the webhook partition schedule, moved unchanged into `startTemporalWork`.
- [x] 6.4 `cmd/start.go` (design D1, D13, D18, D19): `Run` keeps steps 1 to 5 of the design's boot sequence (the key derivations without `UseKey`), then calls `serveConsole(ctx, logger, boot, database, integs, listenTCP(":"+port))`, where `boot` carries the Stripe mode, the key's fingerprint and the integrations' specs `Run` derived, logs what it returned through `reportServeFailure` and exits 1: copy 21 for an error from serving, and, for a `*startStepError`, the message of the step as it was before this change. `serveConsole` calls `stripeintegration.UseKey` first, then runs the overrides, the boot data state, the connection binding, the mounts and the holder, then `server.Bind(serverConfig, listen)`, then `go connectTemporal(...)` when `temporal-host` is set (the Warn and the backlog otherwise, as today), then the bound server's `Serve(ctx)`. The connector runs on a context derived from `ctx` that `serveConsole` cancels when `Serve` returns, for any reason, so a server that fails while Temporal is unreachable still returns; `serveConsole` then waits for the connector (a `sync.WaitGroup`) and for `Discover`, which `Serve` waits for, before it returns. Each step before `Bind` returns a `*startStepError` carrying the message the step logged before this change (`failed to load integration config overrides`, `invalid integration config override`, `boot data state`, `bind integration connections`), and a Temporal authorization that `workflows.NewOAuthTokenProvider` refuses is built once in `newTemporalConnect` and returned the same way, with `failed to connect to Temporal`, before `Bind`; only connection failures are retried. A comment above the first `go` statement says that every configuration write is above it and that the goroutines take their keys once (design D13). The words "before any goroutine starts" are no longer true once `UseKey` runs inside `serveConsole`, after `openDB` has started database/sql's connection opener, so they become "before the listener, and before any goroutine that reads the key" in the `UseKey` doc comment (`stripe.go:288-291`) and in the comment above the call in `cmd/start.go`. The block comments of the old Temporal section and `startStripeEnvironmentCheck`'s "boots at the same speed" paragraph are rewritten to the new order.
- [x] 6.5 `internal/server/server.go` (design D19): `Start(ctx, cfg)` becomes two steps. `Bind(cfg, listen func() (net.Listener, error))` runs `auth.Setup`, `NewHandler` and `listen` and returns the bound server, or the first error having started nothing. `(*Bound).Serve(ctx)` starts the discovery loop (`go authConfig.Discover(ctx)`) and serves on the listener; when `ctx` ends it calls `Shutdown` with a bounded context (five seconds), waits for `Discover` to return, and returns. Their doc comments say that nothing before `listen` waits on Temporal or the identity provider and that a failure of `Bind` starts no background work.
- [x] 6.6 `--temporal-connect-timeout`'s help is copy 23; the viper default stays 90 seconds. The log lines of copy 25 change as that row says (`internal/integrations/fedwiki/web/api.go`, `customdomain.go`, `site_actions.go`, `internal/server/member_domains.go`, `operator_entitlement_recompute.go`, `operator_plan_acts.go`), each at its current level; a test that asserts one of the old texts asserts the new one.
- [x] 6.7 `internal/integrations/stripe/workflows/webhook.go` (design F34): `ProcessWebhookEvent`'s first `UPDATE` adds `AND status IN ('received','processing','failed')` and the activity checks the rows affected; when none is, it returns nil without dispatching, with a comment that says the sweep and a redelivery can start a completed event's id again. The guard is in the activity because the sweep's status read and the workflow start are two steps, and the workflow id's reuse policy lets a completed id run again. Test (`TestProcessWebhookEventSkipsAFinishedEvent`, a database test, one row per status through a named function): a row at `completed`, `skipped`, `refused` and `dead_letter`, with an event type whose handler fails on the empty payload, is returned nil, unchanged and not dispatched; a row at `received`, one at `processing` (a worker that died mid-attempt, which Temporal then retries) and one at `failed` are each dispatched: the activity returns the dispatch error, and the row reads `failed` with `retry_count` 1 and the error, because the handler fails on the payload and the row does not rest at `processing`.
- [x] 6.8 Tests. `TestConnectTemporal` (`cmd/temporal_connect_test.go`), injected fakes and a fake clock, one row per case through a named function: three refused dials, then a dial that succeeds: the holder is published, `afterConnect` runs once after `Publish` (the fakes record the order of `Start`, `Publish` and `afterConnect`), and copy 14 carries `attempts` 4; a precondition that refuses twice and then passes: one dial, no worker until it passes, the holder empty while it refuses, copy 15 at Error, the second reading a cap's wait after the first; a precondition that returns `ErrCannotCheck`: the client is closed and the next attempt dials again after the backoff's first wait; `worker.Start` that fails once: the client is closed, the next attempt dials again, copy 16; `worker.Start` that fails on every attempt for an hour of fake time: at most fourteen lines carry copy 16, all at Error, because the attempt logs nothing outside the decay; a context that ends during a wait: `connectTemporal` returns and publishes nothing; failures past the escalation time: Error lines. `TestServeConsoleBeforeDependencies` (`cmd/serve_db_test.go`, a database test that also needs the stack's session store), with `temporal-host` naming a listener that accepts, counts and closes each connection (a closed port records nothing, so a count that stays zero could not fail), the connector's first wait and cap shortened to 50 milliseconds through the package variables of item 6.3, `oidc-idp-issuer-url` naming an `httptest` server that answers 503 until the test switches it to a discovery document whose issuer is its own URL, `stripe-api-key` `sk_test_serve<suffix>` with a webhook secret, the SDK key empty beforehand, and a servable placement seeded the way the domains registry's own database tests seed one, under the test's suffix. The test clears and reads the SDK key through `stripetest.ClearSDKKey(t)` (which restores it in `Cleanup`) and `stripetest.SDKKey()`, both new in `internal/stripetest`, because `.golangci.yml` lints tests and forbids `stripe.Key` outside `internal/(integrations/stripe|stripetest)/`. `serveConsole` runs in a goroutine with a `listen` that opens `127.0.0.1:0` and sends the address on a channel, which is the happens-before edge the key read needs; within one second of the address being received: the ask for the seeded name answers 200 and for an unknown name 404; `/static/app.css` 200; `/login` 503 with `Retry-After: 5`; a signed Stripe event (`webhook.GenerateTestSignedPayload`) 503 with its row `received`; and `stripetest.SDKKey()`, read after the first answer, is the configured key. After the switch, `/login` answers 302 to the fake's authorization endpoint within five seconds. Cancelling the context makes `serveConsole` return: the listener's count is equal when `serveConsole` returns and 500 milliseconds later, so a connector that outlived it and dialled again within its shortened wait is seen. No test shows that `serveConsole` waits for `Discover`: it returns on its own once the provider answers, and its HTTP client returns as soon as its context ends, so no black-box test in `cmd` can make it outlast the context (a test would belong in `internal/server`, over a `Bound` whose discovery is a fake, which `auth` does not expose). `TestConnectTemporal`'s first row also asserts that copy 14 has been logged when the first `afterConnect` runs. A second row runs `serveConsole` with a `listen` that returns the error of binding an address already in use, and asserts that `serveConsole` returns that error and that, 500 milliseconds after it returned, the counting listener recorded no Temporal dial and the identity provider fake no discovery request. A third row sets `valkey-addr` to a refused port and asserts, with and without a session cookie, that the ask answers 200 and 404, `/static/app.css` 200, and an anonymous `/` 302 to `/login`. A fourth row replaces the dial with one that outlasts its context by 200 milliseconds (a wall-clock sleep, the least that makes an early return observable) and asserts that `serveConsole` returns only after that dial has. A fifth row closes the listener once the connector is dialling the counting address, which makes the server fail, and asserts that `serveConsole` returns that error within five seconds and that no dial follows it. `TestServeConsoleStepFailures` (a database test): with the database closed, `serveConsole` returns a `*startStepError` with `failed to load integration config overrides`, and `reportServeFailure` logs one Error line with that message; with `temporal-oauth-scopes` set and the other three keys unset, it returns one with `failed to connect to Temporal` without having called `listen`. `TestReportServeFailure` asserts that an error from serving, and no error, log one Error line, copy 21. `TestServeConsoleUnderLoad` runs the first row with twenty goroutines requesting `/login`, the ask and a signed-in page across the switch, under `-race` (item 6.11).
- [x] 6.9 Start this worktree's application on the stack (`go run . start` with `test/.env` loaded) and check that it serves; stop the stack's Temporal container and start the application again: the ask and `/login` answer while the boot log shows copy 13 at Warn, then Error after ninety seconds; start the container again: copy 14 appears and the process has not restarted. Point `oidc-idp-issuer-url` at an address that refuses and check that `/login` answers the styled 503 page and that the ask answers.
- [x] 6.10 Breaks:
- `serveConsole` without `UseKey` (the call removed; section 1's record): caught by `TestServeConsoleBeforeDependencies` (the SDK key is empty);
- `UseKey` after the bound server's `Serve` is called: caught by the same assertion;
- `Serve` without `go authConfig.Discover(ctx)` (section 3's record): caught by the switch assertion (`/login` stays 503);
- `Serve` fetching the discovery document before `listen` (a synchronous first attempt that waits for success): caught by the one-second assertions;
- `Serve` fetching the discovery document between `listen` and the server's start (a synchronous first attempt that waits for success; before `listen` the row would fail at "waited 30s for serveConsole to listen" instead): caught by the one-second assertions;
- `serveConsole` waiting for `connectTemporal` before the bound server's `Serve`: caught by the one-second assertions;
- `connectTemporal` started before `listen` (before `server.Bind` returns): caught by the in-use row (a dial recorded);
- `serveConsole` returning without waiting for its goroutines: caught by the first row (a dial recorded after `serveConsole` returned) and, as a command record (`-race`), by `TestServeConsoleUnderLoad`;
- `connectTemporal` started before `server.Bind` returns (started when the bind fails): caught by the in-use row (a dial recorded);
- `serveConsole` returning without waiting for the connector: caught by the fourth row (the held dial has not returned) and, as a command record (`-race`) over that row, by the data race between the connector's write of its result and `stop`'s read, which the row's wait for the held dial makes certain; `TestServeConsoleUnderLoad` showed the race only when the timing allowed it, so the record does not use it;
- `connectTemporal` run under `serveConsole`'s own context in place of the one `Serve`'s end cancels: caught by the fifth row (`serveConsole did not return after serving failed`);
- `Serve` without `discovery.Wait()`: caught by no break, for the reason 6.8 gives, and so not listed;
- `skipSession` without the ask's path, over the real handler: caught by the third row (the ask answers 503 with a cookie);
- `connectTemporal` publishes after `afterConnect`: caught by `TestConnectTemporal`'s order row;
- `connectTemporal` publishes before `worker.Start`: caught by the same row;
- `connectTemporal` publishes before `worker.Start` (the record spans the publication, the attempt's head and `worker.Start`, and moves the publication before the start): caught by the same row;
- the attempt runs `afterConnect` before the loop logs copy 14: caught by the same test's first row (no copy 14 line when the hooks first ran);
- `connectTemporal` returns after the escalation time (the old budget): caught by the escalation row;
- the open-runs refusal closes the client: caught by the refusal row (a second dial);
- `ErrCannotCheck` treated as a refusal: caught by its row (no second dial);
- `worker.Start`'s failure left without closing the client: caught by its row;
- `listRunsError` without its `Is` method: caught by `TestConnectTemporal`'s `ErrCannotCheck` row run against FedWiki's real error type through the fake list;
- `worker.Start`'s failure logged from inside the attempt, outside the decay: caught by `TestConnectTemporal`'s hour-of-failures row (more than fourteen lines);
- `reportServeFailure` logging copy 21 for a step failure: caught by `TestServeConsoleStepFailures`' overrides row;
- the overrides failure returned as a plain error: caught by the same row (not a step failure);
- a Temporal authorization that cannot be built left unrefused: caught by `TestServeConsoleStepFailures`' authorization row (`listen` was called);
- `ProcessWebhookEvent` without its status guard: caught by `TestProcessWebhookEventSkipsAFinishedEvent`'s finished rows (a row read `processing` and the event was dispatched);
- the guard without `failed`: caught by the same test's `failed` row (the retry is not dispatched);
- the guard without `processing`: caught by the same test's `processing` row (nil returned, not dispatched);
- `NewClient` without the deadline: caught by the single attempt's deadline test.
- [ ] 6.11 After `test/reset-test-db.sh`, `go test -race -p 1 -count=1 ./cmd/ ./internal/auth/ ./internal/server/ ./internal/retryloop/ ./internal/workflows/... ./internal/integrations/stripe/... ./internal/integrations/discourse/... ./internal/integrations/fedwiki/...` passes.
- [x] 6.11 After `test/reset-test-db.sh`, `go test -race -p 1 -count=1 ./cmd/ ./internal/auth/ ./internal/server/ ./internal/retryloop/ ./internal/workflows/... ./internal/integrations/stripe/... ./internal/integrations/discourse/... ./internal/integrations/fedwiki/...` passes.
## 7. Docs
@@ -187,7 +193,12 @@ Sections 1 to 7 are one commit each, and their items are that commit's parts; se
- [ ] 7.4 `docs/identity-provider-setup.md` and `internal/auth/README.md`: "refuses to start" and "fails startup" become what the code does: the console serves and keeps sign-in unavailable, logging at Error, until the provider publishes the endpoint. The logout section says what sign-out does without an end-session endpoint, and the provider setup says that a document with the configured issuer and no `end_session_endpoint` leaves sign-in unavailable but does not stop session re-derivation.
- [ ] 7.5 `docs/temporal-authorization-setup.md`: one paragraph on the token URL: an internal address keeps the Temporal connection off the TLS proxy; the console no longer needs it before it serves.
- [ ] 7.6 `docs/stripe.md` and `docs/discourse-setup.md`: one sentence each that `temporal-host` is required, and why. `docs/building-an-integration.md` section 6 gains a `NeedsTemporal` bullet: it goes on a key of the RequiredGroup of an integration that receives events processed only on Temporal, and boot refuses the configured group without `temporal-host`.
- [ ] 7.7 No code changes; `make test` passes.
- [ ] 7.7 Four texts that still describe the boot section 6 replaced:
1. `internal/integrations/fedwiki/workflows/README.md` (around line 210): "the process exits with status 1 when any is open or the list cannot be read. Let an open run finish on the previous release, or terminate it, before starting this release." becomes "...lists the running executions of those types. While any is open the worker does not start: the console serves, every Temporal-backed action answers that it is not available, and the list is read again every fifteen seconds. A list that cannot be read is retried on a new connection. Let an open run finish on the previous release, or terminate it; the worker starts once none is open."
2. `internal/integrations/fedwiki/fedwiki.go` (around line 452): "listRunsError is the refusal to start when the list cannot be read" becomes "listRunsError is the precondition's failure to read the list: it reports workflows.ErrCannotCheck, so the connector closes the client and connects again."
3. `internal/integrations/fedwiki/fedwiki.go` (around lines 469 and 470): "A list that cannot be read refuses too" becomes "A list that cannot be read is returned as workflows.ErrCannotCheck, since it cannot say that none is open."
4. `internal/integrations/stripe/stripe.go` (around line 216): `Startup`'s doc says the key is set "before any goroutine runs"; it becomes "...through UseKey, before the listener and before any goroutine that reads the key."
- [ ] 7.8 No code changes other than the comments of 7.7; `make test` passes.
## 8. Landing