member-console start handles no stop signal, so a stop cuts in-flight requests and never stops the Temporal worker #260

New Issue

2026-10-24T06:51:53Z

cgalo5758 commented

2026-10-24 06:51:53 +00:00

What happens

start installs no signal handler: nothing in cmd/ or internal/ calls signal.Notify or signal.NotifyContext. A SIGTERM or SIGINT therefore ends the process at once, by Go's default action. A container stop or a redeploy:

cuts every request in flight, whatever it was writing or answering;
never stops the Temporal worker, so each activity it was running is retried only after its timeout (one minute for a Stripe webhook event);
never closes the database pool or the session store's connection pool.

The shutdown path exists but nothing reaches it. server.Bound.Serve (internal/server/server.go) calls http.Server.Shutdown with a five-second bound when its context ends, and serveConsole (cmd/start.go) then stops the Temporal connector and calls temporalConnection.stop, which stops the worker and closes the client. But startCmd.Run passes context.Background(), which never ends. After serveConsole returns, Run also logs "server failed to start" and calls os.Exit(1) whatever the reason, and os.Exit skips the deferred closeDB.

What should happen

On SIGTERM or SIGINT, start stops accepting connections, lets in-flight requests finish within a bound, stops the Temporal worker, closes the database and session store pools, and exits with status 0 without logging a start failure. A second signal, or the bound running out, ends the process at once.

Where

cmd/start.go (startCmd.Run, serveConsole)

Steps

Run member-console start against the test stack.
While a request is being answered, or while the Temporal UI shows an activity running on the console's worker, send the process SIGTERM (kill -TERM <pid>, or docker stop for a container).
The process exits immediately: the request's connection is closed with no answer, and the activity stays running in the Temporal UI until its timeout.

Why it matters

Every deploy or restart cuts the requests in flight and delays the background work that was running by its activity timeout.

### What happens `start` installs no signal handler: nothing in `cmd/` or `internal/` calls `signal.Notify` or `signal.NotifyContext`. A SIGTERM or SIGINT therefore ends the process at once, by Go's default action. A container stop or a redeploy: - cuts every request in flight, whatever it was writing or answering; - never stops the Temporal worker, so each activity it was running is retried only after its timeout (one minute for a Stripe webhook event); - never closes the database pool or the session store's connection pool. The shutdown path exists but nothing reaches it. `server.Bound.Serve` (`internal/server/server.go`) calls `http.Server.Shutdown` with a five-second bound when its context ends, and `serveConsole` (`cmd/start.go`) then stops the Temporal connector and calls `temporalConnection.stop`, which stops the worker and closes the client. But `startCmd.Run` passes `context.Background()`, which never ends. After `serveConsole` returns, `Run` also logs "server failed to start" and calls `os.Exit(1)` whatever the reason, and `os.Exit` skips the deferred `closeDB`. ### What should happen On SIGTERM or SIGINT, `start` stops accepting connections, lets in-flight requests finish within a bound, stops the Temporal worker, closes the database and session store pools, and exits with status 0 without logging a start failure. A second signal, or the bound running out, ends the process at once. ### Where `cmd/start.go` (`startCmd.Run`, `serveConsole`) ### Steps 1. Run `member-console start` against the test stack. 2. While a request is being answered, or while the Temporal UI shows an activity running on the console's worker, send the process SIGTERM (`kill -TERM <pid>`, or `docker stop` for a container). 3. The process exits immediately: the request's connection is closed with no answer, and the activity stays running in the Temporal UI until its timeout. ### Why it matters Every deploy or restart cuts the requests in flight and delays the background work that was running by its activity timeout.

cgalo5758 added the

kind

bug

area/ops labels 2026-10-24 06:51:53 +00:00

cgalo5758 commented

2026-10-30 13:25:57 +00:00

When the stop path waits for requests in flight, the bound decides which of them keep their answer: the console lets a handler run for 32 seconds and keeps writing its answer, or its 503, for 8 more. A bound under 40 seconds cuts some requests the console would otherwise answer. A bound of 40 seconds or more needs the orchestrator's stop grace period set above it; Docker's default is 10 seconds, and the deployment guide doesn't set one.

cgalo5758 commented

2026-10-31 00:22:18 +00:00

A member's plan change can keep running after the member's request has been answered: once its first Stripe write is sent, its later Stripe calls and its database writes run to deadlines of their own, up to about 220 seconds in all. The request's timeout handler has already answered at 32 seconds, so a stop path that waits only for the server's tracked requests does not wait for these changes, and closing the database pool then fails the write that records the change, after its write at Stripe. The stop path therefore has to wait for plan changes in flight (for instance a counter that each change holds until its record returns, bounded by about 220 seconds), or the orchestrator's grace period has to cover them; otherwise a stop can cut a change between a Stripe write and the console's record, and each deploy that catches a slow change can leave a change made at Stripe and unrecorded. Running the change's writes as a durable job, which a stop does not cut, was considered and not chosen for #295, because the member's answer needs the charge's outcome at once.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: wiki-cafe/member-console#260

member-console start handles no stop signal, so a stop cuts in-flight requests and never stops the Temporal worker #260

What happens

What should happen

Where

Steps

Why it matters

`member-console start` handles no stop signal, so a stop cuts in-flight requests and never stops the Temporal worker #260