The console serves nothing, not even the certificate ask, until Temporal and the identity provider are reachable #141

New Issue

2026-09-25T08:20:00Z

cgalo5758 commented

2026-09-25 08:20:00 +00:00

What happens

At boot the console opens its HTTP port only after two outside services answer. It connects to Temporal, first fetching an OAuth token from the identity provider when Temporal authorization is configured, and it reads the identity provider's discovery document. Until both succeed, no route is served. If Temporal is not reached within the connect budget, the process exits and the orchestrator restarts it into the same wait.

That includes GET /domains/ask, the endpoint a TLS-terminating proxy calls before it issues or loads an on-demand certificate (the certificate ask). The ask needs only the database.

When the identity provider's host is itself issued on demand by the same proxy, a proxy restart with an empty certificate cache deadlocks. The proxy will not present a certificate for the identity provider until the ask answers, and the console will not answer the ask until it has reached the identity provider through that proxy. Every on-demand host stays without TLS until an operator points the proxy's ask somewhere else.

What should happen

Routes whose only dependency is the database, /domains/ask first among them, are served as soon as migrations have run, whatever the state of Temporal and the identity provider. Routes that need either answer 503 until it is reachable, and the console keeps retrying in the background instead of exiting.

Where

cmd/start.go (the Temporal connection before server.Start), internal/auth/auth.go (oidc.NewProvider, called from server.Start before the listener opens), internal/server/domains.go

Steps

Put the identity provider behind a proxy whose on-demand TLS ask points at the console's /domains/ask, with the identity provider's host issued on demand.
Restart the proxy and the console together; a Docker daemon restart does this.
The proxy refuses TLS for every on-demand host. The console logs Temporal not ready yet, retrying until it exits, and restarts into the same state.

Why it matters

A routine host restart can take every on-demand site offline, and the console, which exists to authorize those certificates, is what keeps them offline. Until this is fixed, a deployment avoids it by giving the identity provider's host an ordinary managed certificate, which the proxy loads at startup without asking.

### What happens At boot the console opens its HTTP port only after two outside services answer. It connects to Temporal, first fetching an OAuth token from the identity provider when Temporal authorization is configured, and it reads the identity provider's discovery document. Until both succeed, no route is served. If Temporal is not reached within the connect budget, the process exits and the orchestrator restarts it into the same wait. That includes `GET /domains/ask`, the endpoint a TLS-terminating proxy calls before it issues or loads an on-demand certificate (the certificate ask). The ask needs only the database. When the identity provider's host is itself issued on demand by the same proxy, a proxy restart with an empty certificate cache deadlocks. The proxy will not present a certificate for the identity provider until the ask answers, and the console will not answer the ask until it has reached the identity provider through that proxy. Every on-demand host stays without TLS until an operator points the proxy's ask somewhere else. ### What should happen Routes whose only dependency is the database, `/domains/ask` first among them, are served as soon as migrations have run, whatever the state of Temporal and the identity provider. Routes that need either answer 503 until it is reachable, and the console keeps retrying in the background instead of exiting. ### Where `cmd/start.go` (the Temporal connection before `server.Start`), `internal/auth/auth.go` (`oidc.NewProvider`, called from `server.Start` before the listener opens), `internal/server/domains.go` ### Steps 1. Put the identity provider behind a proxy whose on-demand TLS ask points at the console's `/domains/ask`, with the identity provider's host issued on demand. 2. Restart the proxy and the console together; a Docker daemon restart does this. 3. The proxy refuses TLS for every on-demand host. The console logs `Temporal not ready yet, retrying` until it exits, and restarts into the same state. ### Why it matters A routine host restart can take every on-demand site offline, and the console, which exists to authorize those certificates, is what keeps them offline. Until this is fixed, a deployment avoids it by giving the identity provider's host an ordinary managed certificate, which the proxy loads at startup without asking.

cgalo5758 added the

kind

bug

area/domains area/ops labels 2026-09-25 08:20:00 +00:00

cgalo5758 referenced this issue

2026-10-09 16:03:03 +00:00

`member-console start` exits with status 0 when the web server fails to start #209

cgalo5758 commented

2026-10-16 03:00:05 +00:00

The fix has to move one more thing off the Temporal path: the Stripe library's API key.

The key is set only in the Stripe integration's startup hook (internal/integrations/stripe/stripe.go:212-214). cmd/start.go runs that hook inside the Temporal block (lines 259 and 313-317), before the server starts (line 482), so today the key is always set before the first request. Once the server starts ahead of the Temporal connection, requests arrive before the hook runs. Until Temporal answers:

Checkout, the plan switch and its preview, and cancellation call Stripe with no key, and every call fails.
Checkout and the plan moves read the environment of the key the library holds (SDKKeyMode, stripe.go:298-300), which is empty. Every other page reads the environment boot derives from the configured key (cmd/start.go:126). The two disagree about which Stripe records are reachable.

Setting the key at boot, beside the line that derives its environment, closes both. Checkout and the plan moves can then read the boot environment like every other page, and SDKKeyMode can go.

The same gap exists today in one configuration: a Stripe key with no Temporal host. The library never gets its key, and webhooks are stored but never processed. If that is not a configuration the console supports, boot can refuse it, as it refuses a key with an unknown prefix.

The fix has to move one more thing off the Temporal path: the Stripe library's API key. The key is set only in the Stripe integration's startup hook (`internal/integrations/stripe/stripe.go:212-214`). `cmd/start.go` runs that hook inside the Temporal block (lines 259 and 313-317), before the server starts (line 482), so today the key is always set before the first request. Once the server starts ahead of the Temporal connection, requests arrive before the hook runs. Until Temporal answers: - Checkout, the plan switch and its preview, and cancellation call Stripe with no key, and every call fails. - Checkout and the plan moves read the environment of the key the library holds (`SDKKeyMode`, `stripe.go:298-300`), which is empty. Every other page reads the environment boot derives from the configured key (`cmd/start.go:126`). The two disagree about which Stripe records are reachable. Setting the key at boot, beside the line that derives its environment, closes both. Checkout and the plan moves can then read the boot environment like every other page, and `SDKKeyMode` can go. The same gap exists today in one configuration: a Stripe key with no Temporal host. The library never gets its key, and webhooks are stored but never processed. If that is not a configuration the console supports, boot can refuse it, as it refuses a key with an unknown prefix.

cgalo5758 commented

2026-10-19 16:17:38 +00:00

A host reboot hit this again: every on-demand host was without TLS until the proxy's ask was pointed elsewhere.

This time the loop also ran through the Temporal token. temporal-oauth-token-url named the identity provider's public host, so fetching the token went through the proxy whose ask was waiting on the console. Pointing the token URL at the identity provider's internal address takes the Temporal connect off the proxy, but it does not break the loop: before the listener opens, server.Start also fetches the discovery document from oidc-idp-issuer-url (internal/auth/auth.go:199), which has to be the public name the issuer matches. Until the fix here lands, the mitigation in the description, an ordinary managed certificate for the identity provider's host, is the one that holds.

Temporal's own fetch of the identity provider's signing keys also goes through the proxy when Temporal is not on the identity provider's network. Once the console answers the ask without waiting on Temporal, that fetch recovers as soon as the proxy issues the certificate, and it needs no change here.

A host reboot hit this again: every on-demand host was without TLS until the proxy's ask was pointed elsewhere. This time the loop also ran through the Temporal token. `temporal-oauth-token-url` named the identity provider's public host, so fetching the token went through the proxy whose ask was waiting on the console. Pointing the token URL at the identity provider's internal address takes the Temporal connect off the proxy, but it does not break the loop: before the listener opens, `server.Start` also fetches the discovery document from `oidc-idp-issuer-url` (`internal/auth/auth.go:199`), which has to be the public name the issuer matches. Until the fix here lands, the mitigation in the description, an ordinary managed certificate for the identity provider's host, is the one that holds. Temporal's own fetch of the identity provider's signing keys also goes through the proxy when Temporal is not on the identity provider's network. Once the console answers the ask without waiting on Temporal, that fetch recovers as soon as the proxy issues the certificate, and it needs no change here.

cgalo5758 added this to the Public launch milestone 2026-10-20 23:14:01 +00:00

cgalo5758 referenced this issue

2026-10-24 05:00:56 +00:00

Nobody is told when the console serves without Temporal, its identity provider or its session store #253

cgalo5758 referenced this issue

2026-10-24 05:12:33 +00:00

Decide whether the domains system should live inside the console #254