An operator's Retry of a failed Stripe sync can be undone by the dead-letter write it answered #316

New Issue

2026-10-31T06:23:53Z

cgalo5758 commented

2026-10-31 06:23:53 +00:00

What happens

The console sends product and price creates to Stripe through the outbox, its queue of Stripe actions. Each queued row is run by its own Temporal workflow (Temporal is the workflow engine the console runs its background work on), retried for up to a day, and then dead-lettered: set to dead_letter, set aside as failed for an operator. The product's readiness panel then shows the sync as failed and offers "Find or create in Stripe", which puts the row back to pending so the queue runs it again.

The dead-letter write is a separate step of the workflow, and Temporal runs it again, up to five times, whenever it does not hear back from an attempt (internal/integrations/stripe/workflows/retry_budget.go:45-46, internal/workflows/common/options.go:14-23). The write sets the row to dead_letter whatever status the row holds (internal/integrations/stripe/workflows/outbox.go:296-305).

When the write's first attempt reaches the database but its answer does not reach Temporal (the worker stops or loses its connection to Temporal at that moment, or the attempt outlasts its one-minute timeout), the panel already shows the failure. If the operator presses "Find or create in Stripe" before Temporal's next attempt, about a minute later, the row goes back to pending (internal/integrations/stripe/catalogsync/catalogsync.go:146-157). The queue does not start the row while the old workflow is still running (outbox.go:113-118), and the old workflow's next attempt of the dead-letter write sets the row to dead_letter again, with the old error. The panel shows the same failure again, as if the button had never been pressed.

The webhook queue's dead-letter write has the same shape.

What should happen

The dead-letter write changes a row only from the statuses its own run leaves it in (failed, or processing when recording the failure did not succeed). A row the operator has sent back to pending stays pending, and the queue runs it once the old workflow has ended.

Where

internal/integrations/stripe/workflows/outbox.go (MarkOutboxDeadLetter)

Why it matters

A retry the panel accepted is silently undone, and the operator has to notice the returning failure and press the button again.

Related: #312.

### What happens The console sends product and price creates to Stripe through the outbox, its queue of Stripe actions. Each queued row is run by its own Temporal workflow (Temporal is the workflow engine the console runs its background work on), retried for up to a day, and then dead-lettered: set to `dead_letter`, set aside as failed for an operator. The product's readiness panel then shows the sync as failed and offers "Find or create in Stripe", which puts the row back to `pending` so the queue runs it again. The dead-letter write is a separate step of the workflow, and Temporal runs it again, up to five times, whenever it does not hear back from an attempt (`internal/integrations/stripe/workflows/retry_budget.go:45-46`, `internal/workflows/common/options.go:14-23`). The write sets the row to `dead_letter` whatever status the row holds (`internal/integrations/stripe/workflows/outbox.go:296-305`). When the write's first attempt reaches the database but its answer does not reach Temporal (the worker stops or loses its connection to Temporal at that moment, or the attempt outlasts its one-minute timeout), the panel already shows the failure. If the operator presses "Find or create in Stripe" before Temporal's next attempt, about a minute later, the row goes back to `pending` (`internal/integrations/stripe/catalogsync/catalogsync.go:146-157`). The queue does not start the row while the old workflow is still running (`outbox.go:113-118`), and the old workflow's next attempt of the dead-letter write sets the row to `dead_letter` again, with the old error. The panel shows the same failure again, as if the button had never been pressed. The webhook queue's dead-letter write has the same shape. ### What should happen The dead-letter write changes a row only from the statuses its own run leaves it in (`failed`, or `processing` when recording the failure did not succeed). A row the operator has sent back to `pending` stays `pending`, and the queue runs it once the old workflow has ended. ### Where `internal/integrations/stripe/workflows/outbox.go` (`MarkOutboxDeadLetter`) ### Why it matters A retry the panel accepted is silently undone, and the operator has to notice the returning failure and press the button again. Related: #312.

cgalo5758 added the

kind

bug

area/billing area/integrations labels 2026-10-31 06:23:53 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: wiki-cafe/member-console#316