Inbound webhook events are kept forever #249

New Issue

2026-10-22T22:21:35Z

cgalo5758 commented

2026-10-22 22:21:35 +00:00

The question

How long should the console keep the inbound webhook events it stores in core.webhook_events, and how does it let them go? Today it keeps every one. Monthly partitions are created ahead of need (internal/db/partitions.go, at boot and on a daily schedule) and never detached or dropped, and no code deletes a row.

What depends on the answer

Stripe's processing reads some old rows back: the invoice sweep reads dead-lettered events, and the ordering check compares an incoming event with completed ones, so a cutoff for Stripe has to leave those reads what they need. Discourse events are never read back once they are handled. Each stored event holds part of a provider's message body, so keeping fewer of them keeps less data about people.

Options known so far

The design's audit log, whose monthly partitioning this table copies, also has the rest of an answer for its own rows: retention tiers assigned when an event is written, and storage temperatures from hot to frozen (design/data-model.md, the audit log and audit_retention_policies sections, and the upstream design's audit log architecture document under design/documents/). Neither the audit log nor that lifecycle is built.
The desired-state outbox deletes completed rows after seven days, which is a precedent for retention on a shared substrate table.
Detach and drop whole monthly partitions past a cutoff, or delete rows by provider and processing status, which lets each provider keep its own window.

Decided when

The retention of each provider's events, and how expired events are removed, are written in docs/models/payments-billing.md and in the webhook partition maintenance spec.

### The question How long should the console keep the inbound webhook events it stores in `core.webhook_events`, and how does it let them go? Today it keeps every one. Monthly partitions are created ahead of need (`internal/db/partitions.go`, at boot and on a daily schedule) and never detached or dropped, and no code deletes a row. ### What depends on the answer Stripe's processing reads some old rows back: the invoice sweep reads dead-lettered events, and the ordering check compares an incoming event with completed ones, so a cutoff for Stripe has to leave those reads what they need. Discourse events are never read back once they are handled. Each stored event holds part of a provider's message body, so keeping fewer of them keeps less data about people. ### Options known so far - The design's audit log, whose monthly partitioning this table copies, also has the rest of an answer for its own rows: retention tiers assigned when an event is written, and storage temperatures from hot to frozen (`design/data-model.md`, the audit log and `audit_retention_policies` sections, and the upstream design's audit log architecture document under `design/documents/`). Neither the audit log nor that lifecycle is built. - The desired-state outbox deletes completed rows after seven days, which is a precedent for retention on a shared substrate table. - Detach and drop whole monthly partitions past a cutoff, or delete rows by provider and processing status, which lets each provider keep its own window. ### Decided when The retention of each provider's events, and how expired events are removed, are written in `docs/models/payments-billing.md` and in the webhook partition maintenance spec.

cgalo5758 added the

kind

design

area/integrations privacy labels 2026-10-22 22:21:35 +00:00

cgalo5758 referenced this issue

2026-10-22 23:32:30 +00:00

The Discourse webhook receiver stores members' email addresses in the raw event log #163

Sign in to join this conversation.