Sweep hourly for a subscription whose collection pause disagrees with the rule
Sixth section of #269's change. An outbox entry can be lost to a dead letter, a Stripe call can land late, and the Dashboard can change a pause, so the hourly scheduled-change sweep now also runs SweepCollectionDrift. It pages the mapped live subscriptions (active, trialing, past_due, unpaid and paused; a mapping with no Stripe id is left out), drops the ones whose mapping the key mode cannot reach, and reads each page's conferring products, live provisions, shapes, displacements and recent reconcile entries once. A subscription whose recorded pause disagrees with CollectionPauseWanted (wanted and not recorded, or not wanted with the console's marker recorded; a Dashboard pause never drifts) gets one reconcile_subscription entry, unless one is pending, processing or failed, or a dead letter is less than a day old. Each page's entries are written in one transaction. SweepScheduledChangesWorkflow runs both activities, logs each failure, and returns the first failure after both have run, so a failed scheduled-change sweep no longer skips the collection sweep. The server's per-subscription displacement conversion moves to entitlements.DisplacementsBySubscription, which the server and the sweep both call. The comment on ListLiveSubscriptionsOfPool now says the drift sweep covers the Stripe-paused subscriptions it leaves out. docs/models/payments-billing.md's Reconciler row names the sweep and the dead letter's one-day wait. Tests: TestCollectionDrift, plus TestCollectionDrift_NamesTheIdsInOrder for the two input orders; TestSweepCollectionDrift_* for a displaced subscription, the recent-entry rows, one read per page (its failure text also names the read), a failed read, a set-less item riding the pause, an unmapped subscription, a mapping for the other key mode or with no Stripe id, and a Stripe-paused subscription with the marker; TestSweepScheduledChangesWorkflowRunsTheCollectionSweepAfterAFailure, plus a case where the collection sweep fails. Review: no blocker or major. Four minors applied (the page's write through db.InTx, a doc comment that said an hour where the code waits a day, a clearer comment on the live-subscriptions query, tests and breaks for the Stripe-id and paused-status predicates) and two nits (no COALESCE; the pause recorded through the test kit). Breaks: 13 of 13 caught. Full suite on reset databases: 4807 pass, 14 skip. make screens: no change beyond the three captures every run since section 2 redraws.
This commit is contained in:
@@ -37,7 +37,7 @@ Database objects sit in the `core` schema (defined in `internal/db/migrations/00
|
||||
| Invoice reconcile | `ReconcileInvoice`: reads the invoice, its lines and the charges of its payment intent back from the Stripe API at the SDK's version, takes a per-invoice lock, and converges the invoice (created and numbered by the first run that finds it issued, then only moved forward), its lines (inserted once), and one payment per settled charge | `internal/integrations/stripe/workflows/invoice_reconcile.go` |
|
||||
| Invoice sweep | The hourly Temporal schedule (`stripe-invoice-sweep`), also triggered at every boot, that reads back invoice mappings never read back, open invoices last read more than a day ago, and invoices named by dead-lettered invoice events, then fills payment methods and subscription links mapped since; it never renumbers, deletes, or reconciles a subscription. The Stripe provider page states when it last finished | `internal/integrations/stripe/workflows/invoice_sweep.go` |
|
||||
| Outbox drainer | The recurring worker that executes outbox rows (create Stripe product / price / customer), retrying with backoff until a cap parks the row as `dead_letter` for operator-driven retry | `internal/integrations/stripe/workflows/outbox.go` |
|
||||
| Reconciler | `ReconcileSubscription`: refetches the subscription from the Stripe API and converges console records and provisions to match. It runs on `customer.subscription.created`, `.updated` and `.deleted`, on `checkout.session.completed` for a subscription, on the checkout return (for the customer's most recent subscription), in the scheduled-change sweeper when it reaches a period-end downgrade or fires a term-boundary switch or cancellation, after each member plan change, and from an invoice event's reconcile when the invoice's subscription or one of its line items has no mapping yet. The reconcile writes the collection pause Stripe reports onto the subscription row on every run. A desired product whose ladder another plan took from the subscription and holds is left alone and logged; when grants hold every ladder the subscription's conferring products are left alone on, the reconcile asks for its collection to be paused; a subscription that pays for two tiers of one ladder gets the higher-ranked one, the other logged as a warning and not conferred (a subscription that alternated between the two before the reconcile chose by rank moves to the higher-ranked tier at its next reconcile and stays there); the reconcile writes in product id order under the pool row | `internal/fulfillment/reconcile.go` |
|
||||
| Reconciler | `ReconcileSubscription`: refetches the subscription from the Stripe API and converges console records and provisions to match. It runs on `customer.subscription.created`, `.updated` and `.deleted`, on `checkout.session.completed` for a subscription, on the checkout return (for the customer's most recent subscription), in the scheduled-change sweeper when it reaches a period-end downgrade or fires a term-boundary switch or cancellation, after each member plan change, and from an invoice event's reconcile when the invoice's subscription or one of its line items has no mapping yet. The reconcile writes the collection pause Stripe reports onto the subscription row on every run. A desired product whose ladder another plan took from the subscription and holds is left alone and logged; when grants hold every ladder the subscription's conferring products are left alone on, the reconcile asks for its collection to be paused. An hourly sweep, a second activity of the scheduled-change sweep workflow, reads the recorded pause and the rule's inputs from the database and asks again, through the outbox, for each live subscription whose pause disagrees with the rule; it skips a subscription that has an entry in flight or a dead letter from the last day, so a dead-lettered request is made again after a day. A subscription that pays for two tiers of one ladder gets the higher-ranked one, the other logged as a warning and not conferred (a subscription that alternated between the two before the reconcile chose by rank moves to the higher-ranked tier at its next reconcile and stays there); the reconcile writes in product id order under the pool row | `internal/fulfillment/reconcile.go` |
|
||||
| Plan-change mechanics | Member-initiated switch and cancel, a preview of the price difference before switching, the commitment gating, and the scheduled-change bookkeeping | `internal/fulfillment/plan_change.go` |
|
||||
| Scheduled-change sweeper | The hourly backstop that fires due scheduled changes Stripe itself cannot fire | `internal/fulfillment/sweep.go`; `internal/workflows/billing/` |
|
||||
| Checkout | Runs the guard checks and starts the Stripe Checkout session; also creates the Stripe customer synchronously if the organization has none yet | `internal/server/billing.go` |
|
||||
|
||||
@@ -267,6 +267,12 @@ type Querier interface {
|
||||
// account id: it is time-ordered, which lists the later-written account first,
|
||||
// so each lists on exactly one page.
|
||||
ListBillingAccountsPage(ctx context.Context, arg ListBillingAccountsPageParams) ([]ListBillingAccountsPageRow, error)
|
||||
// The products of the listed subscriptions' current items whose product has
|
||||
// an entitlement set, one row per subscription and product: the products
|
||||
// the collection pause rule counts (grant-pauses-subscription-billing D2),
|
||||
// in one statement for a page of the drift sweep (D5). A product with no
|
||||
// set rides the pause and is left out.
|
||||
ListCurrentConferringProductsBySubscriptionIDs(ctx context.Context, subscriptionIds []string) ([]ListCurrentConferringProductsBySubscriptionIDsRow, error)
|
||||
// The current items of every subscription of the billing account, the rows
|
||||
// mirrorItems writes, in one statement for the organization page's Billing
|
||||
// card (displaced-subscription-waits D4).
|
||||
|
||||
@@ -50,3 +50,17 @@ FROM core.subscription_items si
|
||||
WHERE si.subscription_id = ANY(sqlc.arg(subscription_ids)::uuid[])
|
||||
AND si.ended_at IS NULL
|
||||
ORDER BY si.subscription_id, si.product_id;
|
||||
|
||||
-- name: ListCurrentConferringProductsBySubscriptionIDs :many
|
||||
-- The products of the listed subscriptions' current items whose product has
|
||||
-- an entitlement set, one row per subscription and product: the products
|
||||
-- the collection pause rule counts (grant-pauses-subscription-billing D2),
|
||||
-- in one statement for a page of the drift sweep (D5). A product with no
|
||||
-- set rides the pause and is left out.
|
||||
SELECT DISTINCT si.subscription_id, si.product_id
|
||||
FROM core.subscription_items si
|
||||
JOIN core.products p ON p.product_id = si.product_id
|
||||
WHERE si.subscription_id = ANY(sqlc.arg(subscription_ids)::uuid[])
|
||||
AND si.ended_at IS NULL
|
||||
AND p.entitlement_set_id IS NOT NULL
|
||||
ORDER BY si.subscription_id, si.product_id;
|
||||
|
||||
@@ -126,6 +126,49 @@ func (q *Queries) GetSubscriptionItemsBySubscriptionID(ctx context.Context, subs
|
||||
return items, nil
|
||||
}
|
||||
|
||||
const listCurrentConferringProductsBySubscriptionIDs = `-- name: ListCurrentConferringProductsBySubscriptionIDs :many
|
||||
SELECT DISTINCT si.subscription_id, si.product_id
|
||||
FROM core.subscription_items si
|
||||
JOIN core.products p ON p.product_id = si.product_id
|
||||
WHERE si.subscription_id = ANY($1::uuid[])
|
||||
AND si.ended_at IS NULL
|
||||
AND p.entitlement_set_id IS NOT NULL
|
||||
ORDER BY si.subscription_id, si.product_id
|
||||
`
|
||||
|
||||
type ListCurrentConferringProductsBySubscriptionIDsRow struct {
|
||||
SubscriptionID string `json:"subscription_id"`
|
||||
ProductID string `json:"product_id"`
|
||||
}
|
||||
|
||||
// The products of the listed subscriptions' current items whose product has
|
||||
// an entitlement set, one row per subscription and product: the products
|
||||
// the collection pause rule counts (grant-pauses-subscription-billing D2),
|
||||
// in one statement for a page of the drift sweep (D5). A product with no
|
||||
// set rides the pause and is left out.
|
||||
func (q *Queries) ListCurrentConferringProductsBySubscriptionIDs(ctx context.Context, subscriptionIds []string) ([]ListCurrentConferringProductsBySubscriptionIDsRow, error) {
|
||||
rows, err := q.db.QueryContext(ctx, listCurrentConferringProductsBySubscriptionIDs, pq.Array(subscriptionIds))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
items := []ListCurrentConferringProductsBySubscriptionIDsRow{}
|
||||
for rows.Next() {
|
||||
var i ListCurrentConferringProductsBySubscriptionIDsRow
|
||||
if err := rows.Scan(&i.SubscriptionID, &i.ProductID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, i)
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
const listCurrentSubscriptionItemsByBillingAccountID = `-- name: ListCurrentSubscriptionItemsByBillingAccountID :many
|
||||
SELECT si.subscription_id, si.product_id
|
||||
FROM core.subscription_items si
|
||||
|
||||
@@ -62,6 +62,24 @@ func DisplacementsFromRows(rows []ListSubscriptionDisplacementsRow) []Displaceme
|
||||
return out
|
||||
}
|
||||
|
||||
// DisplacementsBySubscription is the one mapping of a displacement row's
|
||||
// fields to Displacement, grouped by subscription. The organization page, the
|
||||
// subscriptions list and the collection drift sweep all call it. It reads no
|
||||
// row.
|
||||
func DisplacementsBySubscription(rows []ListAccountSubscriptionDisplacementsRow) map[string][]Displacement {
|
||||
by := make(map[string][]Displacement)
|
||||
for _, d := range rows {
|
||||
by[d.SubscriptionID] = append(by[d.SubscriptionID], Displacement{
|
||||
LadderID: d.PlanLadderID,
|
||||
LadderName: d.LadderName,
|
||||
DisplacedProductID: d.DisplacedProductID,
|
||||
HolderProvisionID: d.HolderProvisionID,
|
||||
HolderSource: d.HolderSource,
|
||||
})
|
||||
}
|
||||
return by
|
||||
}
|
||||
|
||||
// DisplacedProducts maps each missing product the reconcile leaves alone to
|
||||
// the displacement that decides it: the first of its rungs, in rung order,
|
||||
// whose ladder the subscription is displaced on. The check is per ladder, so
|
||||
|
||||
@@ -99,7 +99,8 @@ ORDER BY s.subscription_id
|
||||
// or unpaid), by subscription id. Every act that can change who holds a
|
||||
// ladder writes one reconcile request per row
|
||||
// (grant-pauses-subscription-billing D4); a Stripe-paused subscription is
|
||||
// not listed (design D4).
|
||||
// not listed (design D4); the collection drift sweep, whose page includes
|
||||
// paused subscriptions, asks for its reconcile.
|
||||
func (q *Queries) ListLiveSubscriptionsOfPool(ctx context.Context, poolID string) ([]string, error) {
|
||||
rows, err := q.db.QueryContext(ctx, listLiveSubscriptionsOfPool, poolID)
|
||||
if err != nil {
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"context"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/lib/pq"
|
||||
)
|
||||
|
||||
const countLiveProductHolders = `-- name: CountLiveProductHolders :one
|
||||
@@ -236,6 +237,45 @@ func (q *Queries) GetPoolProvisionByProvisionID(ctx context.Context, provisionID
|
||||
return i, err
|
||||
}
|
||||
|
||||
const listLiveProvisionProductsBySubscriptionIDs = `-- name: ListLiveProvisionProductsBySubscriptionIDs :many
|
||||
SELECT subscription_id::uuid AS subscription_id, product_id
|
||||
FROM core.pool_provisions
|
||||
WHERE subscription_id = ANY($1::uuid[])
|
||||
AND status <> 'ended'
|
||||
ORDER BY subscription_id, product_id
|
||||
`
|
||||
|
||||
type ListLiveProvisionProductsBySubscriptionIDsRow struct {
|
||||
SubscriptionID string `json:"subscription_id"`
|
||||
ProductID string `json:"product_id"`
|
||||
}
|
||||
|
||||
// The subscription and product of each live (non-ended) provision the listed
|
||||
// subscriptions hold, in one statement for a page of the collection drift
|
||||
// sweep (grant-pauses-subscription-billing D5).
|
||||
func (q *Queries) ListLiveProvisionProductsBySubscriptionIDs(ctx context.Context, subscriptionIds []string) ([]ListLiveProvisionProductsBySubscriptionIDsRow, error) {
|
||||
rows, err := q.db.QueryContext(ctx, listLiveProvisionProductsBySubscriptionIDs, pq.Array(subscriptionIds))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
items := []ListLiveProvisionProductsBySubscriptionIDsRow{}
|
||||
for rows.Next() {
|
||||
var i ListLiveProvisionProductsBySubscriptionIDsRow
|
||||
if err := rows.Scan(&i.SubscriptionID, &i.ProductID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, i)
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
const listPoolsCarryingSet = `-- name: ListPoolsCarryingSet :many
|
||||
SELECT DISTINCT pp.pool_id, rp.org_id, o.name AS org_name
|
||||
FROM core.pool_provisions pp
|
||||
|
||||
@@ -434,12 +434,17 @@ type Querier interface {
|
||||
// the preview's other-source lines: the preview names at most 50 such pools,
|
||||
// and reads their sources here in one statement.
|
||||
ListLivePlanSourcesByPools(ctx context.Context, poolIds []string) ([]ListLivePlanSourcesByPoolsRow, error)
|
||||
// The subscription and product of each live (non-ended) provision the listed
|
||||
// subscriptions hold, in one statement for a page of the collection drift
|
||||
// sweep (grant-pauses-subscription-billing D5).
|
||||
ListLiveProvisionProductsBySubscriptionIDs(ctx context.Context, subscriptionIds []string) ([]ListLiveProvisionProductsBySubscriptionIDsRow, error)
|
||||
// The subscriptions of the pool's organization, through its accounts'
|
||||
// billing accounts, whose Stripe status is live (active, trialing, past_due
|
||||
// or unpaid), by subscription id. Every act that can change who holds a
|
||||
// ladder writes one reconcile request per row
|
||||
// (grant-pauses-subscription-billing D4); a Stripe-paused subscription is
|
||||
// not listed (design D4).
|
||||
// not listed (design D4); the collection drift sweep, whose page includes
|
||||
// paused subscriptions, asks for its reconcile.
|
||||
ListLiveSubscriptionsOfPool(ctx context.Context, poolID string) ([]string, error)
|
||||
// The tier-removal preview/commit population in one round trip: every live
|
||||
// (active|suspended) junction on this ladder whose provision delivers this
|
||||
|
||||
@@ -134,7 +134,8 @@ ORDER BY s.subscription_id, pl.name, d.plan_ladder_id::uuid, d.holder_provision_
|
||||
-- or unpaid), by subscription id. Every act that can change who holds a
|
||||
-- ladder writes one reconcile request per row
|
||||
-- (grant-pauses-subscription-billing D4); a Stripe-paused subscription is
|
||||
-- not listed (design D4).
|
||||
-- not listed (design D4); the collection drift sweep, whose page includes
|
||||
-- paused subscriptions, asks for its reconcile.
|
||||
SELECT s.subscription_id::uuid AS subscription_id
|
||||
FROM core.resource_pools rp
|
||||
JOIN core.accounts a ON a.org_id = rp.org_id
|
||||
|
||||
@@ -60,3 +60,13 @@ SELECT COUNT(DISTINCT rp.org_id) AS holders,
|
||||
FROM core.pool_provisions pp
|
||||
JOIN core.resource_pools rp ON rp.pool_id = pp.pool_id
|
||||
WHERE pp.product_id = $1 AND pp.status <> 'ended';
|
||||
|
||||
-- name: ListLiveProvisionProductsBySubscriptionIDs :many
|
||||
-- The subscription and product of each live (non-ended) provision the listed
|
||||
-- subscriptions hold, in one statement for a page of the collection drift
|
||||
-- sweep (grant-pauses-subscription-billing D5).
|
||||
SELECT subscription_id::uuid AS subscription_id, product_id
|
||||
FROM core.pool_provisions
|
||||
WHERE subscription_id = ANY(sqlc.arg(subscription_ids)::uuid[])
|
||||
AND status <> 'ended'
|
||||
ORDER BY subscription_id, product_id;
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
|
||||
package fulfillment
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"slices"
|
||||
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/billing"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/db"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/entitlements"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/integration"
|
||||
internalstripe "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/stripe/store"
|
||||
)
|
||||
|
||||
// collectionSweepPageSize is how many live subscriptions one page of the
|
||||
// collection drift sweep reads.
|
||||
const collectionSweepPageSize = 500
|
||||
|
||||
// firstSweepKey is the keyset's starting point: the nil UUID sorts before
|
||||
// every subscription id.
|
||||
const firstSweepKey = "00000000-0000-0000-0000-000000000000"
|
||||
|
||||
// sweepReads holds the reads of the collection drift sweep, so the database
|
||||
// tests can set a page size and refuse a read. SweepCollectionDrift fills it
|
||||
// with the real queriers.
|
||||
type sweepReads struct {
|
||||
page func(ctx context.Context, after string, size int32) ([]internalstripe.ListLiveSubscriptionsForCollectionSweepRow, error)
|
||||
conferring func(ctx context.Context, ids []string) ([]billing.ListCurrentConferringProductsBySubscriptionIDsRow, error)
|
||||
live func(ctx context.Context, ids []string) ([]entitlements.ListLiveProvisionProductsBySubscriptionIDsRow, error)
|
||||
shapes func(ctx context.Context, products []string) ([]entitlements.ListConferralShapesRow, error)
|
||||
displacements func(ctx context.Context, ids []string) ([]entitlements.ListSubscriptionDisplacementsBySubscriptionIDsRow, error)
|
||||
recent func(ctx context.Context, ids []string) ([]string, error)
|
||||
}
|
||||
|
||||
// databaseSweepReads is the sweep's reads against the database.
|
||||
func databaseSweepReads(db *sql.DB) sweepReads {
|
||||
stripeQ, billingQ, entQ, integrationQ := internalstripe.New(db), billing.New(db), entitlements.New(db), integration.New(db)
|
||||
return sweepReads{
|
||||
page: func(ctx context.Context, after string, size int32) ([]internalstripe.ListLiveSubscriptionsForCollectionSweepRow, error) {
|
||||
return stripeQ.ListLiveSubscriptionsForCollectionSweep(ctx, internalstripe.ListLiveSubscriptionsForCollectionSweepParams{
|
||||
AfterSubscriptionID: after,
|
||||
PageSize: size,
|
||||
})
|
||||
},
|
||||
conferring: billingQ.ListCurrentConferringProductsBySubscriptionIDs,
|
||||
live: entQ.ListLiveProvisionProductsBySubscriptionIDs,
|
||||
shapes: entQ.ListConferralShapes,
|
||||
displacements: entQ.ListSubscriptionDisplacementsBySubscriptionIDs,
|
||||
recent: integrationQ.ListSubscriptionsWithRecentReconcileEntries,
|
||||
}
|
||||
}
|
||||
|
||||
// collectionDrift names the subscriptions whose recorded collection pause
|
||||
// disagrees with the rule, in subscription id order. It reads no row. For each
|
||||
// subscription it passes every recorded conferring product (products) as
|
||||
// desired to entitlements.CollectionPauseWanted, with left the map
|
||||
// entitlements.DisplacedProducts returns over those with no live provision
|
||||
// (live), as convergeConferrals builds it. A subscription drifts when the rule
|
||||
// wants a pause and its mirror records none, or when the rule wants none and
|
||||
// its mirror carries the console's marker. A pause without the marker is a
|
||||
// foreign one and never drifts. A Stripe-paused subscription is never wanted
|
||||
// paused, so one whose mirror carries the marker drifts.
|
||||
func collectionDrift(
|
||||
subs []internalstripe.ListLiveSubscriptionsForCollectionSweepRow,
|
||||
products map[string][]string,
|
||||
live map[string][]string,
|
||||
rungs map[string][]entitlements.Rung,
|
||||
displacements map[string][]entitlements.Displacement,
|
||||
) []string {
|
||||
var drifting []string
|
||||
for _, s := range subs {
|
||||
desired := products[s.SubscriptionID]
|
||||
var missing []string
|
||||
for _, productID := range desired {
|
||||
if !slices.Contains(live[s.SubscriptionID], productID) {
|
||||
missing = append(missing, productID)
|
||||
}
|
||||
}
|
||||
left := entitlements.DisplacedProducts(missing, rungs, displacements[s.SubscriptionID])
|
||||
wanted := entitlements.CollectionPauseWanted(s.Status, desired, left)
|
||||
if collectionDrifts(wanted, s) {
|
||||
drifting = append(drifting, s.SubscriptionID)
|
||||
}
|
||||
}
|
||||
slices.Sort(drifting)
|
||||
return drifting
|
||||
}
|
||||
|
||||
// collectionDrifts compares the rule's answer with a subscription's mirror.
|
||||
func collectionDrifts(wanted bool, s internalstripe.ListLiveSubscriptionsForCollectionSweepRow) bool {
|
||||
if wanted {
|
||||
return !s.PauseCollectionBehavior.Valid
|
||||
}
|
||||
return s.PauseCollectionMarked
|
||||
}
|
||||
|
||||
// SweepCollectionDrift asks, through the outbox, for a reconcile of each live
|
||||
// subscription whose recorded collection pause disagrees with the rule, and
|
||||
// returns how many it asked for. It reads the database only. Run hourly, it
|
||||
// finds an act that forgot to request a reconcile within the hour, and a
|
||||
// dead-lettered entry a day after it was dead-lettered; it is also the
|
||||
// deploy's backfill (grant-pauses-subscription-billing D5). A subscription
|
||||
// that already has an entry in flight, or a dead letter from the last day, is
|
||||
// not asked for again.
|
||||
func SweepCollectionDrift(ctx context.Context, db *sql.DB, logger *slog.Logger, keyMode string) (int, error) {
|
||||
return sweepCollectionDrift(ctx, db, logger, keyMode, collectionSweepPageSize, databaseSweepReads(db))
|
||||
}
|
||||
|
||||
func sweepCollectionDrift(ctx context.Context, db *sql.DB, logger *slog.Logger, keyMode string, pageSize int, reads sweepReads) (int, error) {
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
asked := 0
|
||||
after := firstSweepKey
|
||||
for {
|
||||
rows, err := reads.page(ctx, after, int32(pageSize))
|
||||
if err != nil {
|
||||
return asked, sweepReadFailed(logger, "list live subscriptions", err)
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
break
|
||||
}
|
||||
after = rows[len(rows)-1].SubscriptionID
|
||||
n, err := sweepPage(ctx, db, logger, reachableRows(rows, keyMode), reads)
|
||||
asked += n
|
||||
if err != nil {
|
||||
return asked, err
|
||||
}
|
||||
if len(rows) < pageSize {
|
||||
break
|
||||
}
|
||||
}
|
||||
if asked > 0 {
|
||||
logger.Info("collection sweep: subscriptions asked for a reconcile", slog.Int("count", asked))
|
||||
}
|
||||
return asked, nil
|
||||
}
|
||||
|
||||
// reachableRows drops the rows whose mapping the key cannot reach: the
|
||||
// reconcile such a row would ask for completes as a skip, changes no mirror
|
||||
// and would be asked for again every hour.
|
||||
func reachableRows(rows []internalstripe.ListLiveSubscriptionsForCollectionSweepRow, keyMode string) []internalstripe.ListLiveSubscriptionsForCollectionSweepRow {
|
||||
var out []internalstripe.ListLiveSubscriptionsForCollectionSweepRow
|
||||
for _, r := range rows {
|
||||
if internalstripe.MappingReachable(r.Livemode, r.SyncStatus, keyMode) {
|
||||
out = append(out, r)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// sweepPage reads what the rule needs for one page, once each, and writes one
|
||||
// entry per drifting subscription without a recent entry, in one transaction.
|
||||
// It reads for every non-empty page, though its rows may all be unreachable.
|
||||
func sweepPage(ctx context.Context, db *sql.DB, logger *slog.Logger, rows []internalstripe.ListLiveSubscriptionsForCollectionSweepRow, reads sweepReads) (int, error) {
|
||||
ids := make([]string, len(rows))
|
||||
for i, r := range rows {
|
||||
ids[i] = r.SubscriptionID
|
||||
}
|
||||
drifting, err := pageDrift(ctx, logger, rows, ids, reads)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
recent, err := reads.recent(ctx, ids)
|
||||
if err != nil {
|
||||
return 0, sweepReadFailed(logger, "list recent reconcile entries", err)
|
||||
}
|
||||
var ask []string
|
||||
for _, id := range drifting {
|
||||
if !slices.Contains(recent, id) {
|
||||
ask = append(ask, id)
|
||||
}
|
||||
}
|
||||
if err := requestReconciles(ctx, db, ask); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return len(ask), nil
|
||||
}
|
||||
|
||||
// pageDrift reads one page's inputs and returns the drifting subscription ids.
|
||||
func pageDrift(ctx context.Context, logger *slog.Logger, rows []internalstripe.ListLiveSubscriptionsForCollectionSweepRow, ids []string, reads sweepReads) ([]string, error) {
|
||||
conferring, err := reads.conferring(ctx, ids)
|
||||
if err != nil {
|
||||
return nil, sweepReadFailed(logger, "list conferring products", err)
|
||||
}
|
||||
products := make(map[string][]string)
|
||||
all := make(map[string]struct{})
|
||||
for _, c := range conferring {
|
||||
products[c.SubscriptionID] = append(products[c.SubscriptionID], c.ProductID)
|
||||
all[c.ProductID] = struct{}{}
|
||||
}
|
||||
liveRows, err := reads.live(ctx, ids)
|
||||
if err != nil {
|
||||
return nil, sweepReadFailed(logger, "list live provisions", err)
|
||||
}
|
||||
live := make(map[string][]string)
|
||||
for _, l := range liveRows {
|
||||
live[l.SubscriptionID] = append(live[l.SubscriptionID], l.ProductID)
|
||||
}
|
||||
shapes, err := reads.shapes(ctx, slices.Sorted(maps.Keys(all)))
|
||||
if err != nil {
|
||||
return nil, sweepReadFailed(logger, "list conferral shapes", err)
|
||||
}
|
||||
displaced, err := reads.displacements(ctx, ids)
|
||||
if err != nil {
|
||||
return nil, sweepReadFailed(logger, "list subscription displacements", err)
|
||||
}
|
||||
converted := make([]entitlements.ListAccountSubscriptionDisplacementsRow, len(displaced))
|
||||
for i, d := range displaced {
|
||||
converted[i] = entitlements.ListAccountSubscriptionDisplacementsRow(d)
|
||||
}
|
||||
return collectionDrift(rows, products, live, entitlements.RungsByProduct(shapes), entitlements.DisplacementsBySubscription(converted)), nil
|
||||
}
|
||||
|
||||
// requestReconciles writes one reconcile entry per subscription in one plain
|
||||
// transaction.
|
||||
func requestReconciles(ctx context.Context, database *sql.DB, ids []string) error {
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
}
|
||||
err := db.InTx(ctx, database, db.Plain, func(tx *sql.Tx) error {
|
||||
for _, id := range ids {
|
||||
if err := entitlements.RequestSubscriptionReconcile(ctx, tx, id); err != nil {
|
||||
return fmt.Errorf("request a reconcile of %s: %w", id, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("collection sweep: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// sweepReadFailed logs a failed read (L6) and returns it wrapped.
|
||||
func sweepReadFailed(logger *slog.Logger, what string, err error) error {
|
||||
logger.Error("collection sweep: read failed", slog.Any("error", err))
|
||||
return fmt.Errorf("collection sweep: %s: %w", what, err)
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
|
||||
package fulfillment_test
|
||||
|
||||
// The collection drift sweep asks, through the outbox, for a reconcile of a
|
||||
// subscription whose recorded pause disagrees with the rule (openspec change
|
||||
// grant-pauses-subscription-billing, design D5, tasks 6.2 and 6.4).
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"testing"
|
||||
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/fulfillment"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/stripetest"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/dbtest"
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/testkit/world"
|
||||
)
|
||||
|
||||
// clearReconcileEntries removes the reconcile_subscription entries of one
|
||||
// subscription, so a test starts from none.
|
||||
func clearReconcileEntries(t *testing.T, database *sql.DB, coreSub string) {
|
||||
t.Helper()
|
||||
if _, err := database.ExecContext(context.Background(),
|
||||
`DELETE FROM core.outbox WHERE action_type = 'reconcile_subscription' AND payload->>'subscription_id' = $1`, coreSub); err != nil {
|
||||
t.Fatalf("clear the entries of %s: %v", coreSub, err)
|
||||
}
|
||||
}
|
||||
|
||||
// runSweep runs the sweep under the test key.
|
||||
func runSweep(t *testing.T, database *sql.DB) int {
|
||||
t.Helper()
|
||||
n, err := fulfillment.SweepCollectionDrift(context.Background(), database, discardLogger(), stripetest.KeyMode())
|
||||
if err != nil {
|
||||
t.Fatalf("SweepCollectionDrift: %v", err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// A grant over a subscription whose mirror records no pause is asked for.
|
||||
func TestSweepCollectionDrift_AsksForADisplacedSubscription(t *testing.T) {
|
||||
database, fx, _, _ := grantOver(t, false)
|
||||
core := coreSubID(t, database, fx.stripeSubID)
|
||||
clearReconcileEntries(t, database, core)
|
||||
|
||||
runSweep(t, database)
|
||||
|
||||
if n := reconcileEntries(t, database, core); n != 1 {
|
||||
t.Errorf("the sweep wrote %d entries for the displaced subscription, want 1", n)
|
||||
}
|
||||
}
|
||||
|
||||
// A Stripe-paused subscription that still records the console's marker is asked
|
||||
// for, because the grant pause's page includes paused subscriptions.
|
||||
func TestSweepCollectionDrift_AsksForAStripePausedSubscriptionWithTheMarker(t *testing.T) {
|
||||
database, fx, _, _ := grantOver(t, false)
|
||||
core := coreSubID(t, database, fx.stripeSubID)
|
||||
if _, err := database.ExecContext(context.Background(), `UPDATE core.subscriptions SET status = 'paused' WHERE subscription_id = $1`, core); err != nil {
|
||||
t.Fatalf("pause the subscription: %v", err)
|
||||
}
|
||||
world.New(t, database).PauseCollection(world.Subscription{ID: core})
|
||||
clearReconcileEntries(t, database, core)
|
||||
|
||||
runSweep(t, database)
|
||||
|
||||
if n := reconcileEntries(t, database, core); n != 1 {
|
||||
t.Errorf("the sweep wrote %d entries for a Stripe-paused subscription with the marker, want 1", n)
|
||||
}
|
||||
}
|
||||
|
||||
type recentEntryRow struct {
|
||||
name string
|
||||
status string
|
||||
ageHours int
|
||||
wantWrites int
|
||||
}
|
||||
|
||||
func checkRecentEntry(t *testing.T, row recentEntryRow) {
|
||||
t.Helper()
|
||||
database, fx, _, _ := grantOver(t, false)
|
||||
core := coreSubID(t, database, fx.stripeSubID)
|
||||
clearReconcileEntries(t, database, core)
|
||||
if _, err := database.ExecContext(context.Background(), `
|
||||
INSERT INTO core.outbox (provider, action_type, payload, status, updated_at)
|
||||
VALUES ('stripe', 'reconcile_subscription', jsonb_build_object('subscription_id', $1::text), $2, NOW() - make_interval(hours => $3))`,
|
||||
core, row.status, row.ageHours); err != nil {
|
||||
t.Fatalf("insert the %s entry: %v", row.name, err)
|
||||
}
|
||||
|
||||
runSweep(t, database)
|
||||
|
||||
if got := reconcileEntries(t, database, core) - 1; got != row.wantWrites {
|
||||
t.Errorf("with %s, the sweep wrote %d entries, want %d", row.name, got, row.wantWrites)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSweepCollectionDrift_SkipsASubscriptionWithAnEntry(t *testing.T) {
|
||||
for _, row := range []recentEntryRow{
|
||||
{name: "a pending entry", status: "pending", wantWrites: 0},
|
||||
{name: "a dead letter an hour old", status: "dead_letter", ageHours: 1, wantWrites: 0},
|
||||
{name: "a dead letter two days old", status: "dead_letter", ageHours: 48, wantWrites: 1},
|
||||
} {
|
||||
t.Run(row.name, func(t *testing.T) { checkRecentEntry(t, row) })
|
||||
}
|
||||
}
|
||||
|
||||
// driftingSubscriptions makes n mapped subscriptions that record the console's
|
||||
// pause with nothing to hold it, which the rule wants unpaused.
|
||||
func driftingSubscriptions(t *testing.T, database *sql.DB, n int) []string {
|
||||
t.Helper()
|
||||
w := world.New(t, database)
|
||||
var ids []string
|
||||
for i := range n {
|
||||
o := w.Org(fmt.Sprintf("Drift %d", i), w.OrgType(fmt.Sprintf("drift%d", i)), w.Person(fmt.Sprintf("Drift Owner %d", i)))
|
||||
w.MapAccount(w.BillingAccount(o, fmt.Sprintf("Drift %d", i)))
|
||||
plan := w.Product(fmt.Sprintf("Drift Plan %d", i))
|
||||
w.Price(plan, 1200)
|
||||
sub := w.Subscribe(o, world.SubOpts{}, plan)
|
||||
w.MapSubscription(sub)
|
||||
w.PauseCollection(sub)
|
||||
ids = append(ids, sub.ID)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
// eligibleSubscriptions counts the rows the sweep's page query returns.
|
||||
func eligibleSubscriptions(t *testing.T, database *sql.DB) int {
|
||||
t.Helper()
|
||||
return scalarInt(t, database, `
|
||||
SELECT count(*) FROM core.subscriptions s
|
||||
JOIN stripe.subscription_mappings m
|
||||
ON m.subscription_id = s.subscription_id AND m.stripe_subscription_id IS NOT NULL
|
||||
WHERE s.status IN ('active', 'trialing', 'past_due', 'unpaid', 'paused')`)
|
||||
}
|
||||
|
||||
// Each page of the sweep reads its inputs once, however many subscriptions it
|
||||
// holds. The shared database holds other tests' subscriptions, so the test
|
||||
// counts the non-empty pages the page query returns and compares each
|
||||
// per-page read with it.
|
||||
func TestSweepCollectionDrift_ReadsOncePerPage(t *testing.T) {
|
||||
const pageSize = 5
|
||||
database := dbtest.Open(t)
|
||||
driftingSubscriptions(t, database, 12)
|
||||
counted, stmts := dbtest.Counted(t)
|
||||
pages := (eligibleSubscriptions(t, database) + pageSize - 1) / pageSize
|
||||
stmts.Reset()
|
||||
|
||||
if _, err := fulfillment.SweepCollectionDriftPaged(context.Background(), counted, discardLogger(), stripetest.KeyMode(), pageSize, nil); err != nil {
|
||||
t.Fatalf("sweepCollectionDrift: %v", err)
|
||||
}
|
||||
|
||||
byName := stmts.ByName()
|
||||
for _, name := range []string{
|
||||
"ListCurrentConferringProductsBySubscriptionIDs",
|
||||
"ListLiveProvisionProductsBySubscriptionIDs",
|
||||
"ListConferralShapes",
|
||||
"ListSubscriptionDisplacementsBySubscriptionIDs",
|
||||
"ListSubscriptionsWithRecentReconcileEntries",
|
||||
} {
|
||||
if got := byName[name]; got != pages {
|
||||
t.Errorf("the sweep ran %d statements for %d pages, want %d (%s)", got, pages, pages, name)
|
||||
}
|
||||
}
|
||||
if pages < 3 {
|
||||
t.Errorf("the sweep paged %d times over twelve drifting subscriptions at a page size of %d, want at least 3", pages, pageSize)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSweepCollectionDrift_AFailedReadFailsTheRun(t *testing.T) {
|
||||
database, _, _ := subscribed(t)
|
||||
refused := errors.New("conferring products refused")
|
||||
var logs bytes.Buffer
|
||||
|
||||
_, err := fulfillment.SweepCollectionDriftPaged(context.Background(), database, slog.New(slog.NewTextHandler(&logs, nil)),
|
||||
stripetest.KeyMode(), 500, refused)
|
||||
|
||||
if !errors.Is(err, refused) {
|
||||
t.Errorf("a failed read = %v, want the error", err)
|
||||
}
|
||||
wantInLine(t, logs.String(), "collection sweep: read failed", "error=")
|
||||
}
|
||||
|
||||
// A product with no entitlement set rides the pause: it is not a conferring
|
||||
// product, so a subscription displaced on its tier and recorded paused agrees.
|
||||
func TestSweepCollectionDrift_ASetLessItemRidesThePause(t *testing.T) {
|
||||
database := dbtest.Open(t)
|
||||
fx := seedFixture(t, database)
|
||||
free, stripeFree := addPriced(t, database, fx, "setless")
|
||||
if _, err := database.ExecContext(context.Background(), `UPDATE core.products SET entitlement_set_id = NULL WHERE product_id = $1`, free); err != nil {
|
||||
t.Fatalf("take the set off the extra product: %v", err)
|
||||
}
|
||||
mock := &stripetest.MockBackend{Sub: fx.activeSub(fx.planItem(), item("si_free_"+fx.sfx, stripeFree))}
|
||||
t.Cleanup(stripetest.Install(mock))
|
||||
reconcile(t, database, fx.stripeSubID, "test:created")
|
||||
prod2, _, _ := secondTier(t, database, fx)
|
||||
issueGrant(t, database, fx, prod2, false)
|
||||
core := coreSubID(t, database, fx.stripeSubID)
|
||||
world.New(t, database).PauseCollection(world.Subscription{ID: core})
|
||||
clearReconcileEntries(t, database, core)
|
||||
|
||||
runSweep(t, database)
|
||||
|
||||
if n := reconcileEntries(t, database, core); n != 0 {
|
||||
t.Errorf("the sweep wrote %d entries for a paused subscription with a set-less item, want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
type unmappedRow struct {
|
||||
name string
|
||||
arrange func(t *testing.T, database *sql.DB, core string)
|
||||
}
|
||||
|
||||
func checkUnmapped(t *testing.T, row unmappedRow) {
|
||||
t.Helper()
|
||||
database, fx, _, _ := grantOver(t, false)
|
||||
core := coreSubID(t, database, fx.stripeSubID)
|
||||
row.arrange(t, database, core)
|
||||
clearReconcileEntries(t, database, core)
|
||||
|
||||
_, err := fulfillment.SweepCollectionDrift(context.Background(), database, discardLogger(), stripetest.KeyMode())
|
||||
|
||||
if err != nil {
|
||||
t.Errorf("with %s, the sweep returned %v", row.name, err)
|
||||
}
|
||||
if got := reconcileEntries(t, database, core); got != 0 {
|
||||
t.Errorf("with %s, the sweep wrote %d entries, want 0", row.name, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSweepCollectionDrift_SkipsAnUnmappedSubscription(t *testing.T) {
|
||||
otherMode := stripetest.KeyMode() != "live"
|
||||
for _, row := range []unmappedRow{
|
||||
{"no mapping", func(t *testing.T, database *sql.DB, core string) {
|
||||
if _, err := database.ExecContext(context.Background(), `DELETE FROM stripe.subscription_mappings WHERE subscription_id = $1`, core); err != nil {
|
||||
t.Fatalf("remove the mapping: %v", err)
|
||||
}
|
||||
}},
|
||||
{"a mapping for the other key mode", func(t *testing.T, database *sql.DB, core string) {
|
||||
if _, err := database.ExecContext(context.Background(), `UPDATE stripe.subscription_mappings SET livemode = $2 WHERE subscription_id = $1`, core, otherMode); err != nil {
|
||||
t.Fatalf("record the other environment: %v", err)
|
||||
}
|
||||
}},
|
||||
{"a mapping with no Stripe id", func(t *testing.T, database *sql.DB, core string) {
|
||||
if _, err := database.ExecContext(context.Background(), `UPDATE stripe.subscription_mappings SET stripe_subscription_id = NULL WHERE subscription_id = $1`, core); err != nil {
|
||||
t.Fatalf("clear the Stripe id: %v", err)
|
||||
}
|
||||
}},
|
||||
} {
|
||||
t.Run(row.name, func(t *testing.T) { checkUnmapped(t, row) })
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
|
||||
package fulfillment
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/entitlements"
|
||||
internalstripe "git.coopcloud.tech/wiki-cafe/member-console/internal/integrations/stripe/store"
|
||||
)
|
||||
|
||||
// driftRow is one subscription of the collection drift sweep's rule: its
|
||||
// status and recorded pause, the conferring products of its current items, the
|
||||
// products it holds live and the displacement of the one ladder it is
|
||||
// displaced on (holder "" for none).
|
||||
type driftRow struct {
|
||||
name string
|
||||
status string
|
||||
behavior string // the mirror's behavior, "" for none
|
||||
marked bool
|
||||
desired []string
|
||||
live []string
|
||||
holder string // the displacement's holder source, "" for none
|
||||
want bool
|
||||
}
|
||||
|
||||
const (
|
||||
driftProduct = "product-a"
|
||||
driftOther = "product-b"
|
||||
driftLadder = "ladder-a"
|
||||
driftOtherL = "ladder-b"
|
||||
)
|
||||
|
||||
func driftRows() []driftRow {
|
||||
return []driftRow{
|
||||
{name: "wanted, not recorded", status: "active", desired: []string{driftProduct}, holder: "grant", want: true},
|
||||
{name: "wanted, recorded", status: "active", behavior: "void", marked: true, desired: []string{driftProduct}, holder: "grant"},
|
||||
{name: "not wanted, the marker recorded", status: "active", behavior: "void", marked: true, desired: []string{driftProduct}, live: []string{driftProduct}, want: true},
|
||||
{name: "paused, the marker recorded", status: "paused", behavior: "void", marked: true, desired: []string{driftProduct}, holder: "grant", want: true},
|
||||
{name: "not wanted, a foreign pause", status: "active", behavior: "void", desired: []string{driftProduct}, live: []string{driftProduct}},
|
||||
{name: "displaced by a subscription", status: "active", desired: []string{driftProduct}, holder: "subscription"},
|
||||
{name: "partly displaced, nothing recorded", status: "active", desired: []string{driftProduct, driftOther}, live: []string{driftOther}, holder: "grant"},
|
||||
}
|
||||
}
|
||||
|
||||
func driftInputs(row driftRow, id string) (internalstripe.ListLiveSubscriptionsForCollectionSweepRow, map[string][]string, map[string][]string, map[string][]entitlements.Displacement) {
|
||||
sub := internalstripe.ListLiveSubscriptionsForCollectionSweepRow{
|
||||
SubscriptionID: id,
|
||||
Status: row.status,
|
||||
PauseCollectionMarked: row.marked,
|
||||
PauseCollectionBehavior: sql.NullString{
|
||||
String: row.behavior,
|
||||
Valid: row.behavior != "",
|
||||
},
|
||||
}
|
||||
displacements := map[string][]entitlements.Displacement{}
|
||||
if row.holder != "" {
|
||||
displacements[id] = []entitlements.Displacement{{
|
||||
LadderID: driftLadder, DisplacedProductID: driftProduct, HolderSource: row.holder,
|
||||
}}
|
||||
}
|
||||
return sub, map[string][]string{id: row.desired}, map[string][]string{id: row.live}, displacements
|
||||
}
|
||||
|
||||
func checkCollectionDrift(t *testing.T, row driftRow) {
|
||||
t.Helper()
|
||||
rungs := map[string][]entitlements.Rung{
|
||||
driftProduct: {{LadderID: driftLadder}},
|
||||
driftOther: {{LadderID: driftOtherL}},
|
||||
}
|
||||
sub, products, live, displacements := driftInputs(row, "sub-1")
|
||||
var want []string
|
||||
if row.want {
|
||||
want = []string{"sub-1"}
|
||||
}
|
||||
got := collectionDrift([]internalstripe.ListLiveSubscriptionsForCollectionSweepRow{sub}, products, live, rungs, displacements)
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("collectionDrift(%s) = %v, want %v", row.name, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectionDrift(t *testing.T) {
|
||||
for _, row := range driftRows() {
|
||||
t.Run(row.name, func(t *testing.T) { checkCollectionDrift(t, row) })
|
||||
}
|
||||
}
|
||||
|
||||
// TestCollectionDrift_NamesTheIdsInOrder feeds two drifting subscriptions in
|
||||
// both orders and wants the ids in id order either way.
|
||||
func TestCollectionDrift_NamesTheIdsInOrder(t *testing.T) {
|
||||
rungs := map[string][]entitlements.Rung{driftProduct: {{LadderID: driftLadder}}}
|
||||
row := driftRows()[0]
|
||||
a, productsA, liveA, dispA := driftInputs(row, "sub-a")
|
||||
b, productsB, liveB, dispB := driftInputs(row, "sub-b")
|
||||
merge := func(x, y map[string][]string) map[string][]string {
|
||||
out := map[string][]string{}
|
||||
for _, m := range []map[string][]string{x, y} {
|
||||
for k, v := range m {
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
displacements := map[string][]entitlements.Displacement{"sub-a": dispA["sub-a"], "sub-b": dispB["sub-b"]}
|
||||
products, live := merge(productsA, productsB), merge(liveA, liveB)
|
||||
want := []string{"sub-a", "sub-b"}
|
||||
for name, subs := range map[string][]internalstripe.ListLiveSubscriptionsForCollectionSweepRow{
|
||||
"ascending": {a, b},
|
||||
"descending": {b, a},
|
||||
} {
|
||||
if got := collectionDrift(subs, products, live, rungs, displacements); !slices.Equal(got, want) {
|
||||
t.Errorf("collectionDrift(ids %s) = %v, want %v", name, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
|
||||
package fulfillment
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"log/slog"
|
||||
|
||||
"git.coopcloud.tech/wiki-cafe/member-console/internal/billing"
|
||||
)
|
||||
|
||||
// SweepCollectionDriftPaged runs the collection drift sweep with a page size
|
||||
// and, when failConferring is set, with its conferring-products read refused
|
||||
// with that error.
|
||||
func SweepCollectionDriftPaged(ctx context.Context, db *sql.DB, logger *slog.Logger, keyMode string, pageSize int, failConferring error) (int, error) {
|
||||
reads := databaseSweepReads(db)
|
||||
if failConferring != nil {
|
||||
reads.conferring = func(context.Context, []string) ([]billing.ListCurrentConferringProductsBySubscriptionIDsRow, error) {
|
||||
return nil, failConferring
|
||||
}
|
||||
}
|
||||
return sweepCollectionDrift(ctx, db, logger, keyMode, pageSize, reads)
|
||||
}
|
||||
@@ -10,6 +10,8 @@ package integration
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/lib/pq"
|
||||
)
|
||||
|
||||
const countOutboxByStatus = `-- name: CountOutboxByStatus :one
|
||||
@@ -47,3 +49,43 @@ func (q *Queries) CountOutboxByStatus(ctx context.Context) (CountOutboxByStatusR
|
||||
err := row.Scan(&i.PendingCount, &i.FailedCount, &i.DeadLetterCount)
|
||||
return i, err
|
||||
}
|
||||
|
||||
const listSubscriptionsWithRecentReconcileEntries = `-- name: ListSubscriptionsWithRecentReconcileEntries :many
|
||||
SELECT DISTINCT (payload->>'subscription_id')::text AS subscription_id
|
||||
FROM core.outbox
|
||||
WHERE provider = 'stripe'
|
||||
AND action_type = 'reconcile_subscription'
|
||||
AND payload->>'subscription_id' = ANY($1::text[])
|
||||
AND (status IN ('pending', 'processing', 'failed')
|
||||
OR (status = 'dead_letter' AND updated_at > NOW() - interval '1 day'))
|
||||
ORDER BY 1
|
||||
`
|
||||
|
||||
// The listed subscriptions that already have a reconcile_subscription
|
||||
// entry in flight (pending, processing or failed) or dead-lettered within the
|
||||
// last day, so the collection drift sweep does not ask again for one the
|
||||
// outbox is still working on or has just given up (grant-pauses-
|
||||
// subscription-billing D5). A dead letter older than a day is asked for
|
||||
// again.
|
||||
func (q *Queries) ListSubscriptionsWithRecentReconcileEntries(ctx context.Context, subscriptionIds []string) ([]string, error) {
|
||||
rows, err := q.db.QueryContext(ctx, listSubscriptionsWithRecentReconcileEntries, pq.Array(subscriptionIds))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
items := []string{}
|
||||
for rows.Next() {
|
||||
var subscription_id string
|
||||
if err := rows.Scan(&subscription_id); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, subscription_id)
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
@@ -182,6 +182,13 @@ type Querier interface {
|
||||
ListPlatformResourceKeys(ctx context.Context) ([]string, error)
|
||||
ListProviderOperations(ctx context.Context, provider string) ([]string, error)
|
||||
ListProviders(ctx context.Context) ([]Provider, error)
|
||||
// The listed subscriptions that already have a reconcile_subscription
|
||||
// entry in flight (pending, processing or failed) or dead-lettered within the
|
||||
// last day, so the collection drift sweep does not ask again for one the
|
||||
// outbox is still working on or has just given up (grant-pauses-
|
||||
// subscription-billing D5). A dead letter older than a day is asked for
|
||||
// again.
|
||||
ListSubscriptionsWithRecentReconcileEntries(ctx context.Context, subscriptionIds []string) ([]string, error)
|
||||
// ---------------------------------------------------------------------------
|
||||
// Recomputing one record (design D9), in one READ COMMITTED transaction, in
|
||||
// this order: LockDesiredStateAddress, LockDesiredStateRecord, then
|
||||
|
||||
@@ -18,3 +18,19 @@ SELECT
|
||||
COUNT(*) FILTER (WHERE status = 'failed') AS failed_count,
|
||||
COUNT(*) FILTER (WHERE status = 'dead_letter') AS dead_letter_count
|
||||
FROM core.outbox;
|
||||
|
||||
-- name: ListSubscriptionsWithRecentReconcileEntries :many
|
||||
-- The listed subscriptions that already have a reconcile_subscription
|
||||
-- entry in flight (pending, processing or failed) or dead-lettered within the
|
||||
-- last day, so the collection drift sweep does not ask again for one the
|
||||
-- outbox is still working on or has just given up (grant-pauses-
|
||||
-- subscription-billing D5). A dead letter older than a day is asked for
|
||||
-- again.
|
||||
SELECT DISTINCT (payload->>'subscription_id')::text AS subscription_id
|
||||
FROM core.outbox
|
||||
WHERE provider = 'stripe'
|
||||
AND action_type = 'reconcile_subscription'
|
||||
AND payload->>'subscription_id' = ANY(sqlc.arg(subscription_ids)::text[])
|
||||
AND (status IN ('pending', 'processing', 'failed')
|
||||
OR (status = 'dead_letter' AND updated_at > NOW() - interval '1 day'))
|
||||
ORDER BY 1;
|
||||
|
||||
@@ -111,6 +111,14 @@ type Querier interface {
|
||||
// row from the other environment costs no call; a NULL key_livemode (the
|
||||
// key's mode is unknown) limits nothing. Never-read rows first.
|
||||
ListInvoicesToReadBack(ctx context.Context, arg ListInvoicesToReadBackParams) ([]string, error)
|
||||
// One page of the live subscriptions (active, trialing, past_due, unpaid and
|
||||
// paused) that have a Stripe mapping with a Stripe id, by subscription id after
|
||||
// the keyset, with the mirror of their collection pause and the mapping's
|
||||
// environment (grant-pauses-subscription-billing D5). The join is inner, and
|
||||
// requires the Stripe id, because the reconcile it asks for skips a mapping
|
||||
// with no Stripe id, so such a subscription would drift every hour. The store
|
||||
// holds this query because billing's sqlc does not load the stripe schema.
|
||||
ListLiveSubscriptionsForCollectionSweep(ctx context.Context, arg ListLiveSubscriptionsForCollectionSweepParams) ([]ListLiveSubscriptionsForCollectionSweepRow, error)
|
||||
// The payments recorded against one Stripe payment method, which
|
||||
// payment_method.attached links once it maps the method
|
||||
// (stripe-invoice-reconcile D7).
|
||||
|
||||
@@ -39,3 +39,26 @@ WHERE subscription_id = $1;
|
||||
-- under either key, so it is not in this list.
|
||||
SELECT subscription_id FROM stripe.subscription_mappings
|
||||
WHERE livemode IS NOT NULL AND livemode <> sqlc.arg(key_livemode)::boolean;
|
||||
|
||||
-- name: ListLiveSubscriptionsForCollectionSweep :many
|
||||
-- One page of the live subscriptions (active, trialing, past_due, unpaid and
|
||||
-- paused) that have a Stripe mapping with a Stripe id, by subscription id after
|
||||
-- the keyset, with the mirror of their collection pause and the mapping's
|
||||
-- environment (grant-pauses-subscription-billing D5). The join is inner, and
|
||||
-- requires the Stripe id, because the reconcile it asks for skips a mapping
|
||||
-- with no Stripe id, so such a subscription would drift every hour. The store
|
||||
-- holds this query because billing's sqlc does not load the stripe schema.
|
||||
SELECT s.subscription_id::uuid AS subscription_id,
|
||||
s.status,
|
||||
s.pause_collection_behavior,
|
||||
s.pause_collection_resumes_at,
|
||||
s.pause_collection_marked,
|
||||
m.livemode,
|
||||
m.sync_status
|
||||
FROM core.subscriptions s
|
||||
JOIN stripe.subscription_mappings m
|
||||
ON m.subscription_id = s.subscription_id AND m.stripe_subscription_id IS NOT NULL
|
||||
WHERE s.status IN ('active', 'trialing', 'past_due', 'unpaid', 'paused')
|
||||
AND s.subscription_id > sqlc.arg(after_subscription_id)::uuid
|
||||
ORDER BY s.subscription_id
|
||||
LIMIT sqlc.arg(page_size)::int;
|
||||
|
||||
@@ -53,6 +53,76 @@ func (q *Queries) GetSubscriptionMappingBySubscriptionID(ctx context.Context, su
|
||||
return i, err
|
||||
}
|
||||
|
||||
const listLiveSubscriptionsForCollectionSweep = `-- name: ListLiveSubscriptionsForCollectionSweep :many
|
||||
SELECT s.subscription_id::uuid AS subscription_id,
|
||||
s.status,
|
||||
s.pause_collection_behavior,
|
||||
s.pause_collection_resumes_at,
|
||||
s.pause_collection_marked,
|
||||
m.livemode,
|
||||
m.sync_status
|
||||
FROM core.subscriptions s
|
||||
JOIN stripe.subscription_mappings m
|
||||
ON m.subscription_id = s.subscription_id AND m.stripe_subscription_id IS NOT NULL
|
||||
WHERE s.status IN ('active', 'trialing', 'past_due', 'unpaid', 'paused')
|
||||
AND s.subscription_id > $1::uuid
|
||||
ORDER BY s.subscription_id
|
||||
LIMIT $2::int
|
||||
`
|
||||
|
||||
type ListLiveSubscriptionsForCollectionSweepParams struct {
|
||||
AfterSubscriptionID string `json:"after_subscription_id"`
|
||||
PageSize int32 `json:"page_size"`
|
||||
}
|
||||
|
||||
type ListLiveSubscriptionsForCollectionSweepRow struct {
|
||||
SubscriptionID string `json:"subscription_id"`
|
||||
Status string `json:"status"`
|
||||
PauseCollectionBehavior sql.NullString `json:"pause_collection_behavior"`
|
||||
PauseCollectionResumesAt sql.NullTime `json:"pause_collection_resumes_at"`
|
||||
PauseCollectionMarked bool `json:"pause_collection_marked"`
|
||||
Livemode sql.NullBool `json:"livemode"`
|
||||
SyncStatus string `json:"sync_status"`
|
||||
}
|
||||
|
||||
// One page of the live subscriptions (active, trialing, past_due, unpaid and
|
||||
// paused) that have a Stripe mapping with a Stripe id, by subscription id after
|
||||
// the keyset, with the mirror of their collection pause and the mapping's
|
||||
// environment (grant-pauses-subscription-billing D5). The join is inner, and
|
||||
// requires the Stripe id, because the reconcile it asks for skips a mapping
|
||||
// with no Stripe id, so such a subscription would drift every hour. The store
|
||||
// holds this query because billing's sqlc does not load the stripe schema.
|
||||
func (q *Queries) ListLiveSubscriptionsForCollectionSweep(ctx context.Context, arg ListLiveSubscriptionsForCollectionSweepParams) ([]ListLiveSubscriptionsForCollectionSweepRow, error) {
|
||||
rows, err := q.db.QueryContext(ctx, listLiveSubscriptionsForCollectionSweep, arg.AfterSubscriptionID, arg.PageSize)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
items := []ListLiveSubscriptionsForCollectionSweepRow{}
|
||||
for rows.Next() {
|
||||
var i ListLiveSubscriptionsForCollectionSweepRow
|
||||
if err := rows.Scan(
|
||||
&i.SubscriptionID,
|
||||
&i.Status,
|
||||
&i.PauseCollectionBehavior,
|
||||
&i.PauseCollectionResumesAt,
|
||||
&i.PauseCollectionMarked,
|
||||
&i.Livemode,
|
||||
&i.SyncStatus,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, i)
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
const listSubscriptionIDsOutsideMode = `-- name: ListSubscriptionIDsOutsideMode :many
|
||||
SELECT subscription_id FROM stripe.subscription_mappings
|
||||
WHERE livemode IS NOT NULL AND livemode <> $1::boolean
|
||||
|
||||
@@ -648,7 +648,7 @@ func subscriptionTitlesFor(subs []billing.ListSubscriptionsPageRow, items []bill
|
||||
for i, r := range rows {
|
||||
converted[i] = entitlements.ListAccountSubscriptionDisplacementsRow(r)
|
||||
}
|
||||
byID := displacedByID(refs, productsBySubscription(items), entitlements.RungsByProduct(shapes), displacementsBySubscription(converted))
|
||||
byID := displacedByID(refs, productsBySubscription(items), entitlements.RungsByProduct(shapes), entitlements.DisplacementsBySubscription(converted))
|
||||
titles := make(map[string]string, len(byID))
|
||||
for id, d := range byID {
|
||||
titles[id] = displacementTitle(1, []displacedSubscription{d})
|
||||
|
||||
@@ -1899,23 +1899,6 @@ func displacedByID(subs []subscriptionRef, products map[string][]string, rungs m
|
||||
return out
|
||||
}
|
||||
|
||||
// displacementsBySubscription is the one mapping of a displacement row's
|
||||
// fields to entitlements.Displacement, grouped by subscription. It reads no
|
||||
// row.
|
||||
func displacementsBySubscription(rows []entitlements.ListAccountSubscriptionDisplacementsRow) map[string][]entitlements.Displacement {
|
||||
by := make(map[string][]entitlements.Displacement)
|
||||
for _, d := range rows {
|
||||
by[d.SubscriptionID] = append(by[d.SubscriptionID], entitlements.Displacement{
|
||||
LadderID: d.PlanLadderID,
|
||||
LadderName: d.LadderName,
|
||||
DisplacedProductID: d.DisplacedProductID,
|
||||
HolderProvisionID: d.HolderProvisionID,
|
||||
HolderSource: d.HolderSource,
|
||||
})
|
||||
}
|
||||
return by
|
||||
}
|
||||
|
||||
// itemRow is the shape both current-item row types share.
|
||||
type itemRow interface {
|
||||
billing.ListCurrentSubscriptionItemsByBillingAccountIDRow | billing.ListCurrentSubscriptionItemsBySubscriptionIDsRow
|
||||
@@ -1940,7 +1923,7 @@ func displacementTitleFor(subs []billing.Subscription, items []billing.ListCurre
|
||||
for i, s := range subs {
|
||||
refs[i] = subscriptionRef{ID: s.SubscriptionID, Status: s.Status, PauseBehavior: s.PauseCollectionBehavior.String}
|
||||
}
|
||||
byID := displacedByID(refs, productsBySubscription(items), entitlements.RungsByProduct(shapes), displacementsBySubscription(rows))
|
||||
byID := displacedByID(refs, productsBySubscription(items), entitlements.RungsByProduct(shapes), entitlements.DisplacementsBySubscription(rows))
|
||||
var displaced []displacedSubscription
|
||||
for _, ref := range refs {
|
||||
if d, ok := byID[ref.ID]; ok {
|
||||
|
||||
@@ -45,3 +45,21 @@ func (a *Activities) SweepScheduledChangesActivity(ctx context.Context) (SweepSc
|
||||
}
|
||||
return SweepScheduledChangesOutput{Fired: fired}, nil
|
||||
}
|
||||
|
||||
// SweepCollectionDriftOutput reports how many subscriptions the collection
|
||||
// drift sweep asked the outbox to reconcile.
|
||||
type SweepCollectionDriftOutput struct {
|
||||
Asked int
|
||||
}
|
||||
|
||||
// SweepCollectionDriftActivity asks for a reconcile of each live subscription
|
||||
// whose recorded collection pause disagrees with the rule. It reads the
|
||||
// database only and skips a subscription that already has an entry, so
|
||||
// re-running it is safe.
|
||||
func (a *Activities) SweepCollectionDriftActivity(ctx context.Context) (SweepCollectionDriftOutput, error) {
|
||||
asked, err := fulfillment.SweepCollectionDrift(ctx, a.db, a.logger, a.keyMode)
|
||||
if err != nil {
|
||||
return SweepCollectionDriftOutput{}, err
|
||||
}
|
||||
return SweepCollectionDriftOutput{Asked: asked}, nil
|
||||
}
|
||||
|
||||
@@ -13,16 +13,34 @@ import (
|
||||
// a period-end cancellation normally fires via Stripe's customer.subscription
|
||||
// .deleted webhook, but the sweep catches missed webhooks and term-boundary
|
||||
// ('block'-deferred) changes that Stripe does not fire on its own.
|
||||
//
|
||||
// After the scheduled changes it runs the collection drift sweep. Each
|
||||
// activity's failure is logged and the other still runs; the workflow returns
|
||||
// the first failing activity's error after both have run, so the scheduled run
|
||||
// records the failure.
|
||||
func SweepScheduledChangesWorkflow(ctx workflow.Context) error {
|
||||
logger := workflow.GetLogger(ctx)
|
||||
ctx = workflow.WithActivityOptions(ctx, common.DefaultActivityOptions())
|
||||
|
||||
var activities *Activities
|
||||
var firstErr error
|
||||
|
||||
var out SweepScheduledChangesOutput
|
||||
if err := workflow.ExecuteActivity(ctx, activities.SweepScheduledChangesActivity).Get(ctx, &out); err != nil {
|
||||
logger.Error("SweepScheduledChangesActivity failed", "error", err)
|
||||
return err
|
||||
firstErr = err
|
||||
} else {
|
||||
logger.Info("SweepScheduledChangesWorkflow fired scheduled changes", "fired", out.Fired)
|
||||
}
|
||||
logger.Info("SweepScheduledChangesWorkflow completed", "fired", out.Fired)
|
||||
return nil
|
||||
|
||||
var drift SweepCollectionDriftOutput
|
||||
if err := workflow.ExecuteActivity(ctx, activities.SweepCollectionDriftActivity).Get(ctx, &drift); err != nil {
|
||||
logger.Error("SweepCollectionDriftActivity failed", "error", err)
|
||||
if firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
} else {
|
||||
logger.Info("SweepScheduledChangesWorkflow asked for reconciles", "asked", drift.Asked)
|
||||
}
|
||||
return firstErr
|
||||
}
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
// SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Commercial
|
||||
// SPDX-FileCopyrightText: 2025-2026 Christian Galo
|
||||
|
||||
package billing_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/mock"
|
||||
"go.temporal.io/sdk/temporal"
|
||||
"go.temporal.io/sdk/testsuite"
|
||||
|
||||
wfBilling "git.coopcloud.tech/wiki-cafe/member-console/internal/workflows/billing"
|
||||
)
|
||||
|
||||
// runSweepWorkflow runs SweepScheduledChangesWorkflow with the two activities
|
||||
// answering as given, and reports whether each ran and the workflow's error.
|
||||
func runSweepWorkflow(t *testing.T, scheduledErr, driftErr error) (scheduledRan, driftRan bool, err error) {
|
||||
t.Helper()
|
||||
var suite testsuite.WorkflowTestSuite
|
||||
env := suite.NewTestWorkflowEnvironment()
|
||||
var acts *wfBilling.Activities
|
||||
env.OnActivity(acts.SweepScheduledChangesActivity, mock.Anything).Return(
|
||||
func(context.Context) (wfBilling.SweepScheduledChangesOutput, error) {
|
||||
scheduledRan = true
|
||||
return wfBilling.SweepScheduledChangesOutput{}, scheduledErr
|
||||
})
|
||||
env.OnActivity(acts.SweepCollectionDriftActivity, mock.Anything).Return(
|
||||
func(context.Context) (wfBilling.SweepCollectionDriftOutput, error) {
|
||||
driftRan = true
|
||||
return wfBilling.SweepCollectionDriftOutput{}, driftErr
|
||||
})
|
||||
|
||||
env.ExecuteWorkflow(wfBilling.SweepScheduledChangesWorkflow)
|
||||
|
||||
if !env.IsWorkflowCompleted() {
|
||||
t.Fatal("the sweep workflow did not finish")
|
||||
}
|
||||
return scheduledRan, driftRan, env.GetWorkflowError()
|
||||
}
|
||||
|
||||
// A failed scheduled-change sweep does not stop the collection sweep, and the
|
||||
// run still ends with the first failure, so the scheduled run records it.
|
||||
func TestSweepScheduledChangesWorkflowRunsTheCollectionSweepAfterAFailure(t *testing.T) {
|
||||
failure := temporal.NewNonRetryableApplicationError("scheduled changes refused", "test", errors.New("refused"))
|
||||
|
||||
_, driftRan, err := runSweepWorkflow(t, failure, nil)
|
||||
|
||||
if !driftRan {
|
||||
t.Error("the collection sweep did not run after a failed scheduled-change sweep")
|
||||
}
|
||||
if err == nil {
|
||||
t.Error("the workflow ended without the failed activity's error")
|
||||
}
|
||||
}
|
||||
|
||||
// A failed collection sweep after a good scheduled-change sweep fails the run.
|
||||
func TestSweepScheduledChangesWorkflowReturnsTheCollectionSweepsFailure(t *testing.T) {
|
||||
failure := temporal.NewNonRetryableApplicationError("collection sweep refused", "test", errors.New("refused"))
|
||||
|
||||
scheduledRan, driftRan, err := runSweepWorkflow(t, nil, failure)
|
||||
|
||||
if !scheduledRan || !driftRan {
|
||||
t.Errorf("the activities ran = %v, %v, want both", scheduledRan, driftRan)
|
||||
}
|
||||
if err == nil {
|
||||
t.Error("the workflow ended without the collection sweep's error")
|
||||
}
|
||||
}
|
||||
@@ -130,10 +130,10 @@ Sections 1 to 12 are one commit each, and their items are that commit's parts; s
|
||||
|
||||
## 6. An hourly sweep asks again for a subscription whose pause disagrees with the rule
|
||||
|
||||
- [ ] 6.1 Queries (design D5): `ListLiveSubscriptionsForCollectionSweep` (`internal/integrations/stripe/store/queries/subscription_mappings.sql`, Go package `stripe`, which `fulfillment` imports as `internalstripe`, because billing's sqlc does not load the Stripe store's migrations, `internal/billing/sqlc.yaml:4-6`, and the store's sqlc loads both streams, `internal/integrations/stripe/store/sqlc.yaml:4-6`; `subscription_id`, `status`, the three mirror columns, and the mapping's `livemode` and `sync_status`; joined to `stripe.subscription_mappings`, so an unmapped subscription never appears; the statuses `active`, `trialing`, `past_due`, `unpaid` and `paused`; `subscription_id > $1` keyset, `LIMIT $2`, ordered by id; the join also requires `m.stripe_subscription_id IS NOT NULL`, because `ApplySubscriptionCollection` skips a mapping with no Stripe id with L3 (3.3), so such a subscription would drift every hour; its generated code lands in `internal/integrations/stripe/store/subscription_mappings.sql.go`, which every break of the query edits); `ListCurrentConferringProductsBySubscriptionIDs` (billing, new: the subscription id and product id of each current item, `ended_at IS NULL`, whose product's `entitlement_set_id IS NOT NULL`, for the listed ids, ordered by subscription id and product id; the sweep reads it in place of an items read, and `ListCurrentSubscriptionItemsBySubscriptionIDs` stays as it is, because a column added to it would break `productsBySubscription`'s row-type conversion, `internal/server/operator_enrollment.go:1847-1853`); `ListLiveProvisionProductsBySubscriptionIDs` (entitlements; subscription id and product id of non-ended provisions); `ListSubscriptionsWithRecentReconcileEntries` (`internal/integration/queries/outbox.sql`; `payload->>'subscription_id'` of `reconcile_subscription` entries whose status is `pending`, `processing` or `failed`, or `dead_letter` with `updated_at > NOW() - interval '1 day'`, for the listed ids). `make sqlc-generate`.
|
||||
- [ ] 6.2 `internal/fulfillment/collection_sweep.go`: `collectionDrift(subs, products, live, rungs, displacements) []string`, reading no row, passes every recorded conferring product of a subscription (6.1's `ListCurrentConferringProductsBySubscriptionIDs`) as `desired` to `CollectionPauseWanted`, with `left` the map `DisplacedProducts` returns over those with no live provision, as `convergeConferrals` builds it, and compares the mirror (wanted and no behavior, or not wanted and the marker set, drift; a foreign pause never does), returning the drifting ids in id order. `SweepCollectionDrift(ctx, db, logger, keyMode) (int, error)` calls the unexported `sweepCollectionDrift(ctx, db, logger, keyMode, pageSize int, reads sweepReads)` with 500 and the real queriers; `sweepReads` holds the sweep's reads (the page, the conferring products, the live provisions, the shapes, the displacements and the recent entries), so the database tests can set a page size and refuse a read. It pages, drops the rows whose mapping fails `MappingReachable` under the key, reads once per page, drops ids with a recent entry, and writes one entry per drifting id in one transaction per page; L5 when the count is above zero, L6 and the error on a failed read. The server's `displacementsBySubscription` (`internal/server/operator_enrollment.go:1827-1839`), which converts the displacement rows per subscription, moves to `entitlements.DisplacementsBySubscription`, which the server and the sweep both call, because `fulfillment` cannot call the server's.
|
||||
- [ ] 6.3 `internal/workflows/billing/activities.go`: `SweepCollectionDriftActivity` calls it with `a.keyMode`; `sweep_workflow.go`: `SweepScheduledChangesWorkflow` runs it after `SweepScheduledChangesActivity`, each activity's failure logged and the other still run, and the workflow returns the first failing activity's error after both have run, so the scheduled run records the failure; the activity is registered where the first one is.
|
||||
- [ ] 6.4 Tests:
|
||||
- [x] 6.1 Queries (design D5): `ListLiveSubscriptionsForCollectionSweep` (`internal/integrations/stripe/store/queries/subscription_mappings.sql`, Go package `stripe`, which `fulfillment` imports as `internalstripe`, because billing's sqlc does not load the Stripe store's migrations, `internal/billing/sqlc.yaml:4-6`, and the store's sqlc loads both streams, `internal/integrations/stripe/store/sqlc.yaml:4-6`; `subscription_id`, `status`, the three mirror columns, and the mapping's `livemode` and `sync_status`; joined to `stripe.subscription_mappings`, so an unmapped subscription never appears; the statuses `active`, `trialing`, `past_due`, `unpaid` and `paused`; `subscription_id > $1` keyset, `LIMIT $2`, ordered by id; the join also requires `m.stripe_subscription_id IS NOT NULL`, because `ApplySubscriptionCollection` skips a mapping with no Stripe id with L3 (3.3), so such a subscription would drift every hour; its generated code lands in `internal/integrations/stripe/store/subscription_mappings.sql.go`, which every break of the query edits); `ListCurrentConferringProductsBySubscriptionIDs` (billing, new: the subscription id and product id of each current item, `ended_at IS NULL`, whose product's `entitlement_set_id IS NOT NULL`, for the listed ids, ordered by subscription id and product id; the sweep reads it in place of an items read, and `ListCurrentSubscriptionItemsBySubscriptionIDs` stays as it is, because a column added to it would break `productsBySubscription`'s row-type conversion, `internal/server/operator_enrollment.go:1847-1853`); `ListLiveProvisionProductsBySubscriptionIDs` (entitlements; subscription id and product id of non-ended provisions); `ListSubscriptionsWithRecentReconcileEntries` (`internal/integration/queries/outbox.sql`; `payload->>'subscription_id'` of `reconcile_subscription` entries whose status is `pending`, `processing` or `failed`, or `dead_letter` with `updated_at > NOW() - interval '1 day'`, for the listed ids). `make sqlc-generate`.
|
||||
- [x] 6.2 `internal/fulfillment/collection_sweep.go`: `collectionDrift(subs, products, live, rungs, displacements) []string`, reading no row, passes every recorded conferring product of a subscription (6.1's `ListCurrentConferringProductsBySubscriptionIDs`) as `desired` to `CollectionPauseWanted`, with `left` the map `DisplacedProducts` returns over those with no live provision, as `convergeConferrals` builds it, and compares the mirror (wanted and no behavior, or not wanted and the marker set, drift; a foreign pause never does), returning the drifting ids in id order. `SweepCollectionDrift(ctx, db, logger, keyMode) (int, error)` calls the unexported `sweepCollectionDrift(ctx, db, logger, keyMode, pageSize int, reads sweepReads)` with 500 and the real queriers; `sweepReads` holds the sweep's reads (the page, the conferring products, the live provisions, the shapes, the displacements and the recent entries), so the database tests can set a page size and refuse a read. It pages, drops the rows whose mapping fails `MappingReachable` under the key, reads once per page, drops ids with a recent entry, and writes one entry per drifting id in one transaction per page; L5 when the count is above zero, L6 and the error on a failed read. The server's `displacementsBySubscription` (`internal/server/operator_enrollment.go:1827-1839`), which converts the displacement rows per subscription, moves to `entitlements.DisplacementsBySubscription`, which the server and the sweep both call, because `fulfillment` cannot call the server's.
|
||||
- [x] 6.3 `internal/workflows/billing/activities.go`: `SweepCollectionDriftActivity` calls it with `a.keyMode`; `sweep_workflow.go`: `SweepScheduledChangesWorkflow` runs it after `SweepScheduledChangesActivity`, each activity's failure logged and the other still run, and the workflow returns the first failing activity's error after both have run, so the scheduled run records the failure; the activity is registered where the first one is.
|
||||
- [x] 6.4 Tests:
|
||||
- `TestCollectionDrift` (`internal/fulfillment/collection_sweep_test.go`, no database), rows: `wanted, not recorded` (drifts); `wanted, recorded` (no); `not wanted, the marker recorded` (drifts); `paused, the marker recorded` (drifts); `not wanted, a foreign pause` (no); `displaced by a subscription` (no); `partly displaced, nothing recorded` (no); and the drifting ids fed in two orders. Failure text: `collectionDrift(%s) = %v, want %v`.
|
||||
- `TestSweepCollectionDrift_AsksForADisplacedSubscription` (database): a grant over a subscription with no recorded pause and no entry; one entry. Failure text: `the sweep wrote %d entries for the displaced subscription, want 1`.
|
||||
- `TestSweepCollectionDrift_SkipsASubscriptionWithAnEntry`, rows `a pending entry`, `a dead letter an hour old` and `a dead letter two days old` (its `updated_at` set by the test), each the same fixture with that entry; entries 0, 0, 1. Failure text: `with %s, the sweep wrote %d entries, want %d`.
|
||||
@@ -142,8 +142,8 @@ Sections 1 to 12 are one commit each, and their items are that commit's parts; s
|
||||
- `TestSweepCollectionDrift_ASetLessItemRidesThePause`: a displaced tier and an item whose product has no entitlement set, recorded paused with the marker; the sweep writes no entry. Failure text: `the sweep wrote %d entries for a paused subscription with a set-less item, want 0`.
|
||||
- `TestSweepCollectionDrift_SkipsAnUnmappedSubscription`, rows `no mapping` (a displaced subscription with no Stripe mapping) and `a mapping for the other key mode`; for either, the sweep returns nil and writes no entry. Failure texts: `with %s, the sweep returned %v`; `with %s, the sweep wrote %d entries, want 0`.
|
||||
- `TestSweepScheduledChangesWorkflowRunsTheCollectionSweepAfterAFailure` (`internal/workflows/billing/sweep_workflow_test.go`, new, Temporal's `testsuite`) asserts both activities run and that a failure of the first still runs the second. Failure text: `the collection sweep did not run after a failed scheduled-change sweep`.
|
||||
- [ ] 6.5 Docs: `docs/models/payments-billing.md`, the Reconciler row, names the hourly drift sweep and the dead letter's one-day wait.
|
||||
- [ ] 6.6 Breaks:
|
||||
- [x] 6.5 Docs: `docs/models/payments-billing.md`, the Reconciler row, names the hourly drift sweep and the dead letter's one-day wait.
|
||||
- [x] 6.6 Breaks:
|
||||
- the foreign pause counted as drift: caught by `TestCollectionDrift`, `want_new` `collectionDrift(not wanted, a foreign pause) =`;
|
||||
- `desired` narrowed to the missing products: caught by `TestCollectionDrift`, `want_new` `collectionDrift(partly displaced, nothing recorded) =`;
|
||||
- the set filter dropped from the generated `ListCurrentConferringProductsBySubscriptionIDs` constant: caught by `TestSweepCollectionDrift_ASetLessItemRidesThePause`, `want_new` `the sweep wrote 1 entries for a paused subscription with a set-less item`;
|
||||
@@ -153,7 +153,7 @@ Sections 1 to 12 are one commit each, and their items are that commit's parts; s
|
||||
- the `MappingReachable` filter deleted: caught by `TestSweepCollectionDrift_SkipsAnUnmappedSubscription`, `want_new` `with a mapping for the other key mode, the sweep wrote 1 entries, want 0`;
|
||||
- the sweep's join made a left join (`JOIN stripe.subscription_mappings` replaced by `LEFT JOIN stripe.subscription_mappings` in the generated `ListLiveSubscriptionsForCollectionSweep` constant, `internal/integrations/stripe/store/subscription_mappings.sql.go`): caught by the same test's `no mapping` row, `want_new` `converting NULL to string is unsupported`. The unmapped row then arrives with a NULL `livemode`, which `MappingReachable` passes (`internal/integrations/stripe/store/mapping_environment.go:70-100`), and a NULL `sync_status`, which is `NOT NULL` in the table and so is generated as a `string`; the page's scan fails, and the row prints the scan error through `with no mapping, the sweep returned`;
|
||||
- the workflow returning on the first activity's failure: caught by the workflow test, `want_new` `the collection sweep did not run`.
|
||||
- [ ] 6.7 `make screens`: no change.
|
||||
- [x] 6.7 `make screens`: no change.
|
||||
|
||||
## 7. The act that ends a holder delivers the subscription again (#270)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user