member-console/design/documents/reference-audit-log-architecture.md

PostgreSQL audit log architecture for cooperative SaaS

A multi-tenant cooperative SaaS platform needs an audit log that survives 20+ years, handles 500K–50M events annually, and satisfies IRC §6001, Subchapter T, GDPR, and SOC 2 requirements simultaneously. The architecture below resolves five interlocking design areas — partitioning, retention, indexing, write throughput, and archival — into a single coherent system built on PostgreSQL declarative partitioning, pg_partman automation, and a Parquet-based cold storage tier. Every recommendation includes production-ready DDL.

The central insight binding these five areas together: monthly RANGE partitioning on created_at is the keystone decision because it determines index granularity (per-partition indexes stay small), enables O(1) retention enforcement (drop a partition, not millions of rows), provides the archival unit (detach → export → drop), and keeps query planning overhead under 3 ms even at 240 partitions across 20 years.

---

1. The core schema and partitioning strategy

Partition by created_at timestamp using monthly RANGE partitions. This approach wins over UUIDv7-based partitioning because of full pg_partman compatibility, natural query ergonomics (WHERE created_at BETWEEN ...), and proven partition pruning without helper functions. UUIDv7 remains the primary key type for its sequential insert performance — correlation ≈ 1.0 means minimal B-tree page splits — but it participates in a composite PK alongside the partition key.

The partition granularity choice is driven by hard numbers. PostgreSQL's planner checks all partition bounds during planning, and this cost grows linearly: 100 partitions ≈ 1.3 ms, 500 partitions ≈ 6.2 ms, 1000 partitions ≈ 12.4 ms. Monthly granularity produces 240 partitions over 20 years — roughly 3 ms planning overhead, well within OLTP tolerance. Each monthly partition contains 42K rows at 500K/year or 4.2M rows at 50M/year, both ideal sizes for index performance and vacuum efficiency. Quarterly (80 partitions) is viable but loses retention granularity; yearly produces unmanageably large partitions at scale.

The composite primary key (id, created_at) is required because PostgreSQL enforces that unique constraints on partitioned tables must include all partition key columns. This means uniqueness is enforced per-partition, not globally. For audit logs, this is acceptable — UUIDv7 collision probability is negligible, and audit records are never referenced by foreign keys from other tables.

-- =============================================================
-- EXTENSIONS
-- =============================================================
CREATE SCHEMA IF NOT EXISTS partman;
CREATE EXTENSION IF NOT EXISTS pg_partman SCHEMA partman;
CREATE EXTENSION IF NOT EXISTS pg_cron;
CREATE EXTENSION IF NOT EXISTS pg_parquet;  -- for archival

-- =============================================================
-- CORE SCHEMA
-- =============================================================
CREATE SCHEMA IF NOT EXISTS audit;

-- Retention tier enum
CREATE TYPE audit.retention_tier AS ENUM (
    'critical',      -- Financial/patronage: 20 years
    'security',      -- Auth/access: 7 years
    'compliance',    -- SOC 2 evidence: 7 years
    'operational',   -- API calls, navigation: 1 year
    'debug'          -- Diagnostic: 90 days
);

-- =============================================================
-- PRIMARY AUDIT TABLE
-- =============================================================
CREATE TABLE audit.events (
    id            UUID        NOT NULL DEFAULT gen_random_uuid(),
    tenant_id     UUID        NOT NULL,
    actor_id      UUID,
    actor_type    TEXT        NOT NULL DEFAULT 'user',
    action        TEXT        NOT NULL,
    entity_type   TEXT        NOT NULL,
    entity_id     UUID,
    -- Materialized high-query fields (extracted from JSONB)
    from_status   TEXT,
    to_status     TEXT,
    ip_address    INET,
    -- Flexible payload
    changes       JSONB       DEFAULT '{}'::jsonb,
    metadata      JSONB       DEFAULT '{}'::jsonb,
    -- Classification
    tier          audit.retention_tier NOT NULL DEFAULT 'operational',
    severity      TEXT        NOT NULL DEFAULT 'info'
                  CHECK (severity IN ('critical','high','medium','low','info')),
    -- Timestamp (partition key)
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),

    PRIMARY KEY (id, created_at)
) PARTITION BY RANGE (created_at);

-- Append-only: disallow updates/deletes at the role level
REVOKE UPDATE, DELETE ON audit.events FROM app_role;

-- Fillfactor 100: no space reserved for updates (append-only optimization)
ALTER TABLE audit.events SET (
    fillfactor = 100,
    autovacuum_vacuum_scale_factor = 0.05,
    autovacuum_analyze_scale_factor = 0.02
);

Automated partition management with pg_partman

pg_partman is the battle-tested standard, compatible with AWS RDS, Aurora, Supabase, and self-hosted PostgreSQL. It pre-creates future partitions (the premake setting), handles retention via detach-or-drop, and uses a template table for properties that can't be inherited from the parent.

-- Register with pg_partman for monthly partitions
SELECT partman.create_parent(
    p_parent_table   := 'audit.events',
    p_control        := 'created_at',
    p_interval       := '1 month',
    p_premake        := 4,                -- create 4 months ahead
    p_start_partition := '2026-01-01'::text
);

-- Configure lifecycle behavior
UPDATE partman.part_config SET
    infinite_time_partitions = true,      -- never stop creating partitions
    retention          = '7 years',       -- default retention (overridden per-tier)
    retention_keep_table = true,          -- detach, don't drop (archival picks up)
    retention_keep_index = false          -- drop indexes on detached partitions
WHERE parent_table = 'audit.events';

-- Schedule maintenance (hourly)
SELECT cron.schedule('partman-maintenance', '0 * * * *',
    $$CALL partman.run_maintenance_proc()$$);

The retention_keep_table = true setting is critical: pg_partman detaches expired partitions rather than dropping them. This hands them off to the archival pipeline (Section 5) which exports to Parquet before dropping. The retention_keep_index = false immediately drops indexes on detached partitions, reclaiming space since they'll never be queried through PostgreSQL again.

---

2. Compliance-driven retention tiers and legal holds

Retention periods for a cooperative SaaS platform are driven by an unusual combination of tax law, cooperative-specific Subchapter T requirements, and data privacy regulations. The longest tail comes from cooperative patronage records: nonqualified written notices of allocation under IRC §§1381–1388 have no statutory deadline for redemption, meaning the underlying patronage transaction data may need to survive 15–20+ years.

Retention periods by regulatory framework

Framework	Trigger	Minimum	Safe harbor
IRC §6001 / §6501(a)	Standard assessment period	3 years from filing	7 years
IRC §6501(e)	>25% gross income omission	6 years from filing	7 years
IRC §6501(c)	Fraud or failure to file	Indefinite	Permanent for flagged records
Subchapter T — qualified notices	Patron includes in income at receipt	3–7 years from distribution	7 years
Subchapter T — nonqualified notices	Until redeemed + assessment period	Life of notice + 3–7 years	Life of notice + 7 years (often 15–20 years)
GDPR Article 17	Erasure request during retention	Art. 17(3)(b) exemption during legal obligation	Anonymize after obligation expires
SOC 2 Type II	Audit examination period	≥12-month audit window	1–2 years minimum

The key Subchapter T mechanism: when a cooperative issues nonqualified written notices of allocation (§1388(d)), it pays tax on those amounts immediately. When it later redeems those notices — potentially decades later — it takes a deduction under §1382(b)(2) and recalculates tax under §1383. Records must support the original patronage calculation, member transaction history, and redemption computation spanning the entire chain.

Tiered retention implementation

Rather than a single uniform retention period, classify events at write time and enforce different lifecycles per tier:

-- =============================================================
-- RETENTION POLICY TABLE
-- =============================================================
CREATE TABLE audit.retention_policies (
    tier              audit.retention_tier PRIMARY KEY,
    hot_duration      INTERVAL NOT NULL,     -- NVMe SSD, full indexes
    warm_duration     INTERVAL NOT NULL,     -- SATA SSD, reduced indexes
    cold_duration     INTERVAL NOT NULL,     -- Parquet in S3 Standard
    frozen_duration   INTERVAL,              -- S3 Glacier (NULL = indefinite)
    anonymize_pii     BOOLEAN NOT NULL DEFAULT false,
    description       TEXT
);

INSERT INTO audit.retention_policies VALUES
('critical',    '1 year',  '2 years',  '17 years', NULL,
    false, 'Financial/patronage: 20yr+ total, no anonymization'),
('security',    '90 days', '1 year',   '6 years',  NULL,
    true,  'Auth events: 7yr total, anonymize PII at cold transition'),
('compliance',  '1 year',  '2 years',  '4 years',  NULL,
    false, 'SOC 2 evidence: 7yr total'),
('operational', '30 days', '60 days',  '275 days',  NULL,
    true,  'API calls: 1yr total, anonymize at warm transition'),
('debug',       '30 days', '60 days',  NULL,        NULL,
    true,  'Diagnostic: 90 days total, then delete');

Legal hold mechanism

Legal holds must block all automated deletion, archival, and anonymization for data within scope. The duty to preserve arises when litigation is reasonably anticipated (established in Zubulake v. UBS Warburg, 2003) and supersedes all routine retention policies.

-- =============================================================
-- LEGAL HOLD TABLE
-- =============================================================
CREATE TABLE audit.legal_holds (
    id              BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    hold_name       TEXT NOT NULL,
    matter_ref      TEXT NOT NULL,           -- legal matter reference
    -- Scope (NULL = all)
    tenant_id       UUID,
    actor_id        UUID,
    entity_type     TEXT,
    entity_id       UUID,
    date_range_start TIMESTAMPTZ,
    date_range_end   TIMESTAMPTZ,
    -- Lifecycle
    status          TEXT NOT NULL DEFAULT 'active'
                    CHECK (status IN ('active', 'released')),
    created_by      TEXT NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    released_at     TIMESTAMPTZ,
    released_by     TEXT,
    notes           TEXT
);

CREATE INDEX idx_legal_holds_active
    ON audit.legal_holds (status) WHERE status = 'active';

-- =============================================================
-- CHECK FUNCTION: Is a partition under legal hold?
-- =============================================================
CREATE OR REPLACE FUNCTION audit.partition_has_legal_hold(
    p_start TIMESTAMPTZ,
    p_end   TIMESTAMPTZ,
    p_tenant_id UUID DEFAULT NULL
) RETURNS BOOLEAN LANGUAGE sql STABLE AS $$
    SELECT EXISTS (
        SELECT 1 FROM audit.legal_holds
        WHERE status = 'active'
          AND (date_range_start IS NULL OR date_range_start < p_end)
          AND (date_range_end IS NULL OR date_range_end > p_start)
          AND (tenant_id IS NULL OR tenant_id = p_tenant_id)
    );
$$;

GDPR anonymization approach

For audit records that have passed their legal obligation retention period, irreversible anonymization satisfies Article 17 (confirmed by GDPRHub: "The anonymisation of personal data is generally also considered a means to erase personal data"). The architecture separates PII from audit event structure:

-- Anonymization function for warm→cold transitions
CREATE OR REPLACE FUNCTION audit.anonymize_partition(
    partition_name TEXT
) RETURNS VOID LANGUAGE plpgsql AS $$
BEGIN
    -- Only anonymize if no legal hold
    IF audit.partition_has_legal_hold(
        (SELECT range_start FROM audit.archive_manifest
         WHERE partition_name = anonymize_partition.partition_name),
        (SELECT range_end FROM audit.archive_manifest
         WHERE partition_name = anonymize_partition.partition_name)
    ) THEN
        RAISE EXCEPTION 'Partition % is under legal hold', partition_name;
    END IF;

    EXECUTE format(
        'UPDATE %I SET
            actor_id = NULL,
            ip_address = NULL,
            metadata = metadata - ''{user_agent}'' - ''{session_id}''',
        partition_name
    );
END;
$$;

---

3. Indexing strategy optimized for write throughput

The indexing strategy must balance two competing demands: fast reads for compliance investigations and dashboards, and minimal write amplification on a high-volume append-only table. Every index added to the audit table degrades INSERT throughput — Percona's 2025 benchmarks show that going from 7 to 39 indexes reduces throughput by 58%. The target is 6 carefully chosen indexes with an estimated aggregate write overhead under 15%.

Three key insights drive the design. First, partition pruning already provides coarse time filtering — queries constrained to a 3-month window only touch 3 of 240 partitions. This means time columns belong at the end of composite indexes, not the beginning, since the partition boundary handles the first level of time narrowing. Second, BRIN indexes are 1000x smaller than B-tree for the created_at column on append-only data (24 KB vs 21 MB per million rows in Crunchy Data benchmarks), with negligible write overhead. Third, partial indexes on rare event types (failed logins, permission changes) cost almost nothing to maintain because fewer than 1% of rows match the predicate.

Why entity_type leads the composite index

For polymorphic entity references, the B-tree composite index places entity_type first because it's always an equality filter that narrows to a specific type before scanning entity IDs within that type. GitLab's documentation confirms this ordering is critical for polymorphic associations. Including tenant_id as the leading column ensures per-tenant isolation in every query.

When to materialize JSONB fields

Fields like from_status and to_status are extracted into dedicated columns rather than left in JSONB for three reasons: PostgreSQL has zero statistics on JSONB internals (it uses hardcoded 0.1% selectivity estimates, which Heap.io found can produce plans 2000x slower); JSONB duplicates key names in every row (2x storage overhead vs. columns per Heap.io benchmarks); and expression B-tree indexes on JSONB fields are fragile and less efficient than column indexes.

Broad GIN indexes on the changes JSONB column should be avoided. pganalyze found that GIN indexes on high-churn tables cause substantial write amplification and bloat. If ad-hoc JSONB search is needed, jsonb_path_ops is 2–3x smaller than the default GIN operator class and faster for containment queries.

-- =============================================================
-- INDEX 1: Tenant + entity lookup (primary query pattern)
-- =============================================================
-- Equality columns first, time last (partitioning handles coarse time pruning)
CREATE INDEX idx_audit_entity_lookup
    ON audit.events (tenant_id, entity_type, entity_id, created_at DESC);

-- =============================================================
-- INDEX 2: Actor lookup (who did what?)
-- =============================================================
CREATE INDEX idx_audit_actor
    ON audit.events (tenant_id, actor_id, created_at DESC);

-- =============================================================
-- INDEX 3: Action filtering (what happened?)
-- =============================================================
CREATE INDEX idx_audit_action
    ON audit.events (tenant_id, action, created_at DESC);

-- =============================================================
-- INDEX 4: BRIN on timestamp (broad time-range scans)
-- 1000x smaller than B-tree, near-zero write overhead
-- pages_per_range=32 covers ~1,280 rows per summary entry
-- =============================================================
CREATE INDEX idx_audit_time_brin
    ON audit.events USING BRIN (created_at)
    WITH (pages_per_range = 32);

-- =============================================================
-- INDEX 5: Security monitoring — failed auth (partial index)
-- Only indexes rows matching predicate (~<1% of total)
-- =============================================================
CREATE INDEX idx_audit_login_failures
    ON audit.events (tenant_id, actor_id, created_at DESC, ip_address)
    WHERE action = 'login_failed';

-- =============================================================
-- INDEX 6: Failure tracking (partial index)
-- =============================================================
CREATE INDEX idx_audit_failures
    ON audit.events (tenant_id, entity_type, entity_id, created_at DESC)
    WHERE severity IN ('critical', 'high');

-- =============================================================
-- NOT RECOMMENDED: Broad GIN on JSONB
-- Adds ~20-30% storage overhead and significant write amplification
-- Use targeted expression indexes instead if needed:
-- =============================================================
-- CREATE INDEX idx_audit_changes_gin
--     ON audit.events USING GIN (changes jsonb_path_ops);

-- Better: Expression index on a specific known field
-- CREATE INDEX idx_audit_change_type
--     ON audit.events ((changes->>'change_type'))
--     WHERE changes->>'change_type' IS NOT NULL;

Each of these indexes is automatically created per-partition when defined on the parent table (PostgreSQL 11+). Per-partition indexes remain small — at 4.2M rows/month, each B-tree index partition is roughly 90–130 MB — and benefit from better cache utilization. Dropping a partition instantly drops all its indexes with zero cleanup overhead.

---

4. Hybrid write throughput with transactional guarantees

The write architecture uses a hybrid approach: synchronous same-transaction inserts for governance-critical events, and buffered batch inserts for high-volume operational events. This isn't a performance compromise — it's a compliance requirement. Incomplete audit logs are explicitly listed as one of the most frequently identified gaps in SOC 2 audits, and async pipeline failures that cause gaps are treated the same as having no logging.

Synchronous path performance is not a bottleneck

CYBERTEC's PostgreSQL benchmarks show that a trigger-based audit INSERT adds only 4–12% latency to the parent transaction. For application-level inserts (the recommended approach for cooperative SaaS, since they carry rich domain context like user identity, endpoint, and cooperative ID), the overhead is a single additional INSERT in an already-open transaction — typically under 0.5 ms.

PostgreSQL INSERT throughput on append-only partitioned tables is substantial. CYBERTEC measured 714,000 rows/second via COPY and 200,000 rows/second via multi-row INSERT on PostgreSQL 16. Even single-row INSERTs achieve 12,600 rows/second at 10 parallel connections per Hatchet's 2024 benchmarks. A cooperative SaaS platform at 50M events/year averages only ~1.6 events/second — three orders of magnitude below capacity.

Why LISTEN/NOTIFY is unsuitable for audit

PostgreSQL LISTEN/NOTIFY has a hard 8,000-byte payload limit, provides at-most-once delivery with no persistence, and permanently loses messages when listeners are down. It should never be the delivery path for compliance-critical events. It can serve as a low-latency notification signal ("new events exist, go poll now") but not as a transport mechanism.

Transactional outbox for external consumers

When audit events need to reach external systems (SIEM, Kafka, analytics), the transactional outbox pattern solves the dual-write problem: the outbox INSERT occurs in the same transaction as the business operation, guaranteeing atomicity. A background relay process then forwards events to external consumers.

-- =============================================================
-- OUTBOX TABLE (for external event delivery)
-- =============================================================
CREATE TABLE audit.outbox (
    id            BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    event_id      UUID        NOT NULL,
    event_type    TEXT        NOT NULL,
    tenant_id     UUID        NOT NULL,
    aggregate_type TEXT       NOT NULL,
    aggregate_id  TEXT        NOT NULL,
    payload       JSONB       NOT NULL,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at  TIMESTAMPTZ           -- NULL until delivered
);

-- Efficient polling index (SKIP LOCKED pattern)
CREATE INDEX idx_outbox_unpublished
    ON audit.outbox (id)
    WHERE published_at IS NULL;

-- =============================================================
-- RELAY QUERY (run by background worker every 1-5 seconds)
-- =============================================================
-- Uses SKIP LOCKED for parallel relay workers without contention
-- WITH batch AS (
--     SELECT id FROM audit.outbox
--     WHERE published_at IS NULL
--     ORDER BY id
--     LIMIT 1000
--     FOR UPDATE SKIP LOCKED
-- )
-- UPDATE audit.outbox SET published_at = now()
-- WHERE id IN (SELECT id FROM batch)
-- RETURNING *;

-- =============================================================
-- APPLICATION-LEVEL AUDIT INSERT (pseudocode)
-- =============================================================
-- BEGIN;
--   -- 1. Business operation
--   UPDATE accounts SET balance = balance - 100 WHERE id = $1;
--
--   -- 2. Synchronous audit event (same transaction)
--   INSERT INTO audit.events (
--       tenant_id, actor_id, action, entity_type, entity_id,
--       from_status, to_status, tier, severity, changes, metadata
--   ) VALUES (
--       $tenant, $actor, 'account.debited', 'account', $1,
--       'active', 'active', 'critical', 'high',
--       '{"amount": 100, "currency": "USD"}'::jsonb,
--       jsonb_build_object('ip', $ip, 'endpoint', '/api/transfer')
--   );
--
--   -- 3. Outbox for Kafka/SIEM delivery (same transaction)
--   INSERT INTO audit.outbox (event_id, event_type, tenant_id,
--       aggregate_type, aggregate_id, payload)
--   VALUES (gen_random_uuid(), 'account.debited', $tenant,
--       'account', $1, $payload);
--
-- COMMIT;

-- =============================================================
-- ASYNC BATCH INSERT (for operational/debug events)
-- Buffer in application memory, flush every 500ms or 100 events
-- =============================================================
-- INSERT INTO audit.events (tenant_id, action, entity_type,
--     entity_id, tier, severity, metadata, created_at)
-- VALUES
--     ($1, $2, $3, $4, 'operational', 'info', $5, $6),
--     ($7, $8, $9, $10, 'operational', 'info', $11, $12),
--     ...  -- up to 1000 rows per batch
-- ;

Gap detection for async paths

Async audit pipelines require a compensating control: automated gap detection that reconciles business event counts against audit event counts. Run this hourly via pg_cron:

-- =============================================================
-- GAP DETECTION QUERY (scheduled hourly via pg_cron)
-- =============================================================
-- SELECT cron.schedule('audit-gap-check', '15 * * * *', $$
--     INSERT INTO audit.gap_alerts (hour, expected, actual, gap)
--     SELECT
--         date_trunc('hour', t.created_at) AS hour,
--         COUNT(t.id) AS expected,
--         COUNT(a.id) AS actual,
--         COUNT(t.id) - COUNT(a.id) AS gap
--     FROM business.transactions t
--     LEFT JOIN audit.events a
--         ON a.entity_type = 'transaction'
--         AND a.entity_id = t.id
--         AND a.action = 'transaction.created'
--     WHERE t.created_at >= now() - INTERVAL '2 hours'
--       AND t.created_at < now() - INTERVAL '1 hour'
--     GROUP BY 1
--     HAVING COUNT(t.id) != COUNT(a.id);
-- $$);

---

5. Archival pipeline from PostgreSQL to Parquet

The archival strategy transforms monthly partitions through four lifecycle tiers — hot, warm, cold, frozen — with concrete transition criteria and tooling at each stage. The critical enabling technology is pg_parquet (by Crunchy Data), which extends PostgreSQL's COPY command to natively export to Parquet format with zstd compression, achieving 5–10x compression versus PostgreSQL heap storage.

Lifecycle tiers with concrete criteria

Tier	Age	Storage	Format	Indexes	Recovery SLA	Estimated cost/GB/mo
Hot	0–90 days	NVMe SSD tablespace	PostgreSQL native	Full (6 indexes)	Immediate	~$0.10
Warm	91 days–1 year	SATA SSD tablespace	PostgreSQL native	Reduced (drop partial indexes)	< 5 minutes	~$0.05
Cold	1–7 years	S3 Standard	Parquet (zstd)	None (columnar predicate pushdown)	< 1 hour	~$0.023
Frozen	7–25+ years	S3 Glacier Deep Archive	Parquet (zstd)	None	< 24 hours	~$0.001

At 50M events/year with an estimated 500 bytes/row, annual PostgreSQL storage is ~25 GB. After Parquet+zstd compression (5–10x), annual cold storage is ~3–5 GB. Twenty years of frozen archive costs approximately $2.40/year on Glacier Deep Archive.

Tablespace setup for hot/warm tiers

-- =============================================================
-- TABLESPACES (configured by DBA, filesystem-level)
-- =============================================================
-- CREATE TABLESPACE ts_hot  LOCATION '/nvme_ssd/pg_audit';
-- CREATE TABLESPACE ts_warm LOCATION '/sata_ssd/pg_audit';

-- New partitions default to hot tablespace
-- (pg_partman template table controls this)
ALTER TABLE audit.events SET TABLESPACE ts_hot;

-- Move aging partition to warm (scheduled monthly via pg_cron)
-- ALTER TABLE audit.events_p2025_10 SET TABLESPACE ts_warm;
-- Note: SET TABLESPACE acquires ACCESS EXCLUSIVE lock; schedule during
-- low-traffic windows. For large partitions, this rewrites the table.

Archival workflow: detach → export → verify → drop

The archival pipeline runs monthly, processing partitions that have aged past the warm threshold. DETACH PARTITION CONCURRENTLY (PostgreSQL 14+) avoids blocking the parent table with only a SHARE UPDATE EXCLUSIVE lock.

-- =============================================================
-- ARCHIVE MANIFEST TABLE
-- =============================================================
CREATE TABLE audit.archive_manifest (
    id              BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    partition_name  TEXT NOT NULL,
    parent_table    TEXT NOT NULL DEFAULT 'audit.events',
    range_start     TIMESTAMPTZ NOT NULL,
    range_end       TIMESTAMPTZ NOT NULL,

    -- Archive location
    storage_tier    TEXT NOT NULL CHECK (storage_tier IN
                    ('hot','warm','cold','frozen')),
    file_path       TEXT,            -- s3://audit-archive/2023/01.parquet
    file_format     TEXT DEFAULT 'parquet',
    compression     TEXT DEFAULT 'zstd',
    file_size_bytes BIGINT,

    -- Integrity verification
    row_count       BIGINT NOT NULL,
    pg_row_count    BIGINT,          -- count before export
    checksum_sha256 TEXT,

    -- Lifecycle tracking
    status          TEXT NOT NULL DEFAULT 'active' CHECK (status IN (
        'active','warm','detached','exported',
        'verified','dropped','frozen','restore_in_progress')),
    detached_at     TIMESTAMPTZ,
    exported_at     TIMESTAMPTZ,
    verified_at     TIMESTAMPTZ,
    dropped_at      TIMESTAMPTZ,
    frozen_at       TIMESTAMPTZ,
    last_queried    TIMESTAMPTZ,

    -- Metadata
    managed_by      TEXT DEFAULT current_user,
    notes           TEXT,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now(),

    CONSTRAINT unique_partition
        UNIQUE (parent_table, partition_name),
    CONSTRAINT valid_range
        CHECK (range_end > range_start)
);

CREATE INDEX idx_manifest_range ON audit.archive_manifest
    USING GIST (tstzrange(range_start, range_end));
CREATE INDEX idx_manifest_status ON audit.archive_manifest (status);

-- =============================================================
-- ARCHIVAL PROCEDURE (run monthly for partitions aging past warm)
-- =============================================================
CREATE OR REPLACE PROCEDURE audit.archive_partition(
    p_partition_name TEXT,
    p_range_start    TIMESTAMPTZ,
    p_range_end      TIMESTAMPTZ
) LANGUAGE plpgsql AS $$
DECLARE
    v_row_count BIGINT;
    v_s3_path   TEXT;
BEGIN
    -- 1. Check legal hold
    IF audit.partition_has_legal_hold(p_range_start, p_range_end) THEN
        RAISE NOTICE 'Partition % under legal hold, skipping', p_partition_name;
        RETURN;
    END IF;

    -- 2. Count rows before detach
    EXECUTE format('SELECT count(*) FROM %I', p_partition_name)
        INTO v_row_count;

    -- 3. Record in manifest
    INSERT INTO audit.archive_manifest (
        partition_name, range_start, range_end,
        storage_tier, row_count, pg_row_count, status
    ) VALUES (
        p_partition_name, p_range_start, p_range_end,
        'cold', v_row_count, v_row_count, 'active'
    );

    -- 4. Detach (CONCURRENTLY cannot run inside txn block,
    --    so this procedure should be called from a wrapper script)
    -- ALTER TABLE audit.events
    --     DETACH PARTITION {p_partition_name} CONCURRENTLY;

    -- 5. Export to Parquet via pg_parquet
    v_s3_path := format('s3://audit-archive/%s/%s.parquet',
        to_char(p_range_start, 'YYYY'),
        p_partition_name);

    EXECUTE format(
        'COPY (SELECT * FROM %I ORDER BY created_at) '
        'TO %L WITH (format ''parquet'', compression ''zstd'')',
        p_partition_name, v_s3_path
    );

    -- 6. Update manifest with file path
    UPDATE audit.archive_manifest SET
        file_path = v_s3_path,
        status = 'exported',
        exported_at = now()
    WHERE partition_name = p_partition_name;

    -- 7. Verify (compare row count from parquet metadata)
    -- SELECT row_group_num_rows
    -- FROM parquet.metadata(v_s3_path);

    -- 8. After verification, drop the PostgreSQL table
    EXECUTE format('DROP TABLE %I', p_partition_name);

    UPDATE audit.archive_manifest SET
        status = 'dropped',
        dropped_at = now()
    WHERE partition_name = p_partition_name;
END;
$$;

Querying archived data with pg_duckdb

For cold/frozen data, pg_duckdb (by DuckDB Labs/MotherDuck) embeds the DuckDB vectorized engine inside PostgreSQL, enabling seamless joins between live PostgreSQL tables and Parquet files in S3. This is the recommended approach over parquet_fdw because DuckDB's columnar engine is purpose-built for analytical queries on Parquet and delivers dramatically faster scan performance.

-- =============================================================
-- QUERYING ARCHIVED DATA
-- =============================================================

-- Option A: pg_duckdb (recommended — vectorized, native S3 Parquet)
-- SET duckdb.force_execution = true;
-- SELECT * FROM read_parquet('s3://audit-archive/2023/*.parquet')
-- WHERE tenant_id = $1
--   AND created_at BETWEEN '2023-01-01' AND '2023-06-30';

-- Option B: parquet_fdw (simpler setup, slower)
-- CREATE EXTENSION parquet_fdw;
-- CREATE SERVER parquet_srv FOREIGN DATA WRAPPER parquet_fdw;
-- CREATE FOREIGN TABLE audit.events_2023_01_pq (
--     id UUID, tenant_id UUID, actor_id UUID, action TEXT,
--     entity_type TEXT, entity_id UUID, created_at TIMESTAMPTZ
-- ) SERVER parquet_srv
-- OPTIONS (filename 's3://audit-archive/2023/audit_events_p2023_01.parquet',
--          sorted 'created_at');
-- Note: A parquet foreign table can participate as a partition:
-- ALTER TABLE audit.events ATTACH PARTITION audit.events_2023_01_pq
--     FOR VALUES FROM ('2023-01-01') TO ('2023-02-01');

-- Option C: On-demand restoration (for forensic investigations)
-- CREATE TABLE audit.events_p2023_01_restored
--     (LIKE audit.events INCLUDING ALL) TABLESPACE ts_warm;
-- COPY audit.events_p2023_01_restored
--     FROM 's3://audit-archive/2023/audit_events_p2023_01.parquet'
--     WITH (format 'parquet');

-- Helper: Find which archive files cover a time range
CREATE OR REPLACE FUNCTION audit.find_archive_files(
    query_start TIMESTAMPTZ,
    query_end   TIMESTAMPTZ
) RETURNS TABLE (
    file_path    TEXT,
    storage_tier TEXT,
    row_count    BIGINT,
    status       TEXT
) LANGUAGE sql STABLE AS $$
    SELECT file_path, storage_tier, row_count, status
    FROM audit.archive_manifest
    WHERE tstzrange(range_start, range_end) &&
          tstzrange(query_start, query_end)
      AND status IN ('verified', 'dropped', 'frozen')
    ORDER BY range_start;
$$;

---

How all five areas interact as an integrated system

These five architectural areas form a tightly coupled system where each decision constrains the others. Understanding these interactions prevents contradictory choices.

Partitioning determines everything downstream. Monthly partitions are the unit of index management (per-partition indexes stay small), the unit of retention enforcement (detach a partition, not filter millions of rows), the unit of archival (one Parquet file per partition), and the unit of legal holds (hold scope maps to partition boundaries). Choosing quarterly partitions would reduce management overhead but produce Parquet files that are 3x larger and lose monthly retention granularity.

Retention tiers drive the archival pipeline. The tier column on each audit event determines when it transitions between hot/warm/cold/frozen. The archive procedure checks both the tier's retention policy and active legal holds before processing. Critical-tier events (patronage records) stay in PostgreSQL warm storage for 3 years before moving to cold Parquet, while debug-tier events are deleted after 90 days without archival.

Write throughput constrains index choices. Each additional B-tree index adds roughly 10–15% write overhead. The 6-index budget balances query needs against the ~50M events/year insertion rate. BRIN indexes for time columns add negligible overhead while providing adequate time-range filtering within partitions. The decision to materialize from_status/to_status as columns rather than relying on GIN-indexed JSONB trades storage simplicity for dramatically lower write amplification.

Synchronous writes justify the investment in per-partition indexes. Because governance-critical events are written synchronously in the same transaction, we can guarantee that every financial transaction has a corresponding audit record in the same partition. This means partition pruning plus B-tree indexes are always sufficient for compliance queries — no need for full-table scans or cross-partition joins.

Scale considerations from 500K to 50M events/year

Metric	500K/year	5M/year	50M/year
Events/second (avg)	0.016	0.16	1.6
Monthly partition rows	42K	420K	4.2M
Per-partition B-tree index size	~2 MB	~15 MB	~130 MB
Per-partition BRIN index size	~8 KB	~16 KB	~48 KB
Annual PostgreSQL storage	~250 MB	~2.5 GB	~25 GB
Annual Parquet archive (zstd)	~30 MB	~300 MB	~3 GB
20-year frozen archive total	~600 MB	~6 GB	~60 GB

At 500K/year, the entire system fits comfortably on a single small PostgreSQL instance. All indexes fit in shared_buffers. Archival is optional — 20 years of data is only 5 GB in PostgreSQL. At 50M/year, monthly partitions reach 4.2M rows and per-partition index sets total ~400 MB. This remains well within single-instance PostgreSQL capacity, but archival to Parquet becomes important for keeping the active dataset in memory. The async batch insert path (for operational events) should use multi-row INSERTs of 100–1000 rows per statement, achieving 30,000–90,000 rows/second — far above the 1.6/second average.

PostgreSQL version requirements

The architecture requires PostgreSQL 14 as minimum for DETACH PARTITION CONCURRENTLY and pg_partman 5.x compatibility. PostgreSQL 16+ is recommended for improved partition pruning with subqueries and bulk insert optimizations. PostgreSQL 18 is ideal — it adds native uuidv7(), improved planning efficiency for many partitions, and broader partitionwise join support.

---

Conclusion

The architecture resolves a fundamental tension in multi-decade audit logging: the system must be optimized for writes (append-only, high-throughput) while remaining queryable across regulatory timescales (20+ years). Monthly RANGE partitioning is the keystone that makes this tractable — it bounds index sizes, enables instant retention enforcement, provides natural archival units, and keeps query planning under 3 ms.

Three non-obvious insights emerged from this analysis. First, cooperative Subchapter T requirements create the longest retention tail — not tax law generally, but specifically the nonqualified written notice redemption mechanism that can span decades with no statutory deadline. Second, synchronous audit inserts are not the performance problem they're assumed to be: 4–12% overhead on transactions that already involve multiple writes is trivial compared to the compliance risk of audit gaps from async pipeline failures. Third, the BRIN-plus-partition-pruning combination makes the traditional B-tree-on-timestamp index unnecessary for time-range queries on audit tables — freeing one index slot for a more selective partial index that directly supports security monitoring.

The concrete transition to watch: when monthly Parquet files in S3 start accumulating past year 3, pg_duckdb becomes the primary analytical interface for cold data while PostgreSQL continues to serve hot/warm queries. This hybrid — PostgreSQL for transactional audit writes and recent queries, DuckDB-on-Parquet for historical analysis — is the architecture that actually scales to 20 years without ballooning storage costs or degrading write performance.