Files
cgalo5758 26d48168af Name the coming steps on a limit entry
Ruling 48 answers the field question of task 1.4. The list ruling 10
puts on a limit entry is coming_steps, each element with id and
starts_at as a Timestamp, in the key's step order. While the clock is
paused each element keeps its id and carries no starts_at, so a paused
clock and no coming step differ on the wire. Every field that holds a
moment ends in _at, as replaced_at already does.

next_steps reads as things to do, scheduled_steps invites a timer the
spec forbids, schedule is singular on a list, and applies_at tells a
connector to act on a date. Three behaviours the names cannot carry go
to #172's open questions.
2026-11-22 02:47:08 -06:00

126 KiB

1. Decisions and preparation

  • 1.1 Confirm the core's next migration number when implementation starts (00043 while 00042_limit_sign_and_hidden.sql is the latest) and name the three files of group 8 00043_connector_connections.sql, 00044_workspace_records.sql and 00045_delivery_triggers.sql accordingly; every reference in design.md, the specs and these tasks follows a renumbering.
  • 1.2 The maintainer decides where the specification repository lives and what it is called: its hosting location, its name, its Go module path, and its tag scheme. Until he decides, groups 2 to 7 stay unstarted; the console requires the repository's Go module at a pinned version and holds no copy of its definitions (protocol-specification). Ruled: design.md, rulings 43 and 44.
  • 1.3 The maintainer decides where the Discourse bridge's code lives (the specification repository's reference connectors, a directory of the console's repository, or a repository of its own) and which provider key the test stack registers for it; group 19 cannot start before he does. Ruled: design.md, ruling 47.
  • 1.4 Before group 2 starts, the names class's messages are designed and ruled (second review, item 1; design.md, Open Questions), and the maintainer names the limit entry's field for coming steps and dates (design.md, ruling 10), which 2.2 defines. The field is ruled: design.md, ruling 48.
  • 1.5 Counsel reviews contributor coverage for the specification repository before it is public (October rulings, item 9; the contributor agreement covers the console's repository only), and the maintainer decides what the new repository's own contribution terms say. No code of groups 2 to 7 is published before this passes.
  • 1.6 Before the specification repository goes public, a GitHub account that has never used the forge signs in to it and files an issue (design.md, ruling 43). If it cannot, the maintainer decides where outside authors report faults before the repository goes public.

2. Specification repository: definitions

  • 2.1 Create the repository (name and place of design.md, ruling 43): LICENSE Apache-2.0, NOTICE, SPDX headers on every source file, CONTRIBUTING.md and GOVERNANCE.md stating a public change process in which the console's project keeps the final say, buf.yaml and buf.gen.yaml, the directories proto/ (one directory per protocol version, v0_1 first), gen/openapi/, gen/jsonschema/, vectors/<version>/, spec/, helpers/go/, harness/ and sidecar/. Commit the generated Go code under the module path, in a directory per version, because go get cannot run buf. Give harness/ and sidecar/ each their own go.mod, split before the first tag. Have the README list which protocol versions each release holds (design.md, ruling 44). In the console's repository, add a test that fails when git ls-files '*.proto' is not empty, and require the specification's Go module in go.mod at one pinned version, so that no definition a build can change independently of the published version is copied into it.
  • 2.2 Define the shared messages in proto/ as package v0_1, with the per-file buf lint exception for PACKAGE_VERSION_SUFFIX of design.md, ruling 44: Recipient (kind as RecipientKind with PERSON and WORKSPACE, id, and for a workspace organization_id), Entry (kind as EntryKind with SWITCH and LIMIT, key, optional bool granted, optional int64 limit, plain strings over_limit and step, delete_after as a Protobuf Duration, and coming_steps, a repeated ComingStep with id and starts_at as a Timestamp, in the key's step order, with starts_at absent while the clock is paused (design.md, ruling 48)), Record (recipient, generation int64 of 1 or more, content_hash in lowercase hexadecimal, entitlements, and a person's identity items email and oidc_subject), each message carrying the field protocol_version. Every enum's number 0 is ..._UNSPECIFIED and means "not set" only. allowance and balance are named in the specification text as reserved entry kinds and are not enum values.
  • 2.3 Define the envelope and the operations in package v0_1: every body carries protocol_version, connection_id and op, and every answer carries in_reply_to; the operation names pair, pair_result, announce, announce_result, apply, apply_result, read_back, read_back_result, forget, forget_result, feed, feed_items, position_lost, feed_answer, feed_answer_result, full_read and full_read_page, each with the members its capability gives it: announce_result.outcome (staged, unchanged, refused, with the problem's class, the first field at fault and, for a version, the versions the console speaks), feed_items (position, items of id, op and the operation's fields), full_read_page (records, next_cursor, last written on every page, and under pull feed_position), read_back (cursor, limit) and read_back_result (items, next_cursor). A cursor, position and message identifier are strings of at most 256 characters from A-Z, a-z, 0-9, - and _.
  • 2.4 Define the Connect service that serves them in package v0_1, with the procedures registered at /connectors/pair, /connectors/manifest, /connectors/apply, /connectors/read_back, /connectors/forget, /connectors/feed, /connectors/feed/answers and /connectors/records through connect-go's NewUnaryHandler with those procedure strings, and the console's pairing path /connectors/pair/<connection identifier>. JSON over HTTP is the only binding of this version; the definition serves no gRPC or gRPC-Web (design.md, ruling 45).
  • 2.5 Define the answers: Outcome with applied, failed, unlinked, linked_elsewhere, stale, conflict and forgotten, plus key_replaced (with replaced_at) and code_proved_twice as the answers the console gives to a replaced key and to a second key's proof; apply_result and each read-back item carry recipient, applied_generation (0 allowed, meaning none), applied_content_hash where the connector keeps it, and entitlements whose entries carry granted or limit, held, restricted and locked as optional int64, and on an apply answer moved; next_action as a closed key; failed carries no reason and no text; a refusal for a wrong connection, operation or version carries no outcome.
  • 2.6 Define the manifest message with exactly the members of connector-manifest: protocol_version, provider, display_name, classes (closed list reports, lifecycle, usage, names, regions, sign_on, observe_only, adoption, export), max_read_back_page, identity (email, oidc_subject), keys (key, display_name, recipient, kind, and for a limit unit, enforced_by, order, steps, through; each step id, does, sentence, after, delete_after), on_forget (value delete or detach, sentence), and with the class lifecycle only, operations and states. Durations are Protobuf Durations written as decimal seconds with an s. Put the spec's sample manifest in spec/ and in the vectors.
  • 2.7 Define the messages of the optional classes that the rulings fix, so the first version holds every field a later change uses: lifecycle request and its answer with a closed outcome and a closed reason and no text (provider-lifecycle-contract), usage acquire, commit and release naming a lease, with the field name period reserved and absent (rulings 15c, 33 and 35), the report fields of ruling 15e, and region data with its value types (protocol-personal-data). Define no message for sign_on, adoption or export. Define the names class's messages once 1.4 rules them.
  • 2.8 Write the JSON spelling into the specification's text and the generated artifacts: snake_case names on the wire, an enum by its name, every 64-bit integer (limit, count, generation, sequence) as a JSON string, times and durations as Protobuf's JSON mapping gives them, the name of every field that holds a moment ending in _at (design.md, ruling 48), a reader that accepts either Protobuf field name and a sender that writes snake_case, and no reliance on a runtime's default for any of these (an EmitUnpopulated or UseProtoNames choice is stated in the helpers, not assumed). Write the presence rules: optional on limit, granted and the four counts; an absent or null granted reads off, an absent, null or negative limit reads 0 with no step, an absent, null or negative count changes nothing stored, an absent read-back granted is not reported, a generation or sequence of 0 is invalid, a page size or wait of 0 means the default except in a manifest.
  • 2.9 Write the limit entry's rules into the text and definitions' comments: over_limit holds keep or step, step holds the id only beside step, a later word carries its data in a field of its own, the step id grammar (1 to 64 bytes of lowercase ASCII letters, digits and _, beginning with a letter, never keep, null, undefined or constructor, compared byte for byte), a connector applies the named step only under step with a declared id and never refuses a record because the two fields disagree, delete_after of 30 days or more on every limit entry of a key whose steps end in a reclaim and read as no step when absent or not above 0, and a switch entry carrying kind, key and granted only.
  • 2.10 Write the reservations into the specification's text, not as Protobuf reserved statements: one reserved field name, one reserved name and number in each enum, one reserved person identifier and one reserved workspace identifier no console gives a recipient, and the identifiers and values the probe and harness use for them. Write the version rules into the text: protocol_version as "major.minor", a connection speaking the lower of the manifest's and the console's version, each side speaking every earlier minor of its major, what a later minor may add, the table of the meaning of a value a receiver does not act on, below 1.0 each 0.y as a major spoken alone, 12 months of support per major after its successor, 1.0 tagged from a 0.y both connectors passed with no wire change, and the alias.

3. Specification repository: checks, generated files and vectors

  • 3.1 Add the repository check enum-zero-unspecified: it fails an enum whose number 0 is not named ..._UNSPECIFIED, one marked frozen, and one whose definition comment does not say what an undefined value means where the enum is used. Test it with a passing definition and one definition for each failure.
  • 3.2 Add the check scalar-presence: it lists every singular scalar field without explicit presence in a committed file, and fails a definition that adds one the file does not list, so that adding to the list is a reviewed change. Configure buf breaking with the WIRE_JSON category and the FIELD_SAME_CARDINALITY rule against the last published version of the same major, enabled from 1.0. Test that a field gaining or losing optional fails it once enabled, and that a new unlisted scalar fails the scalar check.
  • 3.3 Add the check reserved-names: it fails any definition that uses the reserved field name, enum name or number, or reserved identifier of 2.10. Test it with a definition that uses each.
  • 3.4 Define the field option personal_data, mark the identity items and every field carrying a data item about a person, and add the check personal-data-marks that fails a definition that adds a field of either kind without the mark. Provide the generated field-mask list the console reads in task 13.6 so that a console writes whatever it keeps of a message from the definition with the marked fields left out.
  • 3.5 Generate gen/openapi/ and gen/jsonschema/ from the definitions, with a CI step that fails when a committed generated file differs from what buf generate produces. Test the generated files: each 64-bit integer is a string, each field has its snake_case name, each enum lists its names, optional fields are not required, and an unknown field is allowed in every message but the manifest.
  • 3.6 Add the repository's CI workflow: it runs buf lint, the generation drift check of 3.5, the repository's own checks of 3.1 to 3.4 and 3.11 and every test, and fails the build on any of them.
  • 3.7 Write the test vectors for protocol 0.1 under vectors/0.1/: each message as JSON (every operation, a switch entry, a limit entry under keep and under step, a limit entry with coming_steps while the clock counts and while it is paused, a reclaim entry with delete_after, a workspace record and a person record), the canonical body bytes a signature covers, and the refusals a receiver gives with the error it answers (an unknown or absent entry kind, an unknown recipient kind, an absent generation, a number written as a JSON number where a string is required, an enum spelled differently from the definition read as an unknown value, a token outside its shape).
  • 3.8 Write the signature vectors: Standard Webhooks' own vector (cited by commit) and one that includes a connection identifier, the same signature checked under another connection's identifier, a v1a entry alone, an Ed25519ph signature over the same content, the identifier upper-cased, two v2a entries during an overlap of keys, a timestamp six minutes old and one four minutes ahead, a webhook-id of msg.1, and a gzipped body signed over its uncoded bytes. Each vector names the verdict.
  • 3.9 Write the pairing vectors: the secret forms (lower case, a dash, a tab, a non-breaking space, a trailing newline, O for 0, I and L for 1, characters after the eighth) that verify, seven characters and a U that refuse with no proof, the scrypt parameters the specification fixes with the derived proof key, the HMAC-SHA-256 proof over the caller's key, the answerer's key and the connection identifier with each in its fixed encoding and a length ahead of each, a pull code with every part, with parts it does not know, and malformed ones, and the unpadded base64url key beside its whpk_ form.
  • 3.10 Write the manifest vectors: the sample manifest, a minimal one ("classes": []), a limited key with its steps, and one refused vector for each refusal of connector-manifest (an unknown field at any depth, an unknown value in any list, forget among the classes, a repeated class, operations without lifecycle, list or describe listed, set_status without a state, a person's limit, a key outside the prefix, a limit without a unit, an order outside the list, a through that is not a switch, a reclaim not last, two reclaims, a later step without after, a reclaim without delete_after, a step id outside the grammar, a length of 0, a missing on_forget sentence, empty keys, no on_forget, steps on a switch, after on the first step, delete_after on a restrict step, a step lacking sentence or does, states without set_status, lifecycle without an operation, an item outside identity, max_read_back_page of 0, a protocol_version above the console's).
  • 3.11 Add the check version-package-kept: it fails when the package of a protocol version, its directory under proto/ or its committed generated Go code, is deleted or renamed, because a package is never deleted (design.md, ruling 44) and buf has no rule for it. Test it with a definition set that has v0_1 removed and one that adds v0_2 beside it.

4. Specification repository: the text of the rules

  • 4.1 Write spec/signatures.md: the headers, the identifier and timestamp rules, the bytes a v2a signature covers (<connection identifier>.<webhook-id>.<webhook-timestamp>.<body> hashed with SHA-256 and signed with pure Ed25519), the five-minute tolerance, signing before content coding and sending requests uncoded, the checks every endpoint makes, in_reply_to, the identifier a push carries and a retry keeps, no refusal of a repeated identifier, key_replaced, and the rule that a 429, a 503 or no answer is not stored as a result. Cite Standard Webhooks by commit.
  • 4.2 Write spec/pairing.md: the origin allowlist's patterns, HTTPS and private connections, the two code forms, the secret's alphabet and the entry normalisation, the proof's bytes, the lapse and the five failed proofs, the second-key rule in both directions, pairing again, replaced keys, suspension, and the stable identifiers.
  • 4.3 Write spec/transports.md: the two transports, each operation's path and answer, the push call, the feed, the answers to feed items, repeated offers, position_lost, the paged full read with its cursor and feed_position, the rate of full reads, the read-back's page and limit, the meaning of no answer with the backoff and Retry-After rules, a connector's 429 setting the console's retry, and the sentence that failed carries no reason and no connector relies on when a record comes again.
  • 4.4 Write spec/obligations.md: the six things of the minimum connector and the rules of connector-obligations: a key a record does not list, entry kinds, a switch going off, a limit entry and its word and step, restricting only the number a step requires, keep and what comes back as room opens, a reclaim's destruction no sooner than the longest length and 30 days, dates never acted on, the counts every answer and read-back carries, what applied covers and the other answers, linking accounts only by an identity the console sent, the store under the connection identifier, forget and what each side keeps, silence changing nothing, no role that can change the connector's credentials, and an acquire with no answer read as a refusal.
  • 4.5 Write spec/personal-data.md and spec/manifest.md: the closed value types of a message a connector sends, region data's types, what the console keeps and logs, the marked fields, and the manifest's members, closed lists, staging, approval and the probe's cases. Write the repository's README.md as the one entry point a connector's author is sent to, so that no step refers an author to the console's source.

5. Specification repository: the helpers

  • 5.1 Go helper helpers/go/sign, shared by every protocol version: sign and verify v2a over the bytes of 4.1, parse and write the space-separated webhook-signature header with one entry per key during an overlap, hold a list of keys and accept any entry that verifies under any key, skip every entry whose prefix is not v2a,, enforce the five-minute tolerance and the webhook-id grammar, and convert a 32-byte public key from unpadded base64url to the padded whpk_ form. Wrap Standard Webhooks' Go verifier and signer once a release ships v2a; until then implement the bytes. Test against the vectors of 3.8.
  • 5.2 Go helper helpers/go/envelope, which exists once per protocol version (v0_1 first), each taking the version constant it sends from the generated package of its own version (design.md, ruling 44): the checks of version, connection, operation and in_reply_to before any handler runs, the signing of each answer when it is sent with no signed bytes stored, a repeated webhook-id answered again, and a connect-go handler option that turns off response compression so a signing layer never sees compressed bytes; a receiver removes any content coding before it verifies. A request whose content type begins with application/grpc is answered 415 and reaches no handler, because connect-go handlers also serve gRPC and gRPC-Web (design.md, ruling 45). Tests: an apply body posted to the forget endpoint, another connection, a missing, unknown and mismatched operation, another major, an answer naming another request, a gzipped answer through a proxy, a request with content type application/grpc+proto.
  • 5.3 Go helper helpers/go/retry, shared by every protocol version: a request that gets 429, 503 or no answer is sent again under the same webhook-id, signed afresh with a new timestamp, after the longer of its own wait (one second, doubling to five minutes, with jitter) and the Retry-After it was given (above five minutes counts as five, a value that is neither whole seconds nor an HTTP date counts as absent); stored signed bytes are never sent again. Tests with Retry-After of 2 and 86400, a bare 503 and a missing answer.
  • 5.4 Go helper helpers/go/pairing, shared by every protocol version: the secret's entry normalisation (remove non-alphanumerics, ASCII capitals, O to 0, I and L to 1, first eight characters, refuse fewer than eight or a U before any call), the scrypt key and HMAC proof, the pull code's parse that ignores parts it does not know, pinning of the answerer's key only after a verified answer, one key kept across retries of one pairing, and the handling of code_proved_twice (show it, do not try that code again, drop a pinned console key). Tests against the vectors of 3.9.
  • 5.5 Go helper helpers/go/decode, which exists once per protocol version (v0_1 first), each taking its version constant from its own package: decoding that discards unknown fields everywhere but the manifest, which it decodes strictly; presence-aware readers (GetGranted() and the count readers) implementing 2.8's absence rules; the table of the meaning of a value a receiver does not act on; a refusal of a record lacking its generation; and the strict manifest decoder from the raw body where Connect's reader discards unknown enum names. Tests against the vectors of 3.7 and 3.10.
  • 5.6 Go helper helpers/go/applyloop, which exists once per protocol version (v0_1 first), each taking its version constant from its own package: one recipient at a time; every entry carried out that can be, none undone; the highest generation begun and the highest carried out whole kept per recipient; stale below the highest begun; failed for a record not carried out whole, never naming the pushed generation or higher as applied; unlinked and linked_elsewhere with no entry carried out; a repeated push carried out again, never answered from a stored result; and one function that builds the counts of an apply answer and a read-back item together (held, restricted, locked, granted, moved, each present when 0). Tests for each outcome and for the repeat.
  • 5.7 Dependency check in the specification repository: go list -deps ./... of the helpers, the harness and the sidecar holds no Temporal SDK, and a CI step fails when one appears.

6. Specification repository: the conformance harness

  • 6.1 harness/ program skeleton: the command line --transport push|pull, the connector's address and, under push, the pairing code the connector shows, --report <file>; it plays the console toward one connector with no database and no deployment, holds its own ed25519 key for the connection it plays, serves every connection over HTTPS with a certificate from a test authority generated for the run and printed so the author can trust it for the run alone, and also serves a certificate from an authority the connector was not told to trust; under push it sends to the connector's endpoints, under pull it serves the pairing endpoint, the feed, the feed answers and the full read, and prints the pull code (the console's address, a connection identifier, its key and a secret) for the connector's author to give it.
  • 6.2 Harness case registry and report: each case has a stable identifier <capability>/<case> and belongs to one of the groups protocol-specification, protocol-versions, connector-pairing, protocol-signatures, protocol-transports, connector-manifest, connector-obligations and protocol-personal-data, run in that order and each to its end when a case fails; the harness reads the classes from the announced manifest, sends nothing of an undeclared class and reports the case skipped with the class named, always exercises forget, reports a case that could not start because pairing failed as failed, writes the JSON report (identifier, passed | failed | skipped, and for a failure the message sent, the answer received and the capability; the harness's release and the protocol version it ran, and the connector's announced version; no pairing code, secret or private key), and exits 0 when every case passed or was skipped and 1 otherwise. A test file lists the published identifiers and fails when one is removed or reused.
  • 6.3 Cases of group protocol-specification: every message the connector sends decodes under the definition and the JSON spelling with the vectors of 3.7; an int64 written as a number is a failure; presence cases (an entry with granted absent, null, a limit absent, null or negative, a count absent or negative); an unknown or absent entry kind and an unknown recipient kind apply nothing and answer with an error; a switch entry with extra fields is read as on or off only.
  • 6.4 Cases of group protocol-versions: the connector announces its version; a message naming another major is refused with nothing applied; an announcement above the console's version is answered with a signed refusal naming versions, and the connector announces again only on a signature it verified and under the pinned keys; it discards a field it does not know at each level and does not mistake the reserved test field for a real one; every value it acts on is sent as the zero value and as the reserved name and number, and each message it sends names its version, holds only fields that version defines and writes every required number, a zero included.
  • 6.5 Cases of group connector-pairing: pairing from a code in each form (the secret forms of 3.9 that verify, seven characters and a U refused with no proof sent), pinning nothing from a wrong proof or a proof under other keys, a lost answer retried under the same key, one key across retries, a pull code with a pairing answer signed by another key pinning nothing and ending with no further call, a certificate from an unaccepted authority paired with nothing, a second console key proving the code first (the connector makes no further call), under push two console keys proving to the connector (it pins neither and answers both), the pinned console key dropped after code_proved_twice, a connector restart keeping its key, a connector paired again onto the same connection with its store deleted reporting the same held counts and taking nothing out before the last page, a connector paired to a second connection identifier applying nothing from the first, and a key no record has listed left alone.
  • 6.6 Cases of group protocol-signatures: the connector signs and verifies as 4.1 says; it refuses a bad signature, a timestamp six minutes old, another connection, another operation (a signed body of every other operation posted to each endpoint, an apply body to forget in particular), another major and a replaced key, and shows key_replaced and sends no retry; every answer names what it was sent on both transports; a gzipped answer through a proxy verifies; a repeated webhook-id is answered again and not refused; the vectors of 3.8 verify, and the right key under another connection's identifier, a v1a entry alone, an Ed25519ph signature and an upper-cased identifier do not.
  • 6.7 Cases of group protocol-transports: a push at the apply path answered with in_reply_to; under pull the connector polls the feed, reads from the first kept item without a position, answers each item at the feed answers path, and treats a repeated offer as a record to carry out again; it reads the full read through its pages to last, keeps reading the feed from feed_position while it reads, and a lowered record whose page was already sent lands before the last page is asked for; an unknown cursor or position makes it read again from the first page; it answers the read-back with the smaller page bound; it treats 429 with Retry-After of 2 and of 86400, a bare 503 and a missing answer as no answer, waits as the specification says and signs afresh under the first identifier; after the harness has sent no message for a fixed number of read-back rounds (ten) its read-back still reports every key as the last record gave.
  • 6.8 Cases of group connector-manifest: the announced manifest is complete and decodes under the manifest vectors; the connector announces again after a restart; it declares every key its application delivers with the right holder and kind, and where a class is declared that class's members are present.
  • 6.9 Cases of group connector-obligations, switches and absence: a declared key a record omits is not held (a switch is off) and a key no record has listed is left alone; granted absent and null read off; turning a switch off ends use at once and destroys nothing, and turning it on restores use; a record at generation 0 or with no generation is refused; and a repeat of a record moves nothing.
  • 6.10 A test of the case registry: the core and the probe hold no case about limit behaviour (a step, keep, a reclaim, a count against a limit), and a limit entry is checked only for decoding, in group protocol-specification (6.3); the cases for limited keys come with the console's limits.
  • 6.11 Cases of group connector-obligations, answers: applied names the pushed generation and carries applied_generation 0 when the connector has applied none, and granted present for each switch entry; one entry that cannot be carried out answers failed with the other entries carried out; generation 17 after 18 answers stale; failed and unlinked naming the pushed generation or higher are protocol errors; a repeat of a record is carried out again and not answered from a stored result; a re-apply after a direct change to the application restores the record; a second recipient whose identity resolves to a linked account gets linked_elsewhere and nothing; silence applies no step.
  • 6.12 Cases of group connector-obligations, links, store and forget: nothing is linked without something held; nothing is taken out for lack of a record, and nothing before the last page of a full read after a lost store; a lost store reads every record again and makes nothing twice; a forget of an unknown identifier answers forgotten, a repeat answers the same, a later record naming the identifier answers forgotten and creates nothing, an identifier forgotten is absent from every later read-back, a pending link never lands, and an apply body to the forget endpoint is refused.
  • 6.13 Cases of group protocol-personal-data: the harness sends every approved data item and the identity as marker values and fails a connector whose messages carry a marker outside region data; no free-text field, no token outside its shape and no application identifier for a thing in a commit or release; failed carries no reason or text.
  • 6.14 The probe as an exported function of the harness package, taking a transport that reaches one connection and its approved manifest, running only connector-obligations/absent-key-reads-as-not-held, connector-obligations/granted-absent-and-null and protocol-versions/reserved-field under the same identifiers and code as the harness, sending only to the reserved recipients and values of 2.10 (a test fails the probe if any record it sends names another recipient). Tests: a connector that keeps an old grant, one that refuses the reserved field, one that reads a field it should discard, and one that passes; the probe's case list is exactly these three.
  • 6.15 Harness self-test fixtures: a correct reference connector under each transport (built on the helpers) that passes every core case with status 0, with the report holding no secret.
  • 6.16 Harness mutant fixtures: one mutant connector for each case family (keeps a missing key, ignores generation order, skips a signature check, retries with stored bytes, answers applied for half a record, echoes a marker) that fails exactly the cases it breaks, with the report holding the exchange for each failure.

7. Specification repository: the sidecar

  • 7.1 sidecar/ skeleton and local interface: one program that serves JSON on --listen (default 127.0.0.1:8470, refusing any non-loopback value and binding only 127.0.0.1 and ::1) and calls the connector's own code at --connector (loopback only); flags --manifest, --store, --repair; JSON in and out; no field of the signed envelope reaches the connector's code. Every 64-bit number in the JSON between the sidecar and the connector's code is a JSON string, as on the wire, and a count the application does not know is left out (design.md, ruling 46). Tests: a non-loopback --listen is refused, a non-loopback --connector is refused, the defaults apply, requests and answers are JSON, a count of 2^53 + 1 reaches the connector's code as a string, and an unknown count is absent.
  • 7.2 Sidecar pairing: the pairing code as one argument; under push it takes the secret in any form 3.9 lists, under pull the pull code with address, connection identifier and console key parsed from it and no further input; a malformed code is refused with no network call; --repair with a new code pairs again, keeps the store for the same connection identifier, signs with the new key only and announces again; key_replaced is shown to the operator and the message is not sent again. Tests with the vectors of 3.9: a push secret in each form, a pull code parsed with no further input, a malformed code refused with no network call, --repair keeping the store and the connection identifier.
  • 7.3 Sidecar manifest: announce the manifest file after pairing, after each restart and whenever the file changes, at the version its build supports; refuse to start for a manifest above its build (a test with a manifest one minor above, and one for a change of the file after start announced again); never ask the connector's code for an undeclared class (a test that a manifest without usage produces no call for it).
  • 7.4 Sidecar store: one file per --store holding, per connection identifier, the console's key, its own key, the highest generation begun and the highest carried out whole per recipient, and the list of forgotten identifiers; no read or write of one connection's store while serving another; a restart keeps keys and generations; after the store is deleted and the sidecar paired again onto the same connection it asks the connector's code for the held counts, reports them in the next read-back, takes nothing out before the last page of the full read and makes no instance twice; paired to another connection identifier it applies nothing from the first. Tests: a restart, a lost store, a second connection, and no read of one connection's store while serving another.
  • 7.5 Sidecar apply: verify each record, drop and answer stale below the highest begun, give the connector's code one recipient's record at a time in a POST to <connector>/records (JSON with kind, id, generation and the entries, every declared key listed), take back an outcome and per-entry counts, sign and send the answer, and store the generation as applied only when the code answered applied; a half-done generation orders what comes after. Tests: a newer generation, an older one answered stale, a failed apply stored as not applied, a half-done generation ordering the next.
  • 7.6 Sidecar read-back, forget and transports: answer the read-back from the connector's counts by a POST to <connector>/counts, passing presence through unchanged and supplying no count the code did not give; carry out a forget by a POST to <connector>/forget, answering forgotten only after the code finished and keeping the identifier alone, and answer forgotten with nothing applied to a later record naming it; under push serve the endpoints, under pull poll the feed and keep reading it from the first page's position during a full read; read a full read to its last page before it acts on an unheard-of recipient; treat 429, 503 or no answer as no answer with the retry rules. Tests: a count that is 0 passed through, a count not known left absent, a forget that completes, one not finished, a record for a forgotten identifier, a push console, the feed read during a full read, and a 429 from the console.
  • 7.7 Sidecar logging and the harness gate: a log line names the connection identifier, the operation, the outcome and the generation and never an identity item, the text of a record or an answer, a secret or a private key (a test logs a marker identity and greps the output); the sidecar's CI runs the harness core on each transport against a connector that does only what the local interface gives it and requires status 0.
  • 7.8 Release the sidecar as ready-built programs for Linux and macOS on amd64 and arm64 and for Windows on amd64, built and attached to each release of the repository, so that an author needs no Go toolchain to run it (design.md, ruling 46). Tests: the release step lists exactly those five programs, and the Linux amd64 program pairs from inside a container image that holds no Go toolchain.

8. Migrations

  • 8.1 internal/db/migrations/00043_connector_connections.sql, recipients and operations: replace chk_resource_keys_recipient with a CHECK of person, workspace or NULL and rewrite each stored organization to workspace in the same statement (no other value changes, so a boot after it changes no stamp); keep platform_keys_no_recipient and person_recipient_keys_boolean; replace chk_provider_operations_verb_valid so that it accepts exactly the five verbs (create, set_status, delete, list, describe) and the nine class names (reports, lifecycle, usage, names, regions, sign_on, observe_only, adoption, export).
  • 8.2 Same file, core.provider_connections: add transport (CHECK push, pull or NULL; NULL for every existing row, the connection of an installed integration), connector_address (CHECK non-NULL exactly when transport = 'push'), and read_back_interval, push_timeout and full_read_rate, each NULL until an operator sets it with a CHECK holding it above 0 (full_read_rate a whole number of 1 or more). The existing status, approval and shape CHECKs, the unique index per provider over every row, and the updated_at trigger stay. The version a connection speaks is computed and has no column.
  • 8.3 Same file, core.connection_keys with key_id (UUIDv7 default), connection_id (foreign key), public_key (bytea, CHECK 32 bytes), pinned_at, last_verified_at, replaced_at and refused_after_replacement (default 0), a partial unique index allowing one row per connection with replaced_at NULL, uq_connection_keys_connection_public_key, and a BEFORE UPDATE OR DELETE trigger that refuses a delete and a change to any column but last_verified_at, replaced_at and refused_after_replacement. Grants: core_owner all, core_reader SELECT, core_writer SELECT, INSERT and column-level UPDATE on those three columns.
  • 8.4 Same file, core.connection_approvals (approval_id UUIDv7, connection_id, kind CHECK first_pairing, pairing_again, manifest_change, approved_by_person_id foreign key to core.persons, approved_at, client_address NULL, no free-text column, a trigger refusing UPDATE and DELETE) and a deferred constraint trigger that holds core.provider_connections.approved_at equal to the latest approved_at of the connection's rows while transport is not NULL and the connection is approved, so both are written in one transaction; a connection with transport NULL has no row and is exempt. Grants: core_writer SELECT and INSERT only.
  • 8.5 Same file, core.connection_manifests (connection_id foreign key, state CHECK approved, staged, protocol_version, manifest JSONB object, received_at, uq_connection_manifests_connection_state (connection_id, state)) and core.connection_resource_keys (connection_id foreign key, resource_key foreign key to core.resource_keys, display_name, recipient CHECK person, workspace, kind CHECK boolean, numeric, primary key over both ids). Grants: core_writer SELECT, INSERT, UPDATE and DELETE on both.
  • 8.6 Same file, core.connection_probe_runs (probe_run_id UUIDv7, connection_id foreign key, run_by_person_id (foreign key to core.persons), ran_at, protocol_version, outcome CHECK passed, failed, cases_run and cases_failed as text[], approval_id foreign key to core.connection_approvals, CHECK (outcome = 'passed') = (approval_id IS NOT NULL), no free-text column, a trigger refusing UPDATE and DELETE) and core.pairing_codes (code_id UUIDv7, connection_id with no foreign key and a database default of uuidv7() for a first pairing, shown_at, lapses_at CHECK one hour after shown_at, proof_key bytea, failed_proofs default 0, bound_key, proved_at, proof_address, ended_at and end_reason CHECK lapsed, failed_proofs, second_key, confirmed, declined, set together; no column holds the secret).
  • 8.7 Same file, identifiers: CREATE EXTENSION IF NOT EXISTS pgcrypto, a one-row table core.recipient_identifier_key holding the 32-byte key with no privilege for any role but core_owner, core.recipient_identifiers (connection_id foreign key, exactly one of person_id, workspace_id, org_id by CHECK, identifier 43 characters of unpadded base64url, unique over each recipient column with connection_id and over (connection_id, identifier), no name, email or subject column) and the enclosed function core.recipient_identifier(p_connection_id, p_kind, p_recipient_id), SECURITY DEFINER, owned by core_owner, search_path = core, pg_temp, EXECUTE revoked from PUBLIC and granted to core_writer: it reads the key the migration generated (the migration inserts the one row with gen_random_bytes(32) when it creates the table, so a backup taken after it holds the key), derives HMAC-SHA-256 over the connection identifier, the kind and the recipient's identifier, inserts the row when missing (ON CONFLICT DO NOTHING) and returns the stored or derived value. core_writer holds SELECT and INSERT on the table and no UPDATE or DELETE; a BEFORE UPDATE OR DELETE trigger refuses both for every role.
  • 8.8 Same file, core.connector_feed_items (connection_id foreign key, position rising within a connection, op CHECK apply, read_back, forget, record_id and generation for a record, created_at; nothing else, so the row holds identifiers only; the retention of an item is the console's own and set by a constant beside the relay's) and core.read_back_runs (run_id UUIDv7, connection_id foreign key, cursor text, started_at, finished_at, items_refused count default 0). Grants per per-schema-roles: core_writer SELECT, INSERT, UPDATE and DELETE on both. COMMENT ON every table of 8.3 to 8.8 as migration 00018 does.
  • 8.9 internal/db/migrations/00044_workspace_records.sql, records: delete every core.desired_state_records row addressed to an organization, replace org_id with workspace_id (foreign key to core.workspaces, the exclusive arc chk_desired_state_records_recipient over person_id and workspace_id, uq_desired_state_records_connection_workspace), add push_wait interval and next_attempt_at timestamptz (both NULL when no wait applies, CHECK push_wait > 0, set together), and leave the content CHECK for the task that follows the content function, because the stored content of 00024 (lines 413 to 418) writes no kind, writes granted only when true and writes targets.
  • 8.10 Same file, projections: delete every core.observed_projections row addressed to an organization, replace org_id with workspace_id as in 8.9, widen chk_observed_projections_result to applied, unlinked, linked_elsewhere, failed, forgotten, add not_carried_out_since (NULL unless the latest report is one the record's instruction cannot explain), set the observed default to '{"entries": []}', rewrite surviving projections to that shape, and make chk_observed_projections_observed require an object holding an entries array. A projection still references no record.
  • 8.11 Same file, the organization's activity: core.organization_counts_as_active(p_org_id), LANGUAGE sql STABLE, SET search_path = core, pg_temp, true while status = 'active' and not (exactly one core.org_members row with status = 'active' whose person has an open core.person_cut_offs row); the System organization never counts as not active through the second condition and an organization with no active member never counts as not active through it. No column of core.organizations, core.org_members or core.persons changes when this answer changes. EXECUTE to core_owner, core_writer and core_reader.
  • 8.12 Same file, core.desired_state_content(p_connection_id, p_person_id, p_workspace_id) replaced: it reads the connection's keys from core.connection_resource_keys alone, lists every key of the record's recipient kind once, each entry carrying key and kind; a switch carries granted true or false; a limit carries limit (0 included) and over_limit keep; no targets, policy or hidden mark; entries sorted by key COLLATE "C". A person's record uses the source set of desired-state-records (owned and member organizations that count as active, never the System organization) and lists every person key off while the person has an open cut-off. A workspace's record reads the primary pool of the active primary core.pool_assignments row and lists every key off or at 0 when the workspace is not active, its organization does not count as active (task 8.11) or it has no such assignment. A key whose recipient is NULL appears in no record. core.generation_ceiling, core.next_generation and the guard trigger are unchanged.
  • 8.13 Same file, after the function: rewrite every surviving record with the new function under a raised generation (core.next_generation, the guard trigger permitting the content change with the rise), then add the content CHECK with the entry rules (a kind of switch or limit; a switch with a boolean granted and none of limit, over_limit, step, delete_after; a limit with a whole-number limit of 0 or more and no granted; a limit's over_limit of keep or step, a non-empty step exactly when over_limit is step; delete_after only as a duration of at least 30 days), and write one connection outbox row for each approved connection through core.insert_desired_state_notice so the relay recomputes workspace records under a generation floored by the clock.
  • 8.14 internal/db/migrations/00045_delivery_triggers.sql, outbox: widen chk_desired_state_outbox_subject_kind to pool, organization, person, workspace, connection and desired_state_outbox_recipient_only_shape to a person or workspace; drop core.target_mappings, its two triggers and functions (desired_state_on_target_mappings) and its indexes; replace core.requeue_desired_state so it inserts one recipient_only row with attempts 1 for a person or workspace that exists and raises for any other kind or an unknown identifier, still executable by core_writer and refusing nothing else.
  • 8.15 Same file, triggers: add core.desired_state_on_workspaces (INSERT of an active row, DELETE of an active row, UPDATE where status or org_id is distinct; one workspace row) and core.desired_state_on_pool_assignments (INSERT or DELETE of a primary active row, UPDATE where pool_id, workspace_id, is_primary or status is distinct while the old or new row is primary and active; one workspace row, a second for the old workspace when workspace_id moved); replace core.desired_state_on_org_members so that it also writes one organization row for the row's organization (and the old one when org_id moved) when the changed row's person has an open cut-off or another active member of that organization has; replace core.desired_state_on_person_cut_offs so that it also writes one organization row for each organization whose only active member is the person it names (and the old person on a move); keep the organizations, entitlement and provider_connections triggers. Each function is SECURITY DEFINER, owned by core_owner, search_path = core, pg_temp, EXECUTE revoked from PUBLIC and granted to no role; no trigger reads core.resource_keys, core.provider_connections or any table but the two named. Each UPDATE trigger's WHEN names every column the content function reads from its table (including core.pool_assignments, core.workspaces).
  • 8.16 The Down sections: drop, in reverse order, the new triggers and functions, restore core.target_mappings with its constraints, triggers and index shapes from 00024 (empty), restore org_id columns (empty tables), the old CHECKs and the old content function, then the tables of 8.3 to 8.8 and the columns of 8.1 and 8.2 and the extension if this change created it. A migration test in internal/db (beside integration_delivery_migration_test.go) applies Up, Down and Up again on a scratch database and asserts the schema after Down equals the schema before Up, and that Up after a database that holds a fedwiki_sites organization record and an organization-keyed projection deletes both and leaves workspace records to be recomputed.
  • 8.17 Reader role: regenerate internal/migrate/testdata/console-reader-columns.txt with TestConsoleReaderColumns -update and check the diff is read core.resource_keys.recipient unchanged and one read line for each column of the new tables, with core.pairing_codes.proof_key, core.pairing_codes.proof_address, core.connection_approvals.client_address and the key column of core.recipient_identifier_key withheld, as scripts/db-reader-role.sql withholds addresses as a class; add each withheld pattern to the personal list of scripts/db-reader-role.sql in this change. A database test runs the script against a migrated database and selects every read column as console_reader, and refuses proof_key and both address columns.

9. Database tests of the schema, the content function and the triggers

  • 9.1 Recipient and operation constraints: core.resource_keys accepts person, workspace and NULL, refuses organization, a recipient on a platform key and a numeric person key; a stored organization becomes workspace and nothing else changes; core.provider_operations accepts each of the five verbs and nine classes and refuses any other name.
  • 9.2 Connection constraints: the database chooses connection_id; an unknown provider, status or approval is refused; approved_at and approval_state agree; a suspended connection stays approved; an update stamps updated_at; a push connection needs an address and a pull connection has none; a transport value outside push and pull and a setting of 0 or below are refused; a second connection for a provider is refused, retired ones included; a declined first pairing is a retired connection; each connection table other than core.provider_connections refuses an unknown connection_id, and core.pairing_codes accepts one with no connection.
  • 9.3 Key tests: one current key per connection; a key the connection had cannot return; no row is deleted and no column but the three changes; a verified message updates last_verified_at and a refused one does not; a pairing again's transaction (replaced_at on the old key, the new key current, one pairing_again approval row, the connection's identifier, provider, display name, identifiers, records and projections untouched, a retired connection active with retired_at cleared, a connection pending after a second key approved with approved_at, one retired by a declined first pairing left pending, suspended_at and an operator's suspended kept) and a failing transaction that keeps the old key.
  • 9.4 Approval, manifest and probe tables: an approval writes its row and the connection's time in one transaction and the deferred trigger refuses disagreement; an unknown kind is refused; no row is updated or deleted; one approved and one staged manifest per connection and a second staged is refused; a staged manifest leaves the key list alone and approval replaces the approved row; a passing probe row names its approval and a failed row cannot; no probe row is updated or deleted; no free-text column exists in these tables (a catalog test over information_schema).
  • 9.5 Pairing-code tests: a pull code names an identifier before any provider exists; the first proof's manifest creates the provider row (provider_kind = 'provisioning', the manifest's display_name, no operation, state or key), the connection and the code's connection_id in one transaction; a code of a pairing again names the existing connection; no column holds the secret; lapses_at is one hour after shown_at; end_reason and ended_at are set together; a declined pairing leaves the provider row.
  • 9.6 Identifier tests: the same recipient gets the same identifier on every call; two connections give two identifiers; a person, a workspace and an organization with equal underlying UUIDs get three; a restore simulated by deleting the identifier rows and calling again gives the same values; a restore from before an identifier was stored derives it again; an UPDATE or DELETE of a row is refused for every role; one identifier names one recipient on a connection; core_writer cannot read the key table or run the HMAC; no row holds a name, email or subject.
  • 9.7 Record constraint tests: exactly one of person_id and workspace_id; one record per connection and recipient; unknown recipients refused; an organization cannot be a recipient; malformed content refused, and for each CHECK of task 8.13 a refused case (a switch without granted, a switch with limit, a limit with granted, a negative limit, a step under keep, a missing step under step, a delete_after under 30 days, an unknown kind) and the content function's output passes; a recipient who never held anything has no record; losing everything keeps the record and raises its generation; the migration replaces organization records.
  • 9.8 Content-function tests per rule, as core_reader: entries sorted in byte order; a switch entry carries granted and a limit entry limit; a switch that is off and a limit of 0 are written; a key the manifest does not declare for the kind is not listed; a staged manifest changes no content; no policy, hidden mark or targets; only a connection with transport NULL lists a limit; an owner without a membership row, an active member, an inactive membership, an organization not active, any one of two organizations granting a boolean, keys reaching only their declared kind, two workspaces of one organization reading their own pools, a workspace with no primary assignment or not active listing every key off, a key with NULL recipient carried by nothing, the System organization and its synthetic owner with no record, and Discourse's desired members agreeing with the persons whose record grants discourse_posting once cut-off persons, System-only persons and persons entitled only through an organization made not active by a cut-off are removed.
  • 9.9 Cut-off and activity tests: cutting off empties every person record of the person and keeps an organization's records that has other members; the person's own organization (sole active member) goes to zero; lifting restores both; a second active member keeps the organization active; a member joining ends the effect; an inactive membership is not a member; an organization with no active members counts as active; the System organization is never made not active; a cut-off person with no record gets none; a grant to a cut-off person's organization does not reach the person. Existing core.set_person_cut_off tests stay green.
  • 9.10 Hash, generation and recompute tests (beside the existing ones): key order does not change the hash and a changed value does; unchanged content writes nothing; a changed source raises the generation once; a floor raises it without a content change and creates a record that holds nothing; concurrent recomputes read after the lock and an interleaved grant and revoke leave no stale grant; core.next_generation and the guard trigger behave as before on workspace records.
  • 9.11 Projection constraint tests: exactly one recipient; a projection for a recipient with no record is accepted; one projection per connection and recipient; each of the five result values accepted and others refused; ahead_generation and ahead_at set together; a non-object or entries-less observed refused; the answers a push can give are storable; a left-out value stays left out.
  • 9.12 Trigger tests for the entitlement, organization and connection tables, each asserting the exact rows written as (subject_kind, subject_id, resource_key): the tests of integration_delivery_triggers_test.go updated for the new list; core.workspaces (a new active workspace, one leaving active, one moved to another organization, a rename writes nothing); core.pool_assignments (a new primary assignment, moving the primary pool, a secondary assignment writes nothing, suspending the primary assignment); a change with no connection and a key whose recipient is not stamped still write a row.
  • 9.13 Trigger tests for the membership and cut-off tables, each asserting the exact rows written as (subject_kind, subject_id, resource_key): core.org_members (an active insert, leaving active, a role change writes nothing, a move writes both sides, a second member joining a cut-off person's organization names the organization, a cut-off person's membership ending names it, an organization with no cut-off member writes no organization row); core.person_cut_offs (cutting off writes one person row, a sole member names the organization, another member means none, lifting writes the same rows, a repeated cut-off writes nothing); no trigger reads a table but the changed row and the two cut-off tables (a test over the function definitions).
  • 9.14 The column guard of integration_delivery_triggers_test.go extended to core.workspaces and core.pool_assignments: it fails when core.desired_state_content reads a column of a table that the trigger's WHEN clause does not name, comparing the function's definition from pg_proc with each trigger's condition from pg_trigger; and the same-transaction tests (a rolled-back change leaves no row, a refused pool in a rule change leaves none, plain SQL outside the application writes a row, first login writes its rows with the signup, a lifecycle request writes none).
  • 9.15 Enclosure and grant tests as member_console's grants and as console_runtime: core_writer cannot INSERT into core.desired_state_outbox, a trigger inserts for it, it cannot call a trigger function, core.requeue_desired_state writes one recipient_only row for a person or workspace and refuses an organization, a pool, a connection and an unknown identifier, a recipient_only row for a workspace is accepted and one for an organization refused by the outbox CHECK, and a retarget of a row is refused while claim columns update; core.recipient_identifiers accepts INSERT and refuses UPDATE and DELETE; core.connection_keys accepts UPDATE of its three columns and refuses the others and DELETE; core.connection_approvals and core.connection_probe_runs refuse UPDATE and DELETE; core.connection_manifests, core.connection_resource_keys, core.pairing_codes, core.connector_feed_items and core.read_back_runs accept INSERT, UPDATE and DELETE; an integration's writer role reads every new table and writes none; the reader computes a record's content; the column enclosure of core.lifecycle_requests in per-schema-roles still holds (the application role writes the nine insert columns and four update columns and no act column).
  • 9.16 Placement tests: on an empty database after migrating, the connection tables and the delivery tables of schema-namespacing exist in core and in no other schema; core.target_mappings does not exist; each of core.connection_keys, core.connection_approvals, core.connection_manifests, core.connection_resource_keys, core.connection_probe_runs, core.recipient_identifiers, core.desired_state_records, core.observed_projections, core.connector_feed_items, core.read_back_runs and core.lifecycle_requests holds a foreign key to core.provider_connections and none into an integration schema; no table of the fedwiki or discourse stream is added and an outside connector gets no schema, no role and no migration stream; the layout counts in schema-namespacing hold.

10. Queries

  • 10.1 internal/integration/queries/connections.sql: statements for the connection's pairing state: insert a pending connection with its provider row from a first proof, pin and replace keys (the pairing-again transaction of task 9.3), update last_verified_at and increment refused_after_replacement, list keys, write approvals, stage, replace and read manifests, replace the approved key list, write and read probe runs, suspend and resume, retire, read and set each of the three settings and clear one, and update a connection's address; keep EnsureInTreeConnection, ReapproveConnection, GetConnectionByProvider and ListDeliverableConnections, adding transport to the last.
  • 10.2 internal/integration/queries/pairing.sql: insert a code, find a code by connection identifier, add a failed proof under FOR UPDATE, bind the first key and proof time and address, end a code with its reason, find codes that lapsed, count a code's refused attempts, and the recipient-identifier call core.recipient_identifier in single and batched form (a page of identifiers in one statement).
  • 10.3 internal/integration/queries/desired_state.sql: the expansion queries for pool, organization, person, workspace, recipient_only and connection rows by recipient kind and identifier after a cursor, filtered to approved, non-retired connections whose key list holds a key of the matching recipient (and the row's key); the workspace content call; the record statements keyed by workspace_id; push bookkeeping with push_wait and next_attempt_at; the lag sweep and ListDesiredStateRecordsToPush excluding records whose next_attempt_at is in the future; the send-again statements (clear one record's wait, clear every record of a connection); feed-item insert, list after a position and the oldest kept item, delete past retention; the full-read page query by recipient identifier after a cursor with the records whole.
  • 10.4 internal/integration/queries/observed.sql: the projection upserts for person and workspace that never lower applied_generation, the push-result update with result_generation, the mark set (keeping the first time) and cleared, the ahead_generation and ahead_at statement, and the list of projections whose mark is set for a connection; read_back_runs insert, cursor write and finish; the count of items refused since the last finished run.
  • 10.5 Run make sqlc-generate; remove the queries that named org_id or target_mappings; go build ./... passes, go run ./scripts/sqlc-unused reports nothing unused and scripts/deadcode.sh reports nothing new.

11. Configuration, addresses and certificates

  • 11.1 Declare the three deployment-only list keys in the core configuration, each DeploymentOnly, never a runtime override or an integration_config_overrides row, shown and edited on no page and effective at the next start: connector-origins, connector-private-origins (entries <pattern> or <pattern>=<file>) and connector-trusted-proxies (CIDR ranges), all empty by default. Add the secret key for the master key from which each connection's console key is derived, named by cmd beside the other secrets and resolved through its -file flag (the spec names none; the implementer names it and records it in docs/environment-reference.md).
  • 11.2 Boot validation in one aggregated error naming each key: an entry that is not an https origin pattern, a private pattern matching an origin no entry of connector-origins matches, a file that cannot be read or holds no certificate, a range that does not parse. Tests for each refusal and for a valid configuration, in internal/config beside validate.go's tests.
  • 11.3 internal/integration/wire/origins.go: match a connection address against the allowlist (a host with an optional port, port 443 when absent, *.example.org matches any host with at least one label before example.org and not example.org itself), refuse userinfo, path, query and fragment and any non-https address before any call, and apply the same rule to an operator's edit of a connection's address (the key stays). Tests: an address outside the list, a subdomain match, a pattern not matching its own domain, plain http, the empty list connecting nothing, an address edit keeping the key.
  • 11.4 internal/integration/wire/tls.go: one HTTPS client constructor that verifies every certificate and has no skip, refuses a connection whose address resolves to a loopback, link-local or private address unless its origin matches an entry of connector-private-origins (checked again at dial time against the resolved address), and trusts a named certificate authority file only for the origins that entry matches. Tests with an httptest TLS server: an unverifiable certificate ends the call, a private address needs the mark, a private origin trusts its authority and no other, and a console never pairs with a connector whose certificate it cannot verify.
  • 11.5 internal/integration/wire/clientaddr.go: read the address a pairing proof came from from the right of the forwarding header, skipping entries inside connector-trusted-proxies, and only on the pairing confirmation path; return unknown when no proxy is trusted. Tests: no trusted proxy gives no address, the right-most untrusted entry wins, and sign-in and logs keep RemoteAddr (a test through the existing sign-in handler).
  • 11.6 The console's key per connection: derive the ed25519 private key for a connection from the master key and the connection identifier (HKDF-SHA-256 with a fixed info string), expose the public key for the pull code and the signing function, and never store a private key. Tests: the same connection gives the same key, two connections differ, and no recipient identifier depends on the master key.

12. Registration, boot and the installed integrations

  • 12.1 internal/integration/contract.go: RecipientKind becomes person and workspace; validateRecipients fails, naming the provider and the key, when an owned key has no entry, an entry names a key the provider does not own or a value is neither; a numeric person key still fails at the stamp through person_recipient_keys_boolean. internal/integrations/fedwiki/store/provider.go declares fedwiki_sites for the workspace; every test manifest that owns keys gains its recipients.
  • 12.2 internal/integration/registration.go: boot stamps the recipient with the provider as before and, in the same transaction, writes each installed integration connection's rows of core.connection_resource_keys from the keys its ProviderSource owns; it calls ReapproveConnection when a stamp or the written list changed (so existing deployments recompute their records after the migration), and leaves an unchanged boot, a pending connection, a retired connection and an outside connector's declarations alone. Boot never registers, changes or removes a declaration of a provider whose connection has a transport.
  • 12.3 internal/integration/approve.go (the registration of an outside connector's approved manifest): in one transaction, insert operations (the lifecycle verbs under the class lifecycle, list for every connector, each declared class) and states, and each manifest key as a core.resource_keys row when absent (the manifest's display_name and unit, provider, recipient, kind boolean for a switch, numeric for a limit); apply the prefix and nesting rules of provider-registry and fail the approval, writing nothing, for a key outside the prefix, a key another provider owns or a provider key that nests under a platform key; a key a later manifest leaves out keeps its row and stamp and leaves the key list. Tests: a first approval registers provider operations and keys, each failure writes nothing, a left-out key keeps its row, a connector declaring create, set_status or delete also records the class lifecycle, a connector with no class still records list, and a class gives no action on the operator pages.
  • 12.4 Registration tests in internal/integration/registration_db_test.go: after boot fedwiki_sites reads workspace and discourse_posting reads person; a missing, misplaced or non-person-or-workspace recipient stops boot; an owned key without a recipient stops boot before any connection is ensured; a recipient for a key the provider does not own stops boot; a numeric person key stops boot and names the key; the database refuses a recipient on a platform key and an unknown value; a changed recipient re-approves the existing connection and writes one connection outbox row, an unchanged boot touches nothing; a stored organization becomes workspace and a boot after it changes no stamp; an operator's status, retirement and display name survive boot; FedWiki's connection key list holds fedwiki_sites for the workspace after boot; a boot leaves an outside connector's declarations as approved; capability rows are not operator-editable.
  • 12.5 internal/integration/writegate.go: CheckWritable(ctx, db, provider) reads the provider's connection and returns an error naming the provider when its transport is not NULL. Call it before each write the installed Discourse integration makes (AddGroupMembers, RemoveGroupMembers and CreateUser in internal/integrations/discourse/client/client.go, reached from internal/integrations/discourse/workflows/activities.go) and each write FedWiki makes to the farm (CreateSite, SetSiteStatus, DeleteSite and HardDeleteSite in internal/integrations/fedwiki/workflows/farmmanager_client.go, reached from its workflows and from web/), refusing when it fails, so an older replica or a rollback cannot write beside a connector that is the writer.
  • 12.6 internal/integrations/discourse/discourse.go and internal/integrations/fedwiki/fedwiki.go Startup: when the integration's connection has a transport, start neither the startup hook's schedules nor the integration, and pause a leftover schedule through corework.PauseLeftoverSchedule; a connection with transport NULL keeps running unchanged. Tests: a write is refused with the connection marked, boot does not start the integration with the mark and starts it without, and the console starts and runs with no outside connector paired; boot does not look for a tree for an outside connector's provider.

13. The wire package: messages, signatures and the outbound client

  • 13.1 internal/integration/transport.go and internal/integration/transporttest: RecipientKind is person or workspace, Recipient carries the identifier minted for the connection (core.recipient_identifier) and, for a workspace, its organization's; DesiredEntitlement becomes an entry with Kind, Granted *bool that writes false, Limit *int64, OverLimit, Step and DeleteAfter, and loses Targets; outcomes gain failed, linked_elsewhere and forgotten; ApplyResult and ListItem carry per-entry counts and granted with presence (Held, Restricted, Locked, Moved as *int64); an ErrNoAnswer carrying a capped Retry-After, an ErrProtocol and an ErrRefused join ErrPermanent. No HTTP, signature or Temporal type appears in the file. The recording transport's script gains each new outcome; its unit test checks the recording and the script.
  • 13.2 internal/integration/wire/decode.go: decode every answer, read-back page and announcement from the specification's Go module with unknown fields discarded and counted (a count per page and per answer, never the field's name or value), the strict decoder for a manifest taken from the raw body, presence tests on granted, limit and the four counts, the table of the meaning of a value the console does not act on (an unknown push outcome is not done, retried and shown; an unknown recipient or entry kind refused; an unknown next action none; an unknown manifest value refused), the refusal of a report that lacks its sequence (the console has no report endpoint in this version, so the rule is held at the decoder), an out-of-shape token (a cursor, position or message identifier outside its grammar) is a protocol error, and unrecognized is read as an unknown field. Tests: the vectors of 3.7 and 3.10 decode as their expected outcome, an unknown field in a page is discarded and counted, an unrecognized list is not read, a negative or absent count changes nothing, an applied generation of 0 is accepted, a marker in an unknown field reaches no log (a captured logger).
  • 13.3 internal/integration/wire/verify.go: inbound verification for every connector request: all three headers present, the webhook-id grammar, the five-minute tolerance, content coding removed before verifying, the body read through internal/httplimit after decoding and refused whole above the bound, then the v2a verification against the connection's keys (a list), then the envelope checks (connection, operation of the endpoint, version the connection speaks, major). A message signed by a key whose replaced_at is set is refused and answered key_replaced, signed with the console's key, naming the request and replaced_at, and counted in refused_after_replacement; a key never bound is refused with no outcome; a suspended or retired connection's message is refused like any refusal; a verified message sets last_verified_at, a refused one does not. Tests for each scenario of protocol-signatures that the console receives: unsigned, stale, inside tolerance, msg.1, a v1a entry alone, a gzipped request, a feed_answer body posted to /connectors/records, another connection, a missing and an unknown operation, another major, the replaced key's answer verifying under the console's key.
  • 13.4 internal/integration/wire/push.go: the push transport for a connection with transport = 'push', built on the helpers and the HTTPS client of 11.4: Apply posts to <address>/connectors/apply and List to <address>/connectors/read_back with protocol_version, connection_id and op, message identifier <record_id>-<generation> for a record and one identifier per read-back page (a retried page keeps it, the next page changes it), each call bounded by the connection's push timeout (10 seconds unless set), requests uncoded and signed afresh on a retry with the first identifier and a new timestamp; content encoded as the definition's messages (int64 as strings, granted: false present), a person's record carrying exactly the identity items the approved manifest declares and a workspace's record its organization's identifier, both holding under push, in feed items (16.12) and in full-read pages (16.13); the answer is read through the bounded reader, verified, checked for its connection_id, op and major, to name the request it answers (in_reply_to) and, for applied and failed, the generation pushed; a refusal of any check is a transport error that is retried and shown. Tests with an httptest TLS connector: the apply and read-back paths, the timeout, every attempt carrying one identifier with fresh signatures, an answer naming another request, an applied answer for another recipient at the same generation and one at another generation refused, a new generation carrying a new identifier, the two answers of 429 and 503, and requests that travel uncoded (no Content-Encoding header).
  • 13.5 internal/integration/wire/retryafter.go: parse a connector's Retry-After (whole seconds or an HTTP date, RFC 9110 section 10.2.3, anything else absent), cap it at five minutes, and return it in ErrNoAnswer; the push and read-back activities take the next delay as the longer of their schedule's delay and that value for a 429 only, and leave their schedule for a 503, a missing or invalid header and any failure to answer (internal/workflows/desiredstate/push.go and readback.go, the retry-delay calls near push.go:356 and readback.go:298). Tests: Retry-After of 2 under a longer schedule, 86400 counted as 300, a bare 503, an HTTP date, a garbage value.
  • 13.6 internal/integration/wire/redact.go: build every workflow input, log line, metric and error the console writes about a message from the definition with the fields marked personal_data left out (the generated mask of 3.4); a log line about an inbound message carries the connection, the operation, the console's own identifiers, the field's name and the problem's class; unknown recipients are counted per page; a transport's error carries the HTTP status and the console's own class and never a body; no connector message is written to core.webhook_events. Console tests post a marker in an unknown field, an over-long token, an error-shaped field and an HTTP 500 body and check that it reaches no table, log line or workflow history (a replay of the history and a scan of every core table).
  • 13.7 internal/httplimit/httplimit.go and the wire client: every connector answer (a push answer, a read-back page, a forget answer, a posted pull answer) is read through the shared reader after content coding is removed and refused whole above the bound with an error naming the connection and the bound and quoting nothing; nothing is stored from a refused body, the call is an answer not done and retried, and the refusal is shown on the connection's page; a body is verified and decoded only after the reader returned it complete. Tests: an oversized push answer, an oversized read-back page storing nothing, a gzip bomb bounded after decoding, a posted pull answer bounded alike, an error that quotes nothing, and the existing FarmManager, Discourse and Temporal token tests unchanged.
  • 13.8 internal/integration/wire/version.go: the console's own versions and the version a connection speaks computed from the approved manifest's protocol_version (the lower of the two, every earlier minor of the major); a manifest above the console's version refused with a signed answer naming the versions it speaks and spending no pairing attempt; a connection at another 0.y sent nothing, shown as different versions and its next announcement taken as a staged change; the alias that makes a console of major 1 speak the last 0.y as 1.0; from the first minor after 1.0 a version-only manifest change staged with the connection speaking the lowest of the approved, waiting and console versions. The later-minor rules (an added field means what the message meant, a new value is not sent toward an older connection, a required number is scoped to its minor) are tested on a synthetic minor. Tests for each scenario of protocol-versions that concerns the console.
  • 13.9 The console's tests hold the specification's vectors: a test in internal/integration/wire decodes each vector of each version the console speaks (the messages, the refusals with the errors expected, the signature vectors, the pairing vectors) and compares the outcome with the vector's expected one; a vector added to the repository and not decoded by the console fails the test.

14. Connector-facing endpoints and pairing

  • 14.1 internal/server/connector_routes.go: mount POST /connectors/pair/{connectionID}, /connectors/manifest, /connectors/feed, /connectors/feed/answers and /connectors/records on the origin of base-url outside the session and CSRF stack (a test: a POST with no cookie and no Origin is served), each through task 13.3's verification, except the first pull pairing call, which 14.2 verifies under the key it carries, and with the specification's procedure strings; a request whose content type begins with application/grpc is answered 415 and reaches no handler (design.md, ruling 45; a test with application/grpc-web+proto); each answer is signed with the connection's console key, carries in_reply_to and its answer op, is sent uncompressed with no signed bytes stored; a repeated webhook-id is answered again; a stored result is replayed before the rate limit, and the rate limit is built; a 429 or 503 the console gives is written by one shared refusal writer that stores nothing as a result of an idempotency key (a test that triggers a rate refusal and finds no row written and the next attempt served).
  • 14.2 internal/integration/pairing, the pull pairing endpoint: find the code by the connection identifier in the path and answer an unknown identifier and a lapsed code alike; verify the proof (HMAC-SHA-256 under the scrypt key stored as proof_key, over the caller's key, the answerer's key and the connection identifier) before pinning anything; a wrong proof spends one attempt and five end the code (failed_proofs); a correct proof binds the key and the time and answers signed by the console's key for the connection; the same key and proof repeated is answered again and spends no attempt; the endpoint serves no key and no message carries the console's key to an unpaired connector. Tests: a correct proof, a wrong proof, a lost answer retried without cost, a proof under other keys, five failures end the code, a lapse after one hour, and an unknown identifier equal to a lapsed code.
  • 14.3 The pull code and the push entry: the code shown as https://<origin>/connectors/pair/<connection identifier>#k=<key>&s=<secret> with <origin> from base-url, the key unpadded base64url and the secret as two groups of four characters from Crockford's alphabet chosen with crypto/rand and stretched by scrypt into the stored proof_key; the push entry field turns off automatic correction, automatic capitals and spell checking, and the entry's normalisation matches 5.4 and refuses a short secret or a U before any call. Tests with the vectors of 3.9: a secret in every form verifies, look-alikes read as digits, a short secret or U refused with no proof, a code lapses, capitals do not use the locale (a test under a Turkish locale string), and the page carries no secret in its markup beyond the display.
  • 14.4 The push pairing act: the operator enters the connector's address (checked by 11.3) and secret; the console makes the first call to <address>/connectors/pair carrying its public key and the proof, and pins the connector's key only after verifying the answer, which carries the manifest; a connection row is inserted pending with the manifest's provider (and the provider row, task 9.5) in one transaction; code_proved_twice is shown to the operator and that code is not tried again; the console never trusts a second key it did not pair with. The recipient identifiers are minted at the first push call. Tests with a TLS connector fixture built on the harness: first pairing, a wrong secret, a certificate from an unaccepted authority paired with nothing, code_proved_twice.
  • 14.5 The second-key rule at the console: a valid proof under a second public key ends the code with end_reason second_key, pins neither key and answers that proof and every later proof of that code code_proved_twice, spending no attempt, while a repeat under the first key is a retry; before confirmation the page shows "Two keys proved this code. Make a new code." and no confirm control; after confirmation the console stops delivering to the connection, sets its approval_state back to pending and leaves what the connection holds as it is until it is paired again. A code ended by confirming still checks every proof until lapses_at. Tests: a second key before confirmation, a second key after confirmation (no push, no change to records), a second key spending no attempt, a retry under the same key.
  • 14.6 Confirm and decline acts: the first-pairing confirmation pins the connector's key and ends the code with confirmed; declining a first pairing retires the pending connection and ends the code with declined; declining a pairing again, or a code that ended before confirmation, changes nothing on the connection, its keys, records or status. Tests per scenario of connector-pairing (a declined pairing changes nothing, a lapsed code changes nothing) and provider-connections (a declined first pairing retires the connection).
  • 14.7 Pairing again from the connection's page (retired or not): run the transaction of task 9.3, stage a differing manifest, write a pairing_again approval row with who, when and the address where known (a connection retired by a declined first pairing stays pending until 15.3 approves it), send the new key nothing and keep accepting the old key between proof and confirmation, refuse the old key from the confirmation; a retired connection accepts no message and returns only by pairing again. Tests per scenario of connector-pairing and provider-connections (identifiers stay, both keys work until confirmation, the old key is refused after, a retired connection accepts nothing, a differing manifest is staged, a failed transaction keeps the old key).
  • 14.8 Refuse a pairing again whose manifest names another provider, any pairing of a connection whose transport is NULL, and a first pairing whose manifest names a taken provider (the refusal names the connection and says a second installation registers under a provider key of its own). Tests per scenario of connector-pairing (another provider refused, a taken provider refused with a reason, an installed integration's connection not paired).
  • 14.9 Suspension and the identifiers: suspending a connection sends it no push, read-back or feed item and refuses every message from its connector while its key, approval and records stay; resuming sets status to active, clears suspended_at and delivers what changed while suspended (one connection outbox row); recipient identifiers are the stored ones for every connection and are untouched by a pairing again, a rotation of the master key and a restore; a connector's store is kept under its connection identifier (harness-side). Tests: a suspended connection receives nothing and is not heard, resuming delivers, suspension keeps key and approval, a pairing again keeps identifiers, a restore (identifier rows deleted) and a master key rotation keep them.

15. Manifest rules, approval and the probe

  • 15.1 internal/integration/manifest/validate.go, a pure function of a decoded manifest and the console's versions, tested one refusal at a time: provider matches ^[a-z0-9]+$ and is the connection's; every key is prefixed <provider>_ and unique; recipient is person or workspace; a person's key is a switch; a limit carries unit and a switch none; enforced_by is console or application, order one of least_recently_used, newest, oldest, everything, through names a switch of the manifest, none of them on a switch; steps non-empty and absent on a switch, restrictions first, at most one reclaim and last, each step after but the first, a reclaim delete_after, lengths above 0, ids matching the grammar, unique within the key, and never keep, null, undefined, constructor; keys is non-empty and on_forget is present and has value delete or detach and a sentence; identity from email and oidc_subject; max_read_back_page of 1 or more; allowance and balance refused; no unlimited or default value.
  • 15.2 announce endpoint: decode the manifest strictly from the raw body, refuse (with a signed answer naming the problem's class and the first field at fault and never the value, and for a version the versions the console speaks, spending no pairing attempt) a manifest with an unknown field at any depth, an unknown value in any list, a protocol_version above the console's, a missing or zero required value, a number below its minimum, a class outside the closed list or repeated, operations or states without lifecycle, list or describe listed, and every other refusal of task 15.1; a refused announcement stages nothing and leaves the approved and staged manifests as they were; a manifest differing from the approved one is staged and replaces any staged one, one equal to the approved one stages nothing, one equal to the staged one changes nothing, and each answers staged, unchanged or refused; a first manifest waits; an announcement ends the resend waits of the connection's records (desired-state-relay). Tests per scenario of connector-manifest and protocol-transports.
  • 15.3 internal/integration/manifest/approve.go, the approval transaction: write one core.connection_approvals row of kind first_pairing or manifest_change, replace the approved core.connection_manifests row with the staged one, write the keys to core.connection_resource_keys, register a first approval's provider, operations, states and keys through task 12.3, and set approved_at, putting proof_address into client_address where a trusted proxy reported it, in one transaction; the trigger then writes one connection outbox row and a record whose content the approval does not change keeps its generation. The approval is refused, leaving the manifest staged and naming each such key or the class on the page, for a manifest that declares a key of kind limit (until #172) and for one that declares observe_only (until #174). A refused approval writes no approval row, no probe row and no stamp. Tests: an approval is recorded, a key left out is counted by the recipients whose record holds it, a changed key list writes one outbox row, an approval that changes no key raises no generation, a limited key and observe_only stay staged, a manifest of switches is approved, no limit entry reaches an outside connector, and a console that speaks fewer versions later keeps the connection on what was approved.
  • 15.4 internal/integration/wire/probe.go: run the specification's probe function (6.14) against a connection over the transport the connection uses, with the reserved recipients only (records are built in memory and never stored in core.desired_state_records or a projection), a record carrying the reserved field at every level of each message; write one core.connection_probe_runs row for each run with its outcome, the case identifiers run and failed and the protocol version the probe spoke; a failed run leaves the manifest staged, shows each failed case by identifier with its group's capability and sends the connector no record for the staged manifest; a passing run lets the approval of 15.3 stand and its row names the approval. Under pull, probe items ride the feed as rows naming the probe run and the case identifier, from which the console builds the item when it is served (protocol-transports). Tests with the harness fixtures: a connector that keeps an old grant, one that refuses the reserved field, one that passes, and a failed run recorded without an approval.
  • 15.5 The re-check at each announcement after approval: once, push or offer in the feed (as a row naming the probe run and the case, task 15.4) a record the connector has already applied with the reserved field added, show a refusal on the connection's page, and let no other push carry the reserved field. Tests: a connector that refuses it shows a refusal, an ordinary push carries no reserved field, a first announcement before approval sends nothing.

16. Relay, push, read-back, feed and full read

  • 16.1 internal/workflows/desiredstate/relay.go and record.go: addresses are person and workspace; expansion at every approved, non-retired connection whose key list holds a key of the matching recipient (and the row's key), reading the recipient when the row is expanded: a pool row reaches the workspaces whose active primary assignment names the pool for a workspace key and the owner and active members of the pool's organization for a person key; an organization row its workspaces, owner and active members; a workspace row that workspace; a person row that person; a recipient_only row its own recipient; a connection row every workspace whose primary pool holds a workspace key, every person who would hold a person key and every recipient that already has a record; no row reaches an organization; a row that reaches no record completes. Tests per scenario of desired-state-relay (several rows recompute once, each row kind, a platform key reaches nothing, a pending or retired connection is not reached, a suspended connection's records stay current, a connection row fills every record, a change straddling the recipient stamp still reaches its record) and the relay tests of relay_db_test.go updated from organizations to workspaces, including the Discourse parity test and the fedwiki_sites limit reaching the workspace's record.
  • 16.2 The recompute under the record rules in record.go: advisory lock on (connection, recipient), record row, content in a new statement, then update with core.next_generation when the hash differs or a floor demands, insert only when the content lists a key that is on or a limit above 0 or a floor exists, never delete a record, never take the materialization lock or lock a source row; a recompute that fails writes a recipient_only row through core.requeue_desired_state for the person or workspace. Tests per scenario: unchanged writes nothing, changed raises, a recipient who never held anything gets no record, a revocation keeps the record and lists the key off or at zero, concurrent recomputes read after the lock, an interleaved grant and revoke end with nothing held, a recompute does not wait on a rule change.
  • 16.3 Push dispatch in dispatch.go: dispatch for a touched record only when it is not delivered (no projection that reached its generation with a matching hash and no mark, and none of unlinked, linked_elsewhere or forgotten at its current generation), has no running resend wait, its connection is approved and active, its provider is active and a transport serves the connection; under push a workflow desired-state-push-<record_id> with the record identifier as its only input (a dispatch onto a running one counts as success), under pull a feed item and no workflow; at most 100 per tick; each stamps last_dispatched_at; a connection with no transport gets records and no pushes, no feed item and no lag. Tests per scenario of desired-state-relay (changed record pushed, offered under pull, suspended connection, suspended provider, delivered, waiting for an account, linked elsewhere, in its wait, running push, cap, no transport and no lag), a test that nothing is sent to a connection at another 0.y, and a test that a pull connection's transport performs no outbound dial (a dialer that fails the test).
  • 16.4 Delivered, waiting and lag are computed from the record and its projection by the queries of 10.3 and 10.4 and never stored. Tests per scenario of observed-state-read-back: a matching generation and hash is delivered, a matching generation with another hash is not, a marked report is not, an unlinked or linked-elsewhere answer for the current generation is waiting, a failed answer and an undelivered record of a served connection are lag, a connection without a transport has no lag, a stale answer is not delivery.
  • 16.5 Push answers in push.go (writePushAnswer and its applied, unlinked, divergent handlers): each answer takes the effect of desired-state-relay's list: applied at G stores generation, hash and entries, clears the error and clears or sets the mark, and an applied that names another generation is a protocol error with the projection unchanged and a retry; failed (naming G' below G) stores applied_generation G', result failed for G and sets not_carried_out_since when unset, leaves the record and ends the round in a retryable error, and failed naming G or higher is a protocol error; unlinked and linked_elsewhere store result for G with the entries and next_action and push no more for G; forgotten is stored, shown and never acted on by erasing; stale above the record and conflict at G raise the record with the reported number as a floor and push again; a reported number above core.generation_ceiling() is a protocol error, stored only as ahead_generation; stale at or below G is a protocol error; an undefined outcome, a refused answer and a transport error leave the projection unchanged, retry and show it; two consecutive stale or conflict rounds end the execution as lag, and any other answer that keeps the loop going resets the count. Tests per scenario of the requirement, including that a stale answer is never delivery and applied_generation is never lowered.
  • 16.6 Push budget in push.go and constants.go: retry from one second doubling to ten minutes within one hour; each call bounded by the connection's push timeout (read per call from the connection row, so a saved setting governs the next call); an ErrPermanent stops at once; a spent budget writes last_push_error (the transport's class and the console's own text) and last_push_failed_at; lag is computed from records and projections and never stored. Tests per scenario: permanent error, spent budget, application text never reaching the record, a connector's Retry-After sets the next delay, above five minutes counts as five, a call bounded by the connection's timeout.
  • 16.7 The resend wait: when an execution ends with its budget spent and every answer an unchanged failure (a failed answer whose entries equal the previous answer's stored for the same record and generation), set push_wait to 15 minutes when NULL and otherwise twice its value up to six hours, and next_attempt_at to its end from the execution's end; no dispatch (tick, sweep or read-back) starts before next_attempt_at; the wait ends at once, with both columns NULL, on a generation rise, an answer or read-back item whose entries differ, an operator's send again, approval of a manifest change, pairing again, the connector announcing again and the connection becoming active again (a test for each of approval, announcement and reactivation). The numbers are constants beside the relay's with their reasons. Tests per scenario: first wait, doubling to a cap, a first failure slows nothing, a changed answer, a new generation, a read-back inside the wait, pairing again for every record.
  • 16.8 The lag sweep (relay.go): list records needing a push at connections a transport serves, excluding records dispatched or failed within 15 minutes and records whose next_attempt_at is in the future, ordered with never-dispatched first, within the per-tick cap, as a push under push and a new feed item under pull; skipped when no connection has a transport. Operator send again: one record, or every record of a connection, clears the chosen records' waits and dispatches at once whether or not the record is delivered. Tests per scenario of the three requirements (cooldown, recent dispatch, resend wait, rotation past the cap, no transport and no sweep, one record, every record).
  • 16.9 internal/workflows/desiredstate/readback.go: the workflow takes a connection identifier (and after continue-as-new the run row's identifier), pages through the connector's read_back with limit the smaller of 200 and the manifest's max_read_back_page; the page checks its connection (approved, active, provider active) before calling the application and ends the run otherwise; each item is looked up by the identifier minted for it at that connection (an unknown recipient or one of a kind the provider owns no key for is counted and not stored, an item of an undefined kind is counted as refused), written under the never-lower rule and compared with the record (above the record's generation or the same generation with another hash follows the raise-above rule; below dispatches a push after commit obeying the resend wait; the same with the same hash changes nothing unless the report rule of 16.10 marks it); the cursor lives in core.read_back_runs and workflow input, result and history carry identifiers and counts only; the workflow continues as new after a fixed number of pages; under pull a page is a feed item answered when the connector posts it, and a page left unanswered for one read-back interval ends the run; a page's items are placed at the moment its identifier was first sent; the count of fields discarded on each page is logged with the connection and the operation and nothing the connector chose. Tests per scenario of observed-state-read-back including history-holds-no-content and an unknown recipient.
  • 16.10 The report rules in readback.go and push.go: a read-back item and an apply answer carry per-entry counts and granted with presence; a zero count is stored, an absent, null or negative one changes nothing stored, a null or absent granted is not reported; an item refused for a missing count is counted in items_refused; applied_generation of 0 is accepted; on an applied answer or an item at the record's generation, set not_carried_out_since (keeping the first time) when an on/off entry's reported granted differs from the record's, with no rule that compares a limit entry's counts with its instruction (those come with the console's limits), clear it when every reported granted matches, handle a marked applied as failed and a marked item as a push that obeys the wait; the page's key names are computed from the stored entries and the record by the same function. Tests per scenario of observed-state-read-back.
  • 16.11 internal/workflows/desiredstate/schedule.go and boot.go: ensure one Temporal schedule per connection desired-state-read-back-<connection_id> with the connection's read-back interval (15 minutes by default) and skip-overlap, only for an approved, active connection with a transport; start one run for each such connection at every boot; pause the schedule of a connection that loses its transport or is suspended or pending; an unanswered pull page ends the run after one interval; a restored database converges through the read-back; saving a new interval on the connection updates its schedule; nothing is scheduled for a connection at another 0.y. Tests per scenario of the requirement.
  • 16.12 The feed in internal/integration/wire/feed.go: POST /connectors/feed answers at once when an item is due after the position and otherwise after at most five seconds with no items; a poll without a position starts at the oldest kept item; an item for a record carries the record as it stands when served under the identifier <record_id>-<generation> (so a connector may receive one generation more than once); a position the console does not serve (older than the oldest kept item or never given) is answered position_lost with no items and never a partial feed; the connector's answers at /connectors/feed/answers take exactly the effect of the same answer under push, an answer naming an item never offered is refused and the item stays unanswered; a record not delivered after the cooldown and resend wait is offered again as a new item by the push rule. Tests per scenario of protocol-transports (an item due, an empty feed, a first poll, rows hold identifiers only, a posted answer has a pushed answer's effect, an answer to an item never offered, a failed record returns and a delivered one does not, a position older than the oldest, an unknown cursor), the read-back under pull (a page as a feed item answered when posted, a page placed at its first send, the count of discarded fields logged) and nothing offered to a connection at another 0.y.
  • 16.13 The full read in internal/integration/wire/fullread.go: POST /connectors/records serves 200 whole records per page in ascending order of recipient identifier with next_cursor naming the last identifier served, absent on the last page, and last written on every page; a request without a cursor starts a read and ends the connection's open read (its cursors stay served at the shared rate); each record present from the first to the last page is on exactly one page; under pull the first page carries feed_position taken before the read began; pages are served at no more than the connection's rate of full reads (one a second by default) by delaying an answer asked too soon and never by refusing it, and no other request of the connection waits on that rate; a cursor the console never gave is answered position_lost. Tests per scenario (every record on one page, feed_position, a cancellation landing before the last page, a second first page ending the open read, the last-page mark always written, a page asked too soon delayed) with a connection of more than 450 records.

17. Operator pages

  • 17.1 internal/server/operator_connections.go and its templates under internal/embeds/templates/: the connection's page in the provider's /operator/integrations/{integrationKey} family, recorded in docs/operator-ia.md, for a session with the operator role and no other: each connection's display name, status and approval state labelled as registry values and never as health (docs/models/provider-integration.md invariant 15), its identifier only where a pairing code carries it, its approved_at, and the time its current key last signed; no page a member can open shows a connection's identifier, status or approval state. Render tests with htmltest and a database test per scenario of provider-connections ("An operator sees each connection's state", "A member page shows no connection").
  • 17.2 The connection page's state lines, written under the copy rule of AGENTS.md (no sentence that explains the interface; a fact only): the version the connection speaks, the console's own version and any approved manifest written above the console's, the last verified contact of the current key, the count of messages refused under a replaced key, the count of items refused since the last complete read-back, a full read that is open with its start, a refused connector answer (oversized or refused by a check), the refusal of the re-check of 15.5, and a connection at another 0.y with the reason it receives nothing, and the operator's edit of the connection's address (checked by 11.3, the key kept). A test per line with a seeded fixture, and a golden.
  • 17.3 The connection's settings: one declared form operator.connection.settings of the settings kind in the table family, one row per setting in the order read-back interval, push timeout, rate of full reads, each with its label, effective value and whether it is the default or set; the controls are text controls carrying 1h30m for a duration and 1 for the rate as the only placeholder; a write follows the operator panel's HTMX shape, requires the operator role and names the connection only by the route's connection identifier; an unchanged value is no change; a non-duration or non-integer is refused with 422 and the settings page's parser sentence; 0 or below is refused with 422 under its control; an emptied control is refused with 422 "Enter a value, or clear the setting." when a value is set and "Enter a value." when none is; a refused write re-renders in place with every submitted value; a set setting carries a clear control with no confirm modal; success navigates by the one helper every navigating success uses. A saved setting governs the next run it applies to with no restart line and no pending-restart marker. Tests per scenario of integration-settings (defaults, two connections differ, clearing restores, a word is not a duration, zero refused, a rate below 1, an untouched save writes nothing, the route names the connection, a non-operator refused, the push timeout and interval and rate applying to the next run) saving an interval updates the connection's schedule, and a form golden.
  • 17.4 The pull pairing page: an operator starts a code from a connection; the page shows the code as https://<origin>/connectors/pair/<connection identifier>#k=<key>&s=<secret> with the secret as two groups of four, its lapse, and the count of refused attempts at the code; after a proof arrives and before confirming, it shows the connector's manifest (the version, display_name, classes, each key with holder and kind, and on_forget with its sentence), the time the proof arrived as a clock time and an age, and the address it came from only where a trusted proxy reported it; Confirm and Decline are declared forms; a code ended by a second key shows "Two keys proved this code. Make a new code." and no confirm control. Tests per scenario of connector-pairing (the time as clock time and age, no trusted proxy and no address, the address read from the right, a declined pairing and a lapsed code change nothing) and goldens.
  • 17.5 The push pairing entry: a declared form with the address field (checked against the allowlist, 422 under the control for a refusal that names the rule) and the code field with automatic correction, capitals and spell checking turned off; the result shows the manifest as in 17.4 before confirming. Tests: an address outside the list refused, a secret too short or holding U refused with no call, the form's field attributes.
  • 17.6 Pairing again: from the connection's page, a retired connection included, the confirmation shows the new address beside the old one (under pull, the address the proof came from where known), when the old key last signed a message the console verified, and the line "Records stop reaching the old connector. What its application holds stays as it is." with Confirm and Decline; the connection's identifiers and records are unchanged. Tests per scenario of connector-pairing (identifiers stay, the confirmation shows what changes, both keys work until confirmation, the old key is refused after, a retired connection accepts nothing, a differing manifest is staged).
  • 17.7 The approval page: the staged manifest's version, display_name and classes, each key's name, holder and kind and on_forget with its sentence; for a change each key added, left out and changed in holder or kind, and for each key left out the number of recipients whose record holds it; a limited key or a class observe_only named as the reason the manifest stays staged; each failed probe case by identifier with the capability of its group; Approve is a declared form that runs the probe. Tests per scenario of connector-manifest ("An approval is recorded", "A key left out is counted", "A limited key is not approved", "observe_only is not approved yet", "A failed probe leaves the manifest staged") and goldens.
  • 17.8 Suspend and resume as declared forms on the connection's page, and the list of records not carried out: each record whose projection carries not_carried_out_since with "not carried out since" and the time, "next attempt" and next_attempt_at (or the end of the 15-minute cooldown when no wait runs), each key whose counts or granted caused the mark by the display name the connection's key list gives it and no key when none does, and a send-again form for one record and for every record of the connection (task 16.8). Tests per scenario of desired-state-relay ("A record not carried out is listed with its times and key", "A mark that names no key", "One record is sent again", "Every record of a connection is sent again").
  • 17.9 internal/server/operator_persons.go and the person page template: "Cut off" while the person has no open cut-off and "Lift cut-off" while it has one, to the operator role only; "Cut off" opens /operator/persons/{personID}/cut-off, a confirmation whose button "Cut off" calls core.set_person_cut_off(P, TRUE, A) with the operator's own person; "Lift cut-off" calls core.set_person_cut_off(P, FALSE, A) at once with no confirmation; each is a declared form and answers a redirect to the person's page, which shows the state the database holds; no field for a reason or a note; a second confirmation changes nothing. This handler is the only production caller of the function. Tests per scenario of person-cut-off (the act that fits the state, confirming opens a cut-off under the operator's name, lifting closes the row, a member cannot reach the acts, a second confirmation changes nothing, the only production caller is the handler through the source test of 17.11).
  • 17.10 What a cut-off reaches, on the confirmation and on the person's page while the cut-off is open, in three lists with the headings "Connections that may still let the person in" (each connection whose approved key list holds a person key, by display name, and each connection whose transport is NULL whatever it delivers), "Organizations that stop delivering" (each organization that counts as not active because the person is its only active member; under each, one to a line, "{key} turns off" for an on/off key its workspaces hold, "{key} falls to 0" for a limit key, "{subscription} keeps billing" for a subscription of the organization that keeps billing; a key by its manifest's display name; the keys of a connection without a transport not named) and "Organizations that keep delivering" (each other organization the person owns or belongs to through an active membership at which an approved manifest declares a workspace key); the page and the confirmation also state when the cut-off began and who opened it; a list with no entry does not render; the page changes no subscription. The word on the page is "organization". Tests per scenario of person-cut-off ("names a connection with a person key", "own organization stops delivering", "a subscription that keeps billing is named", "an organization with other members keeps delivering", "an open cut-off shows the same lists", "an empty list does not render") and goldens.
  • 17.11 internal/workflows/desiredstate/*_test.go payload and replay tests: the relay and push workflows' inputs and activity results carry identifiers, generation numbers, feed positions, outcomes and counts only, and no content, observed state, name, email or subject (payload_test.go extended and a push history inspected); source_test.go keeps its rules (no file under internal/integrations/ touches the outbox or writes records, no non-test Go file writes core.target_mappings since it no longer exists, and core.set_person_cut_off has one production caller, the handler of task 17.9).
  • 17.12 The hide-limit help: change the field's help in internal/server/operator_entitlement_set_forms.go (near line 261) and the column header in internal/embeds/templates/partials/operator_entitlement_set_rules_table.html (near line 74) from "Shown to members once usage reaches it." to "Shown to members in the console once usage reaches it."; edit the expected lines into the two form goldens under internal/server/testdata/forms/ by hand first and the test internal/server/operator_entitlement_set_rules_test.go; and add the tests for entitlements ("A mark-only change writes no outbox row", "The delivered content names no mark", "No message to a connector names the mark": a scan of every message the console builds for a connector in the tests of 13 and 16).
  • 17.13 Registration of the pages with the console's checks: add each new form to the forms registry and the registered-forms table of docs/design-system.md; add a screen-manifest entry (as a screen or a Skip with its reason) for every new GET page route in test/e2e/screens/manifest.go and a walkthrough claim for every new operator mutation route in test/e2e/operator-walkthroughs/ (a connections_test.go for the connection pages and an addition to persons_directory_test.go for the cut-off), so screen-coverage and walkthrough-coverage pass; none of the copy contains a phrase the filler-copy rule lists or an em dash.
  • 17.14 make lint passes with the new pages and forms, and make screens runs and the maintainer reviews the sheets and the changed-screen list; the handoff says what to look for in each sheet and classifies each changed screen as real or drift by pixel diff. make screens-accept is never run by the implementer. The states make screens cannot reach (a pairing code ended by a second key, a pairing again with its confirmation, a failed probe, a staged manifest refused for a limited key and for observe_only, a record not carried out with its key, an open cut-off with all three lists, a settings refusal) are captured by hand at 1280 and 390 wide through the walkthroughs' WALKTHROUGH_STATES_DIR, with the viewport checked by innerWidth.

18. The test stack serves connector connections over HTTPS

  • 18.1 test/bootstrap-stack.sh and test/compose.yaml: generate a test certificate authority and a server certificate for the console's connector origin and for the bridge, never committed and removed by teardown-stack.sh; add a TLS-terminating service (the existing caddy image with its own profile-independent block) in front of the console's connector paths and the bridge's endpoints on collision-free host ports derived as the other ports are; write connector-origins, connector-private-origins (with =<file> naming the test authority, since the stack's addresses resolve to loopback) into test/.env as MC_* variables, and set MC_BASE_URL to the TLS service's https origin for every client, with the Keycloak redirect URI and the e2e, walkthrough and screens base URLs following it, because the pull code's origin is base-url's and the same key drives the OIDC redirect, the cookie's Secure flag and the trusted origin (internal/config/transport.go:12-21; the YAML line is test/mc-config.yaml:17). The script stays idempotent.
  • 18.2 Script tests in test/ (beside scripts_env_test.go and scripts_args_test.go): the generated .env holds the three keys and the authority path, a second run changes nothing, teardown-stack.sh removes the authority, two worktrees get distinct ports, and verify-stack-isolation.sh accepts the new ports. A Go test starts the console's pairing endpoint behind the TLS service and a harness connector fixture that trusts the test authority and completes a pull pairing and a push pairing over HTTPS.
  • 18.3 test/AGENTS.md and docs/testing.md describe the connector HTTPS service, the three keys, the test authority and that the authority is trusted for the run alone.

19. The Discourse bridge in the test stack

  • 19.1 Skeleton of the bridge as a package of the console's module at connectors/discourse/ (design.md, ruling 47): a Go program that uses the specification's helpers (not the sidecar, so a real connector exercises the helpers' signing, retry and apply loop), takes the pairing code as an argument under either transport, keeps its store in a file under the connection identifier, and talks to the forum with the existing client's calls (internal/integrations/discourse/client) through an interface the bridge owns so its tests run against internal/integrations/discourse/discoursetest; it takes its provider key from the required flag --provider, with no default, and refuses to start without it. A test lists the bridge's imports and fails on any package of the console's module other than internal/integrations/discourse/client, internal/integrations/discourse/discoursetest (in tests) and internal/httplimit, with the allowed list kept in one place.
  • 19.2 The bridge's manifest under the provider key it is given (testforum in the test stack): protocol_version of the build, classes: [], one person switch key <provider>_posting, identity email, max_read_back_page, and on_forget with detach and its one sentence; the key is registered only through the pairing and approval of the test stack, so production's registry never holds it.
  • 19.3 The bridge's apply: for a person record it links the forum account only by the email the console sent, only while the record's switch is on, keeps an account with the recipient first linked and answers linked_elsewhere for another, answers unlinked while no forum account exists, adds the member to and removes the member from the bridge's own group (a group of its own that lets nobody join by themselves, and never an automatic group or one that confers administration), carries out every entry that can be carried out, answers failed when the forum refuses, and answers stale below the highest generation begun; every answer carries its counts with presence. Tests with the fake forum: grant, revocation, a second recipient with the same email, no account yet, a rejected forum write.
  • 19.4 The bridge's read-back, store and forget: the read-back reports granted for each recipient from the forum group's membership; after its store is deleted and it is paired again onto the same connection it reads every record, takes no account out before the last page, makes no link twice and reports the same held state; a forget removes the person's identity from the store and detaches the forum account (removes it from the bridge's group and keeps only the identifier), answers forgotten for it and for every repeat and for a later record naming it, and a pending link never lands; the bridge ends no access when it has not heard from the console. Tests per scenario of connector-obligations that apply to a person switch.
  • 19.5 Seeds only where the stack composes the forum: a seed under test/seed/ (run by test/seed-stack.sh only when the discourse profile is on) starts the bridge with --provider testforum, registers that provider through its pairing and approval and writes the plan rules that fund testforum_posting, so that member pages and screens elsewhere do not change and a stack without the forum shows no trace of the key; test/AGENTS.md and test/README.md say so. A script test in test/ (beside scripts_env_test.go) checks that the seed is skipped without the profile.
  • 19.6 The end-to-end test under test/e2e/ (skips unless the stack is up with the discourse profile and the bridge running), first half: pair under pull over HTTPS, approve (the probe passes), grant the plan and see the forum account join the bridge's group, revoke and see it leave; the console's pushes, answers and read-backs are asserted from core.observed_projections. The in-tree Discourse integration is not configured in this stack.
  • 19.7 The end-to-end test, second half: cut the person off through the page of 17.9 and see the person's access end, pair again and see the same identifiers, delete the bridge's store and see convergence.

20. Documents and upstream

  • 20.1 docs/models/provider-integration.md: amend invariant 16 (a key's recipient is a person, a workspace or empty; a numeric key for a person stays refused) and invariant 17 (target mappings no longer exist), and add the connection tables, the pinned key, the approvals and the connection's settings; docs/models/identity-organization-workspace.md: an organization made not active by a cut-off (the sole active member rule), workspaces as the recipient of workspace keys.
  • 20.2 docs/building-an-integration.md: a section for an outside connector (the protocol's repository, pairing, the manifest, the minimum connector's six duties, the harness and the sidecar, no Temporal on the connector side) and the one-line change that an installed integration writes nothing once its connection has a transport; docs/connector-candidates.md: replace the line that says no transport exists; docs/discourse-setup.md: the bridge's setup in the test stack and the forum's rate limits; docs/environment-reference.md and docs/settings-and-configuration.md: the three deployment-only keys and the master key (task 11.1); docs/README.md lists any page this change adds; docs/database-locks.md and the lock-families lint gain any advisory-lock family a task above introduced.
  • 20.3 status/design-feedback.md: add the upstream amendment ask for the design record's Decisions 125, 133 and 147, in the entry format the file uses (a heading, the design-feedback and upstream-membcons-db labels, the filing date and numbered asks): Decisions 125 and 147 (design document 47, sections 7.1 and 7.3), the reduction policy governs only edits of its own rule and no longer sets what a connector does (design.md, rulings 6 and 14); Decision 133, the half that has the console hold each site's desired state so the member chooses there is set aside for outside connectors (design.md, ruling 9).
  • 20.4 Write the deploy notes for the commit that lands the migrations (the maintainer commits after review): a deployment re-runs scripts/db-reader-role.sql after migrating; the three configuration keys and the master key are optional until a connector is paired; the migration deletes records and projections addressed to organizations and recomputes workspace records; a rollback of a release that has paired a connector keeps the identifiers it gave. Task 22.3's check runs empty before the commit.

21. The done criterion

  • 21.1 Publish protocol 0.1 of the specification repository (after 1.5's review), with the harness, the helpers and the sidecar built from it; record the harness's passing runs for the reference connector, the sidecar and the Discourse bridge on each transport they support, with the report files kept in the repository's reports/ directory (no pairing code or key in them).
  • 21.2 A connector written from the specification repository alone, by someone other than its author, passes the harness core on each transport it supports with status 0: give a weak coding agent, chosen by the maintainer, only the repository and the instruction to write a connector for a small application of his choice, and run the harness. Each point the agent got wrong is a defect of the specification (its text, vectors or helpers) and is fixed there, never worked around in the harness.
  • 21.3 Repeat 21.2 with a fresh agent after each fix, for at most three attempts; record for each attempt the number of harness runs, every point got wrong and the outcome. The final passing report is the release's evidence under connector-obligations and conformance-harness ("A connector passes the core only when no core case failed"); if the third attempt does not pass, the maintainer decides what the specification needs before a fourth.

22. Final checks

  • 22.1 openspec validate integration-contract-wire --strict passes.
  • 22.2 make test passes with the test stack up (cd test && ./bootstrap-stack.sh && docker compose up -d first), including make lint, every package, and the specification repository's own CI; the console's tests against the vectors of 13.9 run on the released module version.
  • 22.3 git grep -i "$(printf 'fu%s' 'laus')" is empty (the pattern is built so that this file does not hold the name it guards) and scripts/notebook-citations.sh passes.
  • 22.4 The deploy notes of 20.4 are in the landing commit, with the line that a deployment re-runs scripts/db-reader-role.sql after migrating, and the maintainer has reviewed the make screens sheets and the hand-captured states of 17.13.