An application outside the console cannot connect to it: there is no published protocol #171

Open
opened 2026-10-06 18:11:39 +00:00 by cgalo5758 · 5 comments
Owner

What a person cannot do today

Connect an application to the console without adding code to the console's repository. The console now stores what each application should hold, one desired-state record per connection and recipient with a generation that rises when the content changes, but nothing sends those records anywhere: there is no transport, no way for an application to register, and no specification to write a connector against.

What they should be able to do

Write a connector in any language against a published specification, register it with a deployment, and receive the desired state of every person and organization it serves.

  • The messages are defined in Protobuf and served through Connect, so one definition answers as JSON over HTTP (the required binding), gRPC and gRPC-Web. Field names on the wire are snake_case; OpenAPI and JSON Schema files are generated. The specification lives in a separate repository under Apache-2.0.
  • The console pushes records, and a connector may also pull them. Temporal stays inside the console; a connector does not import a Temporal SDK.
  • A connector announces a manifest. A deployment allowlists connector origins, with patterns such as *.example.org. Pairing uses a one-time code. Requests are signed in both directions with ed25519, naming the connection and the operation. A manifest change is staged until an operator approves it, and approvals are recorded.
  • The minimum connector declares a manifest, signs and verifies, applies desired state and answers the read-back. Every other capability is an optional class it declares; forget is required whenever it asks for personal data.
  • The read-back interval and the push timeout become operator settings on each connection, and the read-back page size is capped by a maximum the manifest declares.
  • An operator can cut a person off at every connection.
  • A numeric entry always carries a limit of zero or more; the contract has no unlimited value (#164).

Why it matters

A deployment must not have to fork the console to connect its application, and each application should need one connector instead of code on both sides. This is the first of four pieces of work that make up the integration contract, all due before launch; about 38 developer-weeks of it remain at the midpoint, between 22 and 55.

Where

internal/integration (registration, the transport interface), internal/workflows/desiredstate (push and read-back exist but have no transport), a new specification repository, and the operator's connection pages.

Done when

A connector built only from the specification repository, by someone other than the specification's author, passes the core of the conformance harness: it pairs with a deployment, receives signed pushes of its records and answers the read-back. An operator can approve its manifest, suspend it and cut a person off from the console.

Order

Comes after #164 and #184.

## What a person cannot do today Connect an application to the console without adding code to the console's repository. The console now stores what each application should hold, one desired-state record per connection and recipient with a generation that rises when the content changes, but nothing sends those records anywhere: there is no transport, no way for an application to register, and no specification to write a connector against. ## What they should be able to do Write a connector in any language against a published specification, register it with a deployment, and receive the desired state of every person and organization it serves. - The messages are defined in Protobuf and served through Connect, so one definition answers as JSON over HTTP (the required binding), gRPC and gRPC-Web. Field names on the wire are snake_case; OpenAPI and JSON Schema files are generated. The specification lives in a separate repository under Apache-2.0. - The console pushes records, and a connector may also pull them. Temporal stays inside the console; a connector does not import a Temporal SDK. - A connector announces a manifest. A deployment allowlists connector origins, with patterns such as `*.example.org`. Pairing uses a one-time code. Requests are signed in both directions with ed25519, naming the connection and the operation. A manifest change is staged until an operator approves it, and approvals are recorded. - The minimum connector declares a manifest, signs and verifies, applies desired state and answers the read-back. Every other capability is an optional class it declares; forget is required whenever it asks for personal data. - The read-back interval and the push timeout become operator settings on each connection, and the read-back page size is capped by a maximum the manifest declares. - An operator can cut a person off at every connection. - A numeric entry always carries a limit of zero or more; the contract has no unlimited value (#164). ## Why it matters A deployment must not have to fork the console to connect its application, and each application should need one connector instead of code on both sides. This is the first of four pieces of work that make up the integration contract, all due before launch; about 38 developer-weeks of it remain at the midpoint, between 22 and 55. ## Where `internal/integration` (registration, the transport interface), `internal/workflows/desiredstate` (push and read-back exist but have no transport), a new specification repository, and the operator's connection pages. ## Done when A connector built only from the specification repository, by someone other than the specification's author, passes the core of the conformance harness: it pairs with a deployment, receives signed pushes of its records and answers the read-back. An operator can approve its manifest, suspend it and cut a person off from the console. ## Order Comes after #164 and #184.
cgalo5758 added this to the Public launch milestone 2026-10-06 18:11:39 +00:00
cgalo5758 added the
kind
enhancement
area/integrations
labels 2026-10-06 18:11:39 +00:00
Author
Owner

Order: followed by #172, then #173, then #174.

Order: followed by #172, then #173, then #174.
Author
Owner

Before the protocol fixes the shape of a lifecycle request, it has to say whether one request can cover several instances. FedWiki's swap of the active site is the case today: it parks one site and activates another as two single-site requests held together by their workflow id (#184).

Before the protocol fixes the shape of a lifecycle request, it has to say whether one request can cover several instances. FedWiki's swap of the active site is the case today: it parks one site and activates another as two single-site requests held together by their workflow id (#184).
Author
Owner

Inbound messages need one rule for personal data, and the contract is the place to state it.

Today each part of the console decides for itself what it keeps from a message. The Discourse receiver stores every event body whole, so the email addresses of everyone who signs up on the forum, members or not, sit in core.webhook_events (#163). The Stripe receiver stores the body minus a list of key names it knows. Workflow inputs carry members' names and sign-in identifiers in plain text (#162), and an error message can carry a person's email into the logs. Each of these gets fixed one list at a time.

The contract's messages are defined in Protobuf, so the definition can say, field by field, which values are personal. Protobuf already has a field option for values that must not be printed (debug_redact, meant for credentials); a field option of the contract's own can mark personal data. Then:

  • whatever the console keeps of a message, whether an inbound event record, a workflow input, a log line or an error, is written from the definition, with marked fields left out or redacted, instead of from a list each integration keeps;
  • the manifest's statement of the personal data a connector asks for can be checked against the fields it uses, and forget, which this issue already requires whenever a connector asks for personal data, covers exactly those fields;
  • a connector's author reads in one place what the console treats as personal.

#163 fixes the Discourse receiver where it stands, keeping only the forum ids processing needs, and cleans the rows already stored. That receiver goes away with #174, so the general rule belongs here.

Inbound messages need one rule for personal data, and the contract is the place to state it. Today each part of the console decides for itself what it keeps from a message. The Discourse receiver stores every event body whole, so the email addresses of everyone who signs up on the forum, members or not, sit in `core.webhook_events` (#163). The Stripe receiver stores the body minus a list of key names it knows. Workflow inputs carry members' names and sign-in identifiers in plain text (#162), and an error message can carry a person's email into the logs. Each of these gets fixed one list at a time. The contract's messages are defined in Protobuf, so the definition can say, field by field, which values are personal. Protobuf already has a field option for values that must not be printed (`debug_redact`, meant for credentials); a field option of the contract's own can mark personal data. Then: - whatever the console keeps of a message, whether an inbound event record, a workflow input, a log line or an error, is written from the definition, with marked fields left out or redacted, instead of from a list each integration keeps; - the manifest's statement of the personal data a connector asks for can be checked against the fields it uses, and forget, which this issue already requires whenever a connector asks for personal data, covers exactly those fields; - a connector's author reads in one place what the console treats as personal. #163 fixes the Discourse receiver where it stands, keeping only the forum ids processing needs, and cleans the rows already stored. That receiver goes away with #174, so the general rule belongs here.
Author
Owner

Six things are settled about how this issue, #172, #173 and #174 are designed and built.

  1. One design, four deliveries. The specification is designed once, across all four issues and the names capability, before any of it is built. The four are still built in the order they have. An operation added later does no harm, because a connector that does not declare it is never asked for it. A decision that changes what a connector does with a message it already receives is different: whether a rule's reduction policy travels in the record, how a connector learns that a target is no longer managed, and the forget message this issue requires but #172 defines.
  2. 0.x until the project's own connectors pass. The specification repository stays below 1.0 until the Discourse bridge and FedWiki both pass the conformance harness, and 1.0 is published at launch. The specification says how a console and a connector agree on a version, and what a connector written for 1.0 may rely on afterwards.
  3. The harness starts here. The core of the conformance harness is built in this issue instead of #174, and #172 and #173 add their cases to it. This issue is done when a connector written from the specification repository alone, by someone other than the specification's author, passes it. The Discourse bridge runs beside the in-tree integration, observing only, before #172 starts.
  4. A sidecar ships at launch. The project maintains a program that does the pairing, the signatures, both transports, the generation store and the read-back, so a connector's own code only exchanges plain JSON with it on localhost. The quickstart uses the sidecar with pull.
  5. Names are one case of a general shape. On the wire, the names capability reads "an instance of one key needs an allocation of another", and names are the only case at launch. The domains registry stays in core (#44).
  6. Room for allowances and balances. Every entitlement entry names its kind, and a usage figure can name the period it belongs to, so an allowance that refills and a balance that is spent can be added without breaking a connector. Neither is built for launch. The design also has to say whether a numeric limit can be delivered to a person, which the model refuses today (docs/models/provider-integration.md).

Two facts the design has to deal with:

  • Connect's Go JSON reader ignores an enum name it does not know (protojson.UnmarshalOptions{DiscardUnknown: true} in codec.go, connectrpc.com/connect v1.21.1). A connector that answers "applied" where the definition spells the value another way is read as having given no answer, with no error. ProtoJSON also writes a 64-bit integer, such as a generation, as a string. The specification pins the JSON spelling of every value, treats an unknown or unspecified value as an error, and ships test vectors.
  • A provider has at most one connection, retired ones included, and a declined pairing is recorded by retiring the connection (openspec/specs/provider-connections/spec.md). Nothing yet says how a pairing is tried again after it was declined, its connection was retired or its key was lost, or how a connector pairs into the approved connection that an in-tree provider such as FedWiki already has. This issue decides both.
Six things are settled about how this issue, #172, #173 and #174 are designed and built. 1. **One design, four deliveries.** The specification is designed once, across all four issues and the names capability, before any of it is built. The four are still built in the order they have. An operation added later does no harm, because a connector that does not declare it is never asked for it. A decision that changes what a connector does with a message it already receives is different: whether a rule's reduction policy travels in the record, how a connector learns that a target is no longer managed, and the forget message this issue requires but #172 defines. 2. **0.x until the project's own connectors pass.** The specification repository stays below 1.0 until the Discourse bridge and FedWiki both pass the conformance harness, and 1.0 is published at launch. The specification says how a console and a connector agree on a version, and what a connector written for 1.0 may rely on afterwards. 3. **The harness starts here.** The core of the conformance harness is built in this issue instead of #174, and #172 and #173 add their cases to it. This issue is done when a connector written from the specification repository alone, by someone other than the specification's author, passes it. The Discourse bridge runs beside the in-tree integration, observing only, before #172 starts. 4. **A sidecar ships at launch.** The project maintains a program that does the pairing, the signatures, both transports, the generation store and the read-back, so a connector's own code only exchanges plain JSON with it on localhost. The quickstart uses the sidecar with pull. 5. **Names are one case of a general shape.** On the wire, the names capability reads "an instance of one key needs an allocation of another", and names are the only case at launch. The domains registry stays in core (#44). 6. **Room for allowances and balances.** Every entitlement entry names its kind, and a usage figure can name the period it belongs to, so an allowance that refills and a balance that is spent can be added without breaking a connector. Neither is built for launch. The design also has to say whether a numeric limit can be delivered to a person, which the model refuses today (`docs/models/provider-integration.md`). Two facts the design has to deal with: - Connect's Go JSON reader ignores an enum name it does not know (`protojson.UnmarshalOptions{DiscardUnknown: true}` in `codec.go`, connectrpc.com/connect v1.21.1). A connector that answers `"applied"` where the definition spells the value another way is read as having given no answer, with no error. ProtoJSON also writes a 64-bit integer, such as a generation, as a string. The specification pins the JSON spelling of every value, treats an unknown or unspecified value as an error, and ships test vectors. - A provider has at most one connection, retired ones included, and a declined pairing is recorded by retiring the connection (`openspec/specs/provider-connections/spec.md`). Nothing yet says how a pairing is tried again after it was declined, its connection was retired or its key was lost, or how a connector pairs into the approved connection that an in-tree provider such as FedWiki already has. This issue decides both.
Author
Owner

The question this issue was waiting on is answered (#184): a lifecycle request covers exactly one instance. The protocol needs four rules from it:

  • A request covers one instance.
  • When one action changes several instances, the console records one request per instance and gives them one act id. The act id names a header row that records who acted, when and why. Each request's role says whether it is for the instance the person asked for or for one the console added to carry the action out.
  • The act id only ties the requests together. When one request of such an action fails, the console undoes the others itself. For example, a swap that parked one site and could not activate the other sends the parked site back.
  • So a connector applies each request on its own, and does not undo the other requests of an act when one fails.

docs/models/provider-integration.md states these as invariants. Whether a connector may instead park a site on its own, which would take the rotation rule out of the console, is a question for #172.

The question this issue was waiting on is answered (#184): a lifecycle request covers exactly one instance. The protocol needs four rules from it: - A request covers one instance. - When one action changes several instances, the console records one request per instance and gives them one act id. The act id names a header row that records who acted, when and why. Each request's role says whether it is for the instance the person asked for or for one the console added to carry the action out. - The act id only ties the requests together. When one request of such an action fails, the console undoes the others itself. For example, a swap that parked one site and could not activate the other sends the parked site back. - So a connector applies each request on its own, and does not undo the other requests of an act when one fails. `docs/models/provider-integration.md` states these as invariants. Whether a connector may instead park a site on its own, which would take the rotation rule out of the console, is a question for #172.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: wiki-cafe/member-console#171