Message type considerations #2

New Issue

glyph · 2021-12-17T09:55:00Z

glyph commented

2021-12-17 09:55:00 +00:00

I'm using this issue to share some background knowledge on SSB data types and to document my thoughts about types in golgi.

Messages in SSB are formatted as key-value-timestamp (KVT) objects. The value portion of the message must contain the following fields: previous, author, sequence, timestamp, hash, content and signature. The content field contains different values depending on the message type. For example, a post message is expected to have a text field and a mentions field. Anyone is free to create new content types.

So, in golgi, we need a way to represent a KVT (SsbMessage) and a value (SsbMessageValue).

Kuska provides a Feed type and a Message type:

pub struct Feed {
    pub key: String,
    pub value: Value,
    pub timestamp: f64,
    pub rts: Option<f64>,
}

pub struct Message {
    pub value: serde_json::Value,
}

Feed implements an into_message() method which returns Result<Message>. The downside is that this results in the message value being validated before the Message type is returned. Since we're relying on go-sbot to perform validation, we want to avoid this step (it is computationally intensive, relatively speaking).

We need to consider these types in the context of end-user convenience; returning a generic JSON Value type leaves lots of work to be done before it is useful in an application. Ideally, I imagine we want to check the type field of a message value content object and serialize the content to a matching struct. Kuska provides some content types which might be useful.

I'm using this issue to share some background knowledge on SSB data types and to document my thoughts about types in golgi. ----- Messages in SSB are formatted as key-value-timestamp (KVT) objects. The `value` portion of the message must contain the following fields: `previous`, `author`, `sequence`, `timestamp`, `hash`, `content` and `signature`. The `content` field contains different values depending on the message type. For example, a `post` message is expected to have a `text` field and a `mentions` field. Anyone is free to create new content types. So, in golgi, we need a way to represent a KVT (`SsbMessage`) and a value (`SsbMessageValue`). Kuska provides a [`Feed` type](https://github.com/Kuska-ssb/ssb/blob/master/src/feed/base.rs#L11) and a [`Message` type](https://github.com/Kuska-ssb/ssb/blob/master/src/feed/message.rs#L56): ```rust pub struct Feed { pub key: String, pub value: Value, pub timestamp: f64, pub rts: Option<f64>, } pub struct Message { pub value: serde_json::Value, } ``` `Feed` implements an `into_message()` method which returns `Result<Message>`. The downside is that this results in the message value being validated before the `Message` type is returned. Since we're relying on go-sbot to perform validation, we want to avoid this step (it is computationally intensive, relatively speaking). We need to consider these types in the context of end-user convenience; returning a generic JSON `Value` type leaves lots of work to be done before it is useful in an application. Ideally, I imagine we want to check the `type` field of a message value `content` object and serialize the `content` to a matching `struct`. Kuska provides some [content types](https://github.com/Kuska-ssb/ssb/blob/master/src/api/dto/content.rs) which might be useful.

glyph commented

2021-12-17 10:20:27 +00:00

Here's an example of how this might play-out at the low-level in golgi:

// create a post
let post = TypedMessage::Post {
    text: "protein synthesis".to_string(),
    mentions: None,
};

// publish the post (returns the msg key)
let msg_ref = sbot_client.publish(post).await?;

// get the msg (as kvt) for the post we just published, using the returned key
let msg_kvt = sbot_client.get(msg_ref).await?;

// get the value of the msg kvt
let msg_value = msg_kvt.value;

println!("{:?}", msg_value);

Output from println:

Object({"previous": String("%LFh2KEWqPvUdu+QKWjyBBjsm9snEqelE7boesJsBuxg=.sha256"), "author": String("@1vxS6DMi7z9uJIQG33W7mlsv21GZIbOpmWE1QEcn9oY=.ed25519"), "sequence": Number(7), "timestamp": Number(1639732871000), "hash": String("sha256"), "content": Object({"mentions": Null, "text": String("protein synthesis"), "type": String("post")}), "signature": String("s3cpn2p4JEwL6M9Kfn6s7lcZeeACzhKHAYpY4eU5WwmVlls1y+Uft694CuvW5hM2doexrMMb5WvuQogjUuzdBg==.sig.ed25519")})

The next step would be to deserialize the msg_value.content into the appropriate struct using serde_json::from_value.

We might match like so:

// get the value of the `type` field in the `content` object
let content_type = msg_value.content.as_str("type").ok_or("no type field found")?;

match content_type {
    "post" => let p: Post = serde_json::from_value(msg_value.content)?,
    "contact" => // ...,
    "about" => // ...,

Of course, we need to comply with the borrow checker...

Here's an example of how this might play-out at the low-level in golgi: ```rust // create a post let post = TypedMessage::Post { text: "protein synthesis".to_string(), mentions: None, }; // publish the post (returns the msg key) let msg_ref = sbot_client.publish(post).await?; // get the msg (as kvt) for the post we just published, using the returned key let msg_kvt = sbot_client.get(msg_ref).await?; // get the value of the msg kvt let msg_value = msg_kvt.value; println!("{:?}", msg_value); ``` Output from `println`: ```json Object({"previous": String("%LFh2KEWqPvUdu+QKWjyBBjsm9snEqelE7boesJsBuxg=.sha256"), "author": String("@1vxS6DMi7z9uJIQG33W7mlsv21GZIbOpmWE1QEcn9oY=.ed25519"), "sequence": Number(7), "timestamp": Number(1639732871000), "hash": String("sha256"), "content": Object({"mentions": Null, "text": String("protein synthesis"), "type": String("post")}), "signature": String("s3cpn2p4JEwL6M9Kfn6s7lcZeeACzhKHAYpY4eU5WwmVlls1y+Uft694CuvW5hM2doexrMMb5WvuQogjUuzdBg==.sig.ed25519")}) ``` The next step would be to deserialize the `msg_value.content` into the appropriate `struct` using [`serde_json::from_value`](https://docs.serde.rs/serde_json/fn.from_value.html). We might match like so: ```rust // get the value of the `type` field in the `content` object let content_type = msg_value.content.as_str("type").ok_or("no type field found")?; match content_type { "post" => let p: Post = serde_json::from_value(msg_value.content)?, "contact" => // ..., "about" => // ..., ``` Of course, we need to comply with the borrow checker...

glyph commented

2021-12-17 14:35:18 +00:00

Deserializing messages into the correct struct is made simpler in cases where we know what type of messages we'll be receiving. The getSubset queries are a good example of this; if we query for {"op": "type", "string": "post"} then we can simply deserialize the results into Vec<Post> (with an appropriate error for a deserialization failure).

I think most of our queries will match the above pattern - where we know what we're expecting - especially for the first release of PeachPub.

Deserializing messages into the correct `struct` is made simpler in cases where we know what type of messages we'll be receiving. The `getSubset` queries are a good example of this; if we query for `{"op": "type", "string": "post"}` then we can simply deserialize the results into `Vec<Post>` (with an appropriate error for a deserialization failure). I think most of our queries will match the above pattern - where we know what we're expecting - especially for the first release of PeachPub.

notplants referenced this issue

2021-12-28 20:08:11 +00:00

Add SsbKVT, SsbMessageValue and SsbMessageContent Types #6

glyph referenced this issue

2021-12-29 14:02:49 +00:00

Add SsbKVT, SsbMessageValue and SsbMessageContent Types #6

notplants commented

2022-01-13 19:40:00 +00:00

Closing this since PR above is now merged.

notplants closed this issue

2022-01-13 19:40:01 +00:00

Sign in to join this conversation.