init
This commit is contained in:
commit
c6bb3cd1fc
GPG Key ID:
03789458B3D0C410
|
|
@ -0,0 +1,4 @@
|
|||
*.yaml
|
||||
*.yml
|
||||
dist/
|
||||
rss-butt-plug
|
||||
|
|
@ -0,0 +1,137 @@
|
|||
# rss-butt-plug
|
||||
|
||||
> :peach: **e x p e r i m e n t a l** :peach:
|
||||
|
||||
A [SSB](https://scuttlebutt.nz) client which "plugs" a RSS feed into the Scuttleverse.
|
||||
|
||||
Created in the same spirit as [Lykin](https://git.coopcloud.tech/glyph/lykin),
|
||||
for educational purposes and to help others dip a toe into the SSB development
|
||||
ecosystem. It's mostly a demo and I don't necessarily intend to maintain it.
|
||||
|
||||
Here's a [short
|
||||
screencast](http://vvvvvvaria.org/~decentral1se/ssb/rss-butt-plug/rbpc.mp4)
|
||||
with a quick tour of the tool and some thoughts on it.
|
||||
|
||||
**Please** read the [Be Gentle](#be-gentle-monocle_face) section before using it on the mainnet.
|
||||
|
||||
## Scuttlin' :nerd_face:
|
||||
|
||||
Get a local copy of `rss-butt-plug`:
|
||||
|
||||
```
|
||||
curl https://vvvvvvaria.org/~decentral1se/ssb/rss-butt-plug/rss-butt-plug_linux_amd64_v1/rss-butt-plug -o rss-butt-plug
|
||||
chmod +x rss-butt-plug
|
||||
```
|
||||
|
||||
If you're not running `amd64` arch, take a look in [vvvvvvaria.org/~decentral1se/ssb/rss-butt-plug](https://vvvvvvaria.org/~decentral1se/ssb/rss-butt-plug) for binaries which suit your system.
|
||||
|
||||
Create a `rss-butt-plug.yaml` in the same directory:
|
||||
|
||||
```yaml
|
||||
---
|
||||
# where all data will be stored, is a relative path to the current working directory
|
||||
data-dir: .rss-butt-plug
|
||||
|
||||
# the RSS feed URL
|
||||
feed: https://openrss.org/opencollective.com/secure-scuttlebutt-consortium/updates
|
||||
|
||||
# the RSS feed profile avatar URL (will be converted to blob)
|
||||
avatar: https://images.opencollective.com/secure-scuttlebutt-consortium/676f245/logo/256.png
|
||||
|
||||
# the internal go-sbot configuration options
|
||||
addr: localhost
|
||||
port: 8008
|
||||
ws-port: 8989
|
||||
shs-cap: "1KHLiKZvAvjbY1ziZEHMXawbCEIM6qwjCDm3VYRan/s="
|
||||
```
|
||||
|
||||
Run it:
|
||||
|
||||
```
|
||||
./rss-butt-plug
|
||||
```
|
||||
|
||||
Wire up a local Patchwork client (or something that supports *legacy*
|
||||
replication, [see FAQ for
|
||||
more](https://github.com/ssbc/go-ssb/blob/master/docs/faq.md#what-is-legacy-replication))
|
||||
for reading & sharing with the broader SSB ecosystem.
|
||||
|
||||
`rss-butt-plug` generates an invite every time it runs which you can use to
|
||||
invite clients with. Feeds will be polled every 5 minutes by default, you can
|
||||
configure this with `-p`.
|
||||
|
||||
## Limitations :stop_sign:
|
||||
|
||||
* Multiple RSS feeds are not supported. It would be great to have but I don't
|
||||
know how to do it (afaiu, `go-sbot` is one identity per-instance). You could
|
||||
run multiple instances of `rss-butt-plug`, an instance per RSS feed. You'd
|
||||
need to tweak the ports in the `rss-butt-plug.yaml` to not have conflicts but
|
||||
it could work.
|
||||
|
||||
* The HTML -> Markdown might be a bit dodgy, so I would recommend doing some
|
||||
testing on local throwaway Patchwork / `rss-butt-plug` identities before
|
||||
doing any mainnet replication. You can test parse a feed by passing it as an
|
||||
argument, e.g. `./rss-butt-plug https://laipower.xyz/rss` and the first post
|
||||
will be shown.
|
||||
|
||||
* All `go-ssb` experimental caveats apply, see [the
|
||||
FAQ](https://github.com/ssbc/go-ssb/blob/master/docs/faq.md) for more.
|
||||
|
||||
## Be gentle :monocle_face:
|
||||
|
||||
You may feel the sudden urge to plug loads of RSS feeds into the Scuttleverse.
|
||||
I certainly did. However, with some testing, I quickly realised this isn't a
|
||||
very good idea.
|
||||
|
||||
One example, a lot of great SSB people are also on the Fediverse but the RSS
|
||||
feeds provided by Mastodon trim the titles and don't carry over the images in
|
||||
the posts properly (have tested in a couple of feed readers).
|
||||
|
||||
Also, you can't `@` a plug identity on SSB because there are no humans behind
|
||||
it. So, if you plug an RSS feed connected with other aspects of your online
|
||||
life then how do people speak to you?
|
||||
|
||||
I think it's a question of going slow and waiting for people to test out the
|
||||
tool and see what dynamic it produces. Please test your feeds first, see how it
|
||||
looks, see what content the feed provides and ofc, see if it "fits" being
|
||||
plugged into SSB.
|
||||
|
||||
## Internal features :computer:
|
||||
|
||||
Some internal things that might be nice to know / refer to if also hacking on
|
||||
this or building your own stuff. I would guess that a lot of the following
|
||||
ingredients could be generalised into other kinds of clients, scripts and
|
||||
tools.
|
||||
|
||||
`rss-butt-plug` does the following...
|
||||
|
||||
* Internally manages a `go-sbot` instance, handy for emdedding magic scuttlin'
|
||||
superpowers in your Go scripts.
|
||||
|
||||
* Creates a TCP client which makes requests to the managed `go-sbot` via the
|
||||
MUXRPC interface. E.g. generates an invite.
|
||||
|
||||
* In general, uses some "direct" Go API bindings which work on the local log.
|
||||
These APIs are more low-level and involved than the MUXRPC interface but are
|
||||
more powerful.
|
||||
|
||||
* Image uploads. The HTML is parsed to look for images while converting it to
|
||||
Markdown. When an image is found, it is uploaded as a blob and then the blob
|
||||
ref replaces the traditional link. Clients like Patchwork then know how to
|
||||
show the images in the renderer.
|
||||
|
||||
* Breaking up large posts into root + reply threads so that we do not go over
|
||||
the length limit of a post. The implementation of this is quite a hack, so go
|
||||
easy on me.
|
||||
|
||||
* Uses a `goreleaser` config to create cross-platform binaries.
|
||||
|
||||
## Inspo :sunflower:
|
||||
|
||||
* `%fzitwDLKcKgGL3RKv6BrBHzQRr1qC7QCeC6PpirMwwE=.sha256`
|
||||
* `%JmVycMs1KqxH0LYzXDnbhNCKH9sCOtBZE/yDU+cBWoY=.sha256`
|
||||
|
||||
## ACK :+1:
|
||||
|
||||
* [`go-ssb`](https://github.com/ssbc/go-ssb)
|
||||
* [`egonelbre/gophers`](https://github.com/egonelbre/gophers)
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
module decentral1se/rss-butt-plug
|
||||
|
||||
go 1.18
|
||||
|
||||
require (
|
||||
github.com/JohannesKaufmann/html-to-markdown v1.3.6
|
||||
github.com/PuerkitoBio/goquery v1.8.0
|
||||
github.com/mmcdole/gofeed v1.1.3
|
||||
github.com/ssbc/go-luigi v0.3.7-0.20221019204020-324065b9a7c6
|
||||
github.com/ssbc/go-ssb v0.2.2-0.20221114231348-43505cca26d4
|
||||
github.com/ssbc/go-ssb-refs v0.5.2-0.20221019090322-8b558c2f31de
|
||||
gopkg.in/yaml.v2 v2.4.0
|
||||
)
|
||||
|
||||
require (
|
||||
filippo.io/edwards25519 v1.0.0 // indirect
|
||||
github.com/RoaringBitmap/roaring v1.2.1 // indirect
|
||||
github.com/andybalholm/cascadia v1.3.1 // indirect
|
||||
github.com/bits-and-blooms/bitset v1.3.3 // indirect
|
||||
github.com/cespare/xxhash v1.1.0 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.1.2 // indirect
|
||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||
github.com/dgraph-io/badger/v3 v3.2103.4 // indirect
|
||||
github.com/dgraph-io/ristretto v0.1.1 // indirect
|
||||
github.com/dgraph-io/sroar v0.0.0-20220527172339-b92b7eaaf6e0 // indirect
|
||||
github.com/dustin/go-humanize v1.0.0 // indirect
|
||||
github.com/go-kit/kit v0.12.0 // indirect
|
||||
github.com/go-logfmt/logfmt v0.5.1 // indirect
|
||||
github.com/gogo/protobuf v1.3.2 // indirect
|
||||
github.com/golang/glog v1.0.0 // indirect
|
||||
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
|
||||
github.com/golang/protobuf v1.5.2 // indirect
|
||||
github.com/golang/snappy v0.0.4 // indirect
|
||||
github.com/google/flatbuffers v22.10.26+incompatible // indirect
|
||||
github.com/gorilla/websocket v1.5.0 // indirect
|
||||
github.com/hashicorp/errwrap v1.1.0 // indirect
|
||||
github.com/hashicorp/go-multierror v1.1.1 // indirect
|
||||
github.com/json-iterator/go v1.1.12 // indirect
|
||||
github.com/karrick/bufpool v1.2.0 // indirect
|
||||
github.com/karrick/gopool v1.2.2 // indirect
|
||||
github.com/keks/persist v0.0.0-20210520094901-9bdd97c1fad2 // indirect
|
||||
github.com/klauspost/compress v1.15.12 // indirect
|
||||
github.com/libp2p/go-reuseport v0.2.0 // indirect
|
||||
github.com/machinebox/progress v0.2.0 // indirect
|
||||
github.com/mmcdole/goxpp v0.0.0-20200921145534-2f3784f67354 // indirect
|
||||
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
|
||||
github.com/modern-go/reflect2 v1.0.2 // indirect
|
||||
github.com/mschoch/smat v0.2.0 // indirect
|
||||
github.com/pkg/errors v0.9.1 // indirect
|
||||
github.com/rs/cors v1.8.2 // indirect
|
||||
github.com/ssbc/go-gabbygrove v0.0.0-20221025092911-c274a44c3523 // indirect
|
||||
github.com/ssbc/go-metafeed v1.1.3-0.20221019090205-458925e39156 // indirect
|
||||
github.com/ssbc/go-muxrpc/v2 v2.0.14-0.20221111190521-10382533750c // indirect
|
||||
github.com/ssbc/go-netwrap v0.1.5-0.20221019160355-cd323bb2e29d // indirect
|
||||
github.com/ssbc/go-secretstream v1.2.11-0.20221111164233-4b41f899f844 // indirect
|
||||
github.com/ssbc/go-ssb-multiserver v0.1.5-0.20221019203850-917ae0e23d57 // indirect
|
||||
github.com/ssbc/margaret v0.4.4-0.20221101112304-4f5815095ef3 // indirect
|
||||
github.com/ugorji/go/codec v1.2.7 // indirect
|
||||
github.com/zeebo/bencode v1.0.0 // indirect
|
||||
go.cryptoscope.co/nocomment v0.0.0-20210520094614-fb744e81f810 // indirect
|
||||
go.mindeco.de v1.12.0 // indirect
|
||||
go.opencensus.io v0.23.0 // indirect
|
||||
golang.org/x/crypto v0.2.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20221025133541-111beb427cde // indirect
|
||||
golang.org/x/net v0.2.0 // indirect
|
||||
golang.org/x/sync v0.1.0 // indirect
|
||||
golang.org/x/sys v0.2.0 // indirect
|
||||
golang.org/x/text v0.4.0 // indirect
|
||||
gonum.org/v1/gonum v0.12.0 // indirect
|
||||
google.golang.org/protobuf v1.28.1 // indirect
|
||||
)
|
||||
|
|
@ -0,0 +1,622 @@
|
|||
// package main is a SSB client which "plugs" a RSS feed into the Scuttleverse.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
md "github.com/JohannesKaufmann/html-to-markdown"
|
||||
"github.com/PuerkitoBio/goquery"
|
||||
"github.com/mmcdole/gofeed"
|
||||
"github.com/ssbc/go-luigi"
|
||||
"github.com/ssbc/go-ssb"
|
||||
refs "github.com/ssbc/go-ssb-refs"
|
||||
ssbClient "github.com/ssbc/go-ssb/client"
|
||||
"github.com/ssbc/go-ssb/message"
|
||||
"github.com/ssbc/go-ssb/sbot"
|
||||
"gopkg.in/yaml.v2"
|
||||
)
|
||||
|
||||
// Config is a rss-butt-plug config file.
|
||||
type Config struct {
|
||||
DataDir string `yaml:"data-dir"`
|
||||
Feed string `yaml:"feed"`
|
||||
Addr string `yaml:"addr"`
|
||||
Port string `yaml:"port"`
|
||||
WsPort string `yaml:"ws-port"`
|
||||
ShsCap string `yaml:"shs-cap"`
|
||||
KeyPair ssb.KeyPair `yaml:"key-pair,omitempty"`
|
||||
Avatar string `yaml:"avatar,omitempty"`
|
||||
}
|
||||
|
||||
// Post is a ssb post message.
|
||||
type Post struct {
|
||||
Type string `json:"type"`
|
||||
Link string `json:"link"`
|
||||
Text string `json:"text"`
|
||||
Root string `json:"root,omitempty"`
|
||||
}
|
||||
|
||||
// help is the rss-butt-plug CLI help output.
|
||||
const help = `rss-butt-plug [options] [<feed>]
|
||||
|
||||
A SSB client which "plugs" a RSS feed into the Scuttleverse.
|
||||
|
||||
An example configuration file:
|
||||
|
||||
---
|
||||
data-dir: ~/.rss-butt-plug
|
||||
feed: https://openrss.org/opencollective.com/secure-scuttlebutt-consortium/updates
|
||||
addr: localhost
|
||||
port: 8008
|
||||
ws-port: 8989
|
||||
shs-cap: "1KHLiKZvAvjbY1ziZEHMXawbCEIM6qwjCDm3VYRan/s="
|
||||
avatar: https://images.opencollective.com/secure-scuttlebutt-consortium/676f245/logo/256.png
|
||||
|
||||
Arguments:
|
||||
<feed> a feed to test parsing
|
||||
|
||||
Options:
|
||||
-h output help
|
||||
-c path to config file
|
||||
-p feed poll frequency in minutes
|
||||
`
|
||||
|
||||
// maxPostLength is a post limit set by rss-butt-plug which is smaller than the
|
||||
// actual max post length of 8192, to allow some buffer. Any RSS post with a
|
||||
// greater length will be split up into threads.
|
||||
const maxPostLength = 7000
|
||||
|
||||
var helpFlag bool
|
||||
var debugFlag bool
|
||||
var configFlag string
|
||||
var pollFrequencyFlag int
|
||||
|
||||
// handleCliFlags parses CLI flags.
|
||||
func handleCliFlags() error {
|
||||
flag.BoolVar(&helpFlag, "h", false, "output help")
|
||||
flag.StringVar(&configFlag, "c", "rss-butt-plug.yaml", "config file")
|
||||
flag.IntVar(&pollFrequencyFlag, "p", 5, "feed poll frequency in minutes")
|
||||
flag.Parse()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// parseRSSFeed parses an entire RSS feed into memory.
|
||||
func parseRSSFeed(url string) (gofeed.Feed, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
feedParser := gofeed.NewParser()
|
||||
feed, err := feedParser.ParseURLWithContext(url, ctx)
|
||||
if err != nil {
|
||||
return gofeed.Feed{}, fmt.Errorf("unable to parse %s: %w", url, err)
|
||||
}
|
||||
|
||||
return *feed, nil
|
||||
}
|
||||
|
||||
// getImage retrieves an image from the internet.
|
||||
func getImage(url string) (io.Reader, error) {
|
||||
response, err := http.Get(url)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("getImage: unable to retrieve %s: %w", url, err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
|
||||
if response.StatusCode != 200 {
|
||||
return nil, fmt.Errorf("getImage: unable to retrieve %s: HTTP %d", url, response.StatusCode)
|
||||
}
|
||||
|
||||
body, error := ioutil.ReadAll(response.Body)
|
||||
if error != nil {
|
||||
return nil, fmt.Errorf("getImage: unable to read response body: %w", err)
|
||||
}
|
||||
|
||||
return bytes.NewReader(body), nil
|
||||
}
|
||||
|
||||
// htmlToMarkdown converts HTML to Markdown. Image links are processed into
|
||||
// blob refs for SSB client readers.
|
||||
func htmlToMarkdown(content string, pub *sbot.Sbot, postBlobs bool) (string, error) {
|
||||
var markdown string
|
||||
|
||||
converter := md.NewConverter("", true, nil)
|
||||
|
||||
converter.AddRules(
|
||||
md.Rule{
|
||||
Filter: []string{"img"},
|
||||
Replacement: func(content string, selec *goquery.Selection, opt *md.Options) *string {
|
||||
if postBlobs {
|
||||
src, _ := selec.Attr("src")
|
||||
srcReader, err := getImage(src)
|
||||
if err != nil {
|
||||
log.Fatal(fmt.Errorf("htmlToMarkdown: %w", err))
|
||||
}
|
||||
|
||||
ref, err := pub.BlobStore.Put(srcReader)
|
||||
if err != nil {
|
||||
log.Fatal(fmt.Errorf("htmlToMarkdown: %w", err))
|
||||
}
|
||||
|
||||
log.Printf("htmlToMarkdown: successfully posted %s as blob", src)
|
||||
|
||||
return md.String("![](" + ref.String() + ")")
|
||||
}
|
||||
|
||||
return nil
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
markdown, err := converter.ConvertString(content)
|
||||
if err != nil {
|
||||
return markdown, fmt.Errorf("htmlToMarkdown: unable to convert html to markdown: %w", err)
|
||||
}
|
||||
|
||||
return markdown, nil
|
||||
}
|
||||
|
||||
// firstRSSPost retrieves the post content of the first message of a RSS feed.
|
||||
func firstRSSPost(testFeed string, pub *sbot.Sbot) (string, error) {
|
||||
var markdown string
|
||||
|
||||
feed, err := parseRSSFeed(testFeed)
|
||||
if err != nil {
|
||||
return markdown, fmt.Errorf("firstRSSPost: %w", err)
|
||||
}
|
||||
|
||||
for _, feed := range feed.Items {
|
||||
content := feed.Content
|
||||
if feed.Content == "" {
|
||||
content = feed.Description
|
||||
}
|
||||
|
||||
log.Printf("firstRSSPost: converting '%s' to markdown", feed.Title)
|
||||
|
||||
markdown, err = htmlToMarkdown(content, pub, false)
|
||||
if err != nil {
|
||||
return markdown, fmt.Errorf("firstRSSPost: %w", err)
|
||||
}
|
||||
|
||||
return markdown, err
|
||||
}
|
||||
|
||||
return markdown, fmt.Errorf("firstRSSPost: %w", err)
|
||||
}
|
||||
|
||||
// loadYAMLConfig loads a rss-butt-plug YAML user config.
|
||||
func loadYAMLConfig() (Config, error) {
|
||||
var cfg Config
|
||||
|
||||
configPath, err := filepath.Abs(configFlag)
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("loadYAMLConfig: unable to convert %s to an absolute path: %w", configFlag, err)
|
||||
}
|
||||
|
||||
conf, err := os.ReadFile(configPath)
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("loadYAMLConfig: unable to read %s: %w", configFlag, err)
|
||||
}
|
||||
|
||||
err = yaml.UnmarshalStrict(conf, &cfg)
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("loadYAMLConfig: unable to unmarshal %s: %w", string(conf), err)
|
||||
}
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// generatePublicInvite generates an invite by speaking to a local go-sbot instance. It
|
||||
// uses a high "uses" value (666) so as to make the invite usable to more
|
||||
// people. It's more a public share invite in that sense.
|
||||
func generatePublicInvite(pub *sbot.Sbot) (string, error) {
|
||||
var token string
|
||||
|
||||
client, err := ssbClient.NewTCP(pub.KeyPair, pub.Network.GetListenAddr())
|
||||
if err != nil {
|
||||
return token, fmt.Errorf("generatePublicInvite: unable to initalise TCP client: %w", err)
|
||||
}
|
||||
|
||||
token, err = client.InviteCreate(message.InviteCreateArgs{Uses: 666})
|
||||
if err != nil {
|
||||
return token, fmt.Errorf("generatePublicInvite: unable to create invite: %w", err)
|
||||
}
|
||||
|
||||
if err := client.Close(); err != nil {
|
||||
return token, fmt.Errorf("generatePublicInvite: unable to close TCP client: %w", err)
|
||||
}
|
||||
|
||||
if strings.Contains(token, "[::]") {
|
||||
token = strings.Replace(token, "[::]", "localhost", 1)
|
||||
}
|
||||
|
||||
return token, nil
|
||||
}
|
||||
|
||||
// messagesFromLog retrieves all messages from the user log.
|
||||
func messagesFromLog(pub *sbot.Sbot) ([]Post, error) {
|
||||
var posts []Post
|
||||
|
||||
src, err := pub.ReceiveLog.Query()
|
||||
if err != nil {
|
||||
return posts, fmt.Errorf("messagesFromLog: unable to query log: %w", err)
|
||||
}
|
||||
|
||||
for {
|
||||
var post Post
|
||||
|
||||
v, err := src.Next(context.Background())
|
||||
if luigi.IsEOS(err) {
|
||||
break
|
||||
}
|
||||
|
||||
message := v.(refs.Message)
|
||||
content := message.ContentBytes()
|
||||
if err = json.Unmarshal(content, &post); err != nil {
|
||||
return posts, fmt.Errorf("messagesFromLog: unable to unmarshal %s: %w", string(content), err)
|
||||
}
|
||||
|
||||
posts = append(posts, post)
|
||||
}
|
||||
|
||||
return posts, nil
|
||||
}
|
||||
|
||||
// newSbot instantiates a new go-sbot instance.
|
||||
func newSbot(cfg Config) (*sbot.Sbot, error) {
|
||||
dataDir, err := filepath.Abs(cfg.DataDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("newSbot: unable to convert %s to an absolute path: %w", cfg.DataDir, err)
|
||||
}
|
||||
|
||||
sbotOpts := []sbot.Option{
|
||||
sbot.EnableAdvertismentBroadcasts(true),
|
||||
sbot.EnableAdvertismentDialing(true),
|
||||
sbot.LateOption(sbot.WithUNIXSocket()),
|
||||
sbot.WithHops(2),
|
||||
sbot.WithListenAddr(fmt.Sprintf(":%s", cfg.Port)),
|
||||
sbot.WithRepoPath(dataDir),
|
||||
sbot.WithWebsocketAddress(fmt.Sprintf(":%s", cfg.WsPort)),
|
||||
}
|
||||
|
||||
pub, err := sbot.New(sbotOpts...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("newSbot: unable to initialise sbot: %w", err)
|
||||
}
|
||||
|
||||
c := make(chan os.Signal, 1)
|
||||
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
|
||||
go func() {
|
||||
<-c
|
||||
|
||||
pub.Shutdown()
|
||||
time.Sleep(2 * time.Second)
|
||||
|
||||
if err := pub.Close(); err != nil {
|
||||
log.Fatal(fmt.Errorf("newSbot: %w", err))
|
||||
}
|
||||
|
||||
time.Sleep(2 * time.Second)
|
||||
os.Exit(0)
|
||||
}()
|
||||
|
||||
return pub, nil
|
||||
}
|
||||
|
||||
// serveSbot serves a go-sbot over the network.
|
||||
func serveSbot(pub *sbot.Sbot) {
|
||||
for {
|
||||
ctx := context.TODO()
|
||||
err := pub.Network.Serve(ctx)
|
||||
if err != nil {
|
||||
log.Fatal(fmt.Errorf("serveSbot: %w", err))
|
||||
}
|
||||
|
||||
time.Sleep(1 * time.Second)
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
err := pub.Close()
|
||||
if err != nil {
|
||||
log.Fatal(fmt.Errorf("serveSbot: %w", err))
|
||||
}
|
||||
default:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// getNewRSSPosts gathers new posts from a RSS feed.
|
||||
func getNewRSSPosts(feed gofeed.Feed, posts []Post, pub *sbot.Sbot) ([]map[string]interface{}, error) {
|
||||
var messages []map[string]interface{}
|
||||
|
||||
for idx := len(feed.Items) - 1; idx >= 0; idx-- {
|
||||
feed := feed.Items[idx]
|
||||
alreadyPosted := false
|
||||
|
||||
for _, post := range posts {
|
||||
if feed.Link == post.Link {
|
||||
alreadyPosted = true
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if alreadyPosted {
|
||||
log.Printf("getNewRSSPosts: skipping %s, already posted", feed.Link)
|
||||
continue
|
||||
}
|
||||
|
||||
feedContent := feed.Content
|
||||
if feed.Content == "" {
|
||||
feedContent = feed.Description
|
||||
}
|
||||
|
||||
log.Printf("getNewRSSPosts: converting '%s' to markdown", feed.Title)
|
||||
|
||||
markdown, err := htmlToMarkdown(feedContent, pub, true)
|
||||
if err != nil {
|
||||
return messages, fmt.Errorf("getNewRSSPosts: %w", err)
|
||||
}
|
||||
|
||||
content := fmt.Sprintf("# %s\n", feed.Title)
|
||||
|
||||
if feed.Image != nil {
|
||||
srcReader, err := getImage(feed.Image.URL)
|
||||
if err != nil {
|
||||
return messages, fmt.Errorf("getNewRSSPosts: %w", err)
|
||||
}
|
||||
|
||||
ref, err := pub.BlobStore.Put(srcReader)
|
||||
if err != nil {
|
||||
return messages, fmt.Errorf("getNewRSSPosts: unable to upload blob: %w", err)
|
||||
}
|
||||
|
||||
content += "\n![](" + ref.String() + ")\n"
|
||||
}
|
||||
|
||||
content += markdown
|
||||
content += "\n---\n[Clearnet link](" + feed.Link + ")\n"
|
||||
|
||||
messages = append(messages, map[string]interface{}{
|
||||
"type": "post",
|
||||
"link": feed.Link,
|
||||
"text": content,
|
||||
})
|
||||
}
|
||||
|
||||
return messages, nil
|
||||
}
|
||||
|
||||
// createAboutMessage publishes an about message with accompanying avatar, if available in config).
|
||||
func createAboutMessage(pub *sbot.Sbot, posts []Post, feed gofeed.Feed, cfg Config) (map[string]interface{}, bool, error) {
|
||||
for _, post := range posts {
|
||||
if post.Type == "about" {
|
||||
log.Printf("createAboutMessage: skipping about message post, already done")
|
||||
return nil, false, nil
|
||||
}
|
||||
}
|
||||
|
||||
message := map[string]interface{}{
|
||||
"type": "about",
|
||||
"about": pub.KeyPair.ID(),
|
||||
"name": feed.Title,
|
||||
}
|
||||
|
||||
if cfg.Avatar != "" {
|
||||
srcReader, err := getImage(cfg.Avatar)
|
||||
if err != nil {
|
||||
return nil, false, fmt.Errorf("createAboutMessage: %w", err)
|
||||
}
|
||||
|
||||
ref, err := pub.BlobStore.Put(srcReader)
|
||||
if err != nil {
|
||||
return nil, false, fmt.Errorf("createAboutMessage: unable to post blob: %w", err)
|
||||
}
|
||||
|
||||
message["image"] = ref.String()
|
||||
}
|
||||
|
||||
log.Printf("createAboutMessage: creating about message post")
|
||||
|
||||
return message, true, nil
|
||||
}
|
||||
|
||||
// chunkByLine chunks a full markdown converted RSS post into a thread.
|
||||
// Meaning, a series of chunks which fit under the max post size of a ssb post.
|
||||
func chunkByLine(content string) []string {
|
||||
var chunks []string
|
||||
|
||||
toChunk := content
|
||||
for toChunk != "" {
|
||||
if len(toChunk) <= maxPostLength {
|
||||
chunks = append(chunks, toChunk)
|
||||
toChunk = ""
|
||||
continue
|
||||
}
|
||||
|
||||
chunkIdx := maxPostLength
|
||||
for toChunk[chunkIdx] != 10 {
|
||||
chunkIdx--
|
||||
}
|
||||
|
||||
chunks = append(chunks, toChunk[:chunkIdx])
|
||||
toChunk = toChunk[chunkIdx:]
|
||||
}
|
||||
|
||||
return chunks
|
||||
}
|
||||
|
||||
// publishAsThread posts a message as a series of linked messages. This is
|
||||
// useful when the content of the RSS post is too long.
|
||||
func publishAsThread(publish ssb.Publisher, message map[string]interface{}) error {
|
||||
chunks := chunkByLine(message["text"].(string))
|
||||
|
||||
root := map[string]interface{}{
|
||||
"type": "post",
|
||||
"link": message["link"],
|
||||
"text": chunks[0],
|
||||
}
|
||||
|
||||
ref, err := publish.Publish(root)
|
||||
if err != nil {
|
||||
return fmt.Errorf("publishAsThread: failed to publish: %w", err)
|
||||
}
|
||||
|
||||
for _, chunk := range chunks[1:] {
|
||||
threadReply := map[string]interface{}{
|
||||
"type": "post",
|
||||
"link": message["link"],
|
||||
"text": chunk,
|
||||
"root": ref.Key().String(),
|
||||
}
|
||||
_, err := publish.Publish(threadReply)
|
||||
if err != nil {
|
||||
return fmt.Errorf("publishAsThread: failed to publish: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// postMessagesToLog posts messages to the local user feed.
|
||||
func postMessagesToLog(messages []map[string]interface{}, pub *sbot.Sbot) error {
|
||||
publish, err := message.OpenPublishLog(pub.ReceiveLog, pub.Users, pub.KeyPair)
|
||||
if err != nil {
|
||||
return fmt.Errorf("postMessagesToLog: failed to open publish log: %w", err)
|
||||
}
|
||||
|
||||
for _, message := range messages {
|
||||
if message["type"] == "post" {
|
||||
log.Printf("postMessagesToLog: publishing %s to log", message["link"])
|
||||
|
||||
if len(message["text"].(string)) > maxPostLength {
|
||||
log.Printf("postMessagesToLog: turning content of %s into thread, too long", message["link"])
|
||||
if err := publishAsThread(publish, message); err != nil {
|
||||
return fmt.Errorf("postMessagesToLog: unable to thread content for %s: %w", message["link"], err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
_, err := publish.Publish(message)
|
||||
if err != nil {
|
||||
return fmt.Errorf("postMessagesToLog: failed to publish: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// main is the main CLI entrypoint.
|
||||
func main() {
|
||||
handleCliFlags()
|
||||
|
||||
if helpFlag {
|
||||
fmt.Printf(help)
|
||||
os.Exit(0)
|
||||
}
|
||||
|
||||
cfg, err := loadYAMLConfig()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
log.Printf("loaded %s", configFlag)
|
||||
|
||||
pub, err := newSbot(cfg)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
args := os.Args[1:]
|
||||
if len(args) > 0 {
|
||||
markdown, err := firstRSSPost(args[0], pub)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Println(markdown)
|
||||
return
|
||||
}
|
||||
|
||||
go serveSbot(pub)
|
||||
|
||||
log.Print("main: bootstrapped internally managed go-sbot")
|
||||
|
||||
cfg.KeyPair = pub.KeyPair
|
||||
feed, err := parseRSSFeed(cfg.Feed)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
log.Printf("main: parsed %s", cfg.Feed)
|
||||
|
||||
posts, err := messagesFromLog(pub)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
log.Printf("main: retrieved %d posts from log", len(posts))
|
||||
|
||||
var messages []map[string]interface{}
|
||||
|
||||
aboutMessage, posted, err := createAboutMessage(pub, posts, feed, cfg)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
if posted {
|
||||
messages = append(messages, aboutMessage)
|
||||
}
|
||||
|
||||
newRSSPosts, err := getNewRSSPosts(feed, posts, pub)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
for _, newRSSPost := range newRSSPosts {
|
||||
messages = append(messages, newRSSPost)
|
||||
}
|
||||
|
||||
if err := postMessagesToLog(messages, pub); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
token, err := generatePublicInvite(pub)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
log.Printf("main: pub invite: %s", token)
|
||||
|
||||
for {
|
||||
log.Printf("main: going to sleep for %d minutes...", pollFrequencyFlag)
|
||||
time.Sleep(time.Duration(pollFrequencyFlag) * time.Minute)
|
||||
log.Printf("main: waking up to poll %s for new posts", cfg.Feed)
|
||||
|
||||
posts, err := messagesFromLog(pub)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
messages, err := getNewRSSPosts(feed, posts, pub)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
if err := postMessagesToLog(messages, pub); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue